Disk Driver

[facekapow]

Could a disk driver be written for runtime? Because, (correct me if i'm wrong, but) runtimejs aims to be able to be a server OS, right? Like, Node (except an OS)? If it does, doesn't it need somewhere to store the content it'll serve, persistently (not in memory, since it'd be lost when the server shuts down)? I'm just saying, you know. It could become a long term goal, but, eventually it would need a disk driver.

[iefserge]

Yes, we can write one using virtio block device. It's not a priority though, static content could be packaged into initrd image (supported, but not exposed in runtimeify). I agree, mutable filesystem could be useful to store, for example, log or database files. Filesystem infrastructure is insanely complex, however it probably makes sense to have it as a long term goal.

[facekapow]

@iefserge Have you seen https://github.com/natevw/fatfs ? It looks suitable for this project. FAT is a simple fs, maybe a driver for a block device could be written and expose the needed functions for this module. I'll have to take a look at the source later, but it looks OS-independent.

[piranna]

I've already proposed to use fatfs module :-) It only needs a block device driver, that in a first implementation it would be for VirtIO and later for a real hard drive. +1 for this, the idea to use objects that follow the Node.js fs API deems cool :-) El 11/07/2015 13:10, "Ariel Abreu" notifications@github.com escribió:

Have you seen https://github.com/natevw/fatfs ? It looks suitable for this project. FAT is a simple fs, maybe a driver for a block device could be written and expose the needed functions for this module. I'll have to take a look at the source later, but it looks like it isn't OS-dependent.

— Reply to this email directly or view it on GitHub https://github.com/runtimejs/runtime/issues/74#issuecomment-120604939.

[facekapow]

Oh, sorry. Still, it seems pretty good. This weekend-ish i'll see what I can do about the driver, attach an img with a .txt and see if it reads it.

[piranna]

It would be cool! :-D I think the easiest one would be to create a memory-only object with the block API fatfs needs, this way you could start tinkering with it :-) Problem would be to format it... :-P El 11/07/2015 13:41, "Ariel Abreu" notifications@github.com escribió:

Oh, sorry. Still, it seems pretty good. This weekend-ish i'll see what I can do about the driver, attach an img with a .txt and see if it reads it.

— Reply to this email directly or view it on GitHub https://github.com/runtimejs/runtime/issues/74#issuecomment-120606863.

[facekapow]

You mean erasing stuff? That can be done by zeroing the bytes (for now, better implementation later, since that erases all the data on those bytes).

[piranna]

No, I talk about the internal FAT structures that fatfs will read. The in-memory block device can be build with a UInt8Array, but it needs to be filked in the way fatfs expects... El 11/07/2015 14:06, "Ariel Abreu" notifications@github.com escribió:

You mean erasing stuff? That can be sone by zeroing the bytes (for now, better implementation later, since that erases all the data on those bytes).

— Reply to this email directly or view it on GitHub https://github.com/runtimejs/runtime/issues/74#issuecomment-120608090.

[facekapow]

Well, that, yeah.

[facekapow]

I'll cross that bridge when I get there.

[facekapow]

@piranna Happen to know how to attach a disk in QEMU? I've tried -hda, -drive file=, and even plain old qemu-system-x86_64 diskname (other options), but nothing makes it's subsytem id (2) show up.

[piranna]

-hda {file} should be enough... El 11/07/2015 15:13, "Ariel Abreu" notifications@github.com escribió:

@piranna https://github.com/piranna Happen to know how to attach a disk in QEMU? I've tried -hda, -drive file=, and even plain old qemu-system-x86_64 diskname (other options), but nothing makes it's subsytem id (2) show up.

— Reply to this email directly or view it on GitHub https://github.com/runtimejs/runtime/issues/74#issuecomment-120615796.

[facekapow]

Nope, that doesn't make the disk show up, and i'm pretty sure i've formatted it as FAT, yet it tells me to specify the format as 'raw'. I'm on Mac, so that may be the problem (which conforms to Unix, not Linux, stupid Apple).

[piranna]

Format doesn't matter, it's shown as a block device independent of what's inside. Probably runtime.js is not capable of detect it, or maybe you need to initialize IDE bus first. That's why I told you about using VirtIO or crafting an on-memory object with the block API fatfs expects. El 11/07/2015 15:21, "Ariel Abreu" notifications@github.com escribió:

Nope, that doesn't make the disk show up, and i'm pretty sure i've formatted it as FAT, yet it tells me to specify the format as 'raw'. I'm on Mac, so that may be the problem (which conforms to Unix, not Linux).

— Reply to this email directly or view it on GitHub https://github.com/runtimejs/runtime/issues/74#issuecomment-120616714.

[iefserge]

Probably need to tell qemu to use virtio for the disk

[facekapow]

@piranna I need a driver for a real disk image with persistent storage. In-memory drives would be easier, but not persistent. @iefserge How?

[piranna]

http://www.linux-kvm.org/page/Virtio

-drive enable virtio block device. El 11/07/2015 15:33, "Ariel Abreu" notifications@github.com escribió:

@piranna https://github.com/piranna I need a driver for a real disk image with persistent storage. In-memory drives would be easier, but not persistent. @iefserge https://github.com/iefserge How?

— Reply to this email directly or view it on GitHub https://github.com/runtimejs/runtime/issues/74#issuecomment-120618300.

[iefserge]

http://wiki.qemu.org/download/qemu-doc.html extra option if=virtio

[facekapow]

Thanks. Though, I get: qemu-system-x86_64: -drive file=disk.img: drive with bus=0, unit=0 (index=0) exists, probably because of the initrd.

[facekapow]

Even removing the initrd I still get the qemu-system-x86_64: -drive file=disk.img: drive with bus=0, unit=0 (index=0) exists error. Any ideas?

[facekapow]

@iefserge How do I write headers? How does net do it? I saw the virtioHeader function, but it's never used. I was able to get the disk in (instead of a space in between file=disk.img and if=virtio, it needed a comma) and setup a queue, but I need to write the headers for every operation (otherwise I get qemu-system-x86_64: virtio-blk missing headers), so how do write the headers?

[iefserge]

@ArielAbreu it should be in virtio spec, haven't really looked into disk io. There is probably some format it expects and it's different for network and disk. Spec: http://ozlabs.org/~rusty/virtio-spec/virtio-0.9.5.pdf

[facekapow]

No, I mean how do I write it? Using JS? I know the headers (I saw the spec), but I want to know how to actually write the header in JS.

[iefserge]

yep, using regular Uint8Array, I'm using this https://github.com/iefserge/u8-view. 64-bit read/writes could be done using two dwords. Example: https://github.com/runtimejs/runtime/blob/master/js/core/net/ip4-header.js#L51

[facekapow]

Still get the error. My code (the relevant part):

function setUint32BE(u8, offset, value) {
  u8[offset] = value >>> 24;
  u8[offset + 1] = value >>> 16;
  u8[offset + 2] = value >>> 8;
  u8[offset + 3] = value >>> 0;
  return u8;
};

function setUint64BE(u8, offset, value) {
  u8[offset] = value >>> 40;
  u8[offset + 1] = value >>> 32;
  u8[offset + 2] = value >>> 24;
  u8[offset + 3] = value >>> 16;
  u8[offset + 4] = value >>> 8;
  u8[offset + 5] = value >>> 0;
  return u8;
}

// ...

var res = new Uint8Array(16);
res = setUint32BE(res, 0, type);
res = setUint32BE(res, 4, ioprio);
res = setUint64BE(res, 8, sector);
res[15] = data;

reqQueue.placeBuffers([res], true); // it fails here

[piranna]

function setUint64BE(u8, offset, value) { u8[offset] = value >>> 40; u8[offset + 1] = value >>> 32; u8[offset + 2] = value >>> 24; u8[offset + 3] = value >>> 16; u8[offset + 4] = value >>> 8; u8[offset + 5] = value >>> 0; return u8; }

That function is wrong, the code is from a setUint48BE implementation, not 64, you are loosing the two higher bytes and also they are offset incorrectly. Anyway, I would not work with 64 bits integers on Javascript directly but instead concatenating two 32 bits integers, or if so, I would use a big nums library.

[facekapow]

My new code (still doesn't work):

function setUint32BE(u8, offset, value) {
  u8[offset] = value >>> 24;
  u8[offset + 1] = value >>> 16;
  u8[offset + 2] = value >>> 8;
  u8[offset + 3] = value >>> 0;
  return u8;
};

function setUint64BE(u8, offset, value) {
  u8[offset] = value >>> 52;
  u8[offset + 1] = value >>> 48;
  u8[offset + 2] = value >>> 40;
  u8[offset + 3] = value >>> 32;
  u8[offset + 4] = value >>> 24;
  u8[offset + 5] = value >>> 16;
  u8[offset + 6] = value >>> 8;
  u8[offset + 7] = value >>> 0;
  return u8;
}

// ...

var res = new Uint8Array(18);
res = setUint32BE(res, 0, type);
res = setUint32BE(res, 4, ioprio);
res = setUint64BE(res, 8, sector);
res[17] = data;

reqQueue.placeBuffers([res], true); // it fails here

[iefserge]

@ArielAbreu >>> operator always converts value to Uint32.

You'd need something like:

setUint64BE(hi, lo) {
  setUint32BE(offset, hi);
  setUint32BE(offset + 4, lo);
}

I think js is going to get support for Uint64/Int64 at some point, so this is the temporary workaround.

[facekapow]

Newer code (still broken):

function setUint64BE(u8, offset, value) {
  var half = (value === 0 || value === 1) ? value : value / 2;
  setUint32BE(u8, offset, half);
  setUint32BE(u8, offset + 4, half);
  return u8;
}

// ...

var res = new Uint8Array(18 + data.length);
res = setUint32BE(res, 0, type);
res = setUint32BE(res, 4, ioprio);
res = setUint64BE(res, 8, sector);

for (var i = 0; i < data.length; i++) {
  res[17 + i] = data[i];
};

reqQueue.placeBuffers([res], true); // it still fails here, for the third time

[facekapow]

Examining QEMU's source, https://github.com/qemu/qemu/blob/58e8b33518fd2bb6dce0ba7b6347c3df85aea3c6/hw/block/virtio-blk.c#L489-L492, it has something to do with the req->elm.in_num and req->elm.out_num (probably the in_num, since i'm reading from the disk). Didn't have time to find out where those come from, i'll check it later.

[iefserge]

In case you need to dump Uint8Arrays https://github.com/iefserge/buffer-hexdump for debugging.

[piranna]

Newer code (still broken):

function setUint64BE(u8, offset, value) { var half = (value === 0 || value === 1) ? value : value / 2; setUint32BE(u8, offset, half); setUint32BE(u8, offset + 4, half); return u8; }

This code don't make sense, what's half? Why you are spliting it that way? It's better that you forget about using 64 bits operators and use 32 bits ones (prefered), or use a BigNums library. A correct version of your function would be:

function setUint64BE(u8, offset, hi, lo) { setUint32BE(u8, offset, hi); setUint32BE(u8, offset + 4, lo); return u8; }

That as you can see, it's all about using 32 bits operations and data internally. Where did you get the 'sector' variable? Maybe you should fetch it in two independent 32 bits operations, or if it's only about filling the uint8array, maybe you could be able to compose it or fill it with some copy() methods...

[facekapow]

Thanks, just tried that but still no luck. I can't seem to figure out this error. While doing a global Github code search for VIRTIO_BLK_T_IN, I found this. I'm gonna ask the guy that made it for help on virtio-blk, since he wrote a driver in JavaScript interfacing virtio to the block device for his project.

[piranna]

I've found https://github.com/jhermsmeier/node-disk and https://github.com/jhermsmeier/node-blockdevice, seems they could be interesting to add block devices support to runtime.js (also they can be able to parse and write disk MBR! :-O ). That guy seems has done some interesting low level things... :-)

[facekapow]

They look great! The problem still is that QEMU complains about headers. I'm going to have to try doing it from C++.

[iefserge]

@piranna yeah, this looks pretty cool. I'm curious how filesystem API should look like in a JavaScript non-UNIX OS? Maybe something like DOM with nodes where you could do:

var root = runtime.fs.root;
var dir = root.getChild('tmp');
var file = new File();
file.write('hello');
file.appendChild(dir);

Or maybe that's too complicated?

[piranna]

I'm curious how filesystem API should look like in a JavaScript non-UNIX OS?

Node.js fs module API is easy to use, and filesystem objects would return it directly so any object that expose that API can be used as a filesystem in a similar way FUSE does...

@iefserge, your DOM-like API is interesting, but going to do things different, maybe better use properties and literal objects as directories (a folder is mostly a set of archives with unique names...)

[iefserge]

The path syntax could be supported as well

var file = root.locate('/tmp/file.txt');

The issue with object properties is that it would require to load all fs tree in memory and it's harder to synchronize it between isolates and separate heaps.

[iefserge]

We can even define a JS iterators for FS objects

for (let file of dir) {
  console.log(file.name)
}

[piranna]

The issue with object properties is that it would require to load all fs tree in memory and it's harder to synchronize it between isolates and separate heaps.

That's not necesarily true by using properties and Proxy objects...

We can even define a JS iterators for FS objects

Definitely :-)

[iefserge]

Proxy objects is interesting idea

var file = root.dir1.dir2['file.txt'];

Node.js fs module API is easy to use, and filesystem objects would return it directly so any object that expose that API can be used as a filesystem in a similar way FUSE does...

Yeah, but this doesn't require any OS support. We would probably want to abstract implementation details using VFS and mount API anyway. :)

[piranna]

Proxy objects is interesting idea

var file = root.dir1.dir2['file.txt'];

You got the idea ;-)

Yeah, but this doesn't require any OS support. We would probably want to abstract implementation details using VFS and mount API anyway. :)

I think too, that's why I proposed it that way: runtime.js provides access to the block devices by using an API, and later filesystem objects are build on top of it.

[iefserge]

Basically, I'm curious if fs interface could be made better and more fun to use using available JS features and because we don't have to depend on UNIX interface. We can really do crazy stuff here :)

[iefserge]

Yeah, we can always provide node fs API on top.

[piranna]

Yeah, we can always provide node fs API on top.

That's what the fatfs module does :-)

[iefserge]

yeah, but what if we want to control file permissions and pass only a subtree of FS to some untrusted code running in a web worker? :)

[piranna]

Interesting question... The subtree is easy, just give it a FS object whose root is the folder you want to give it, just like PyFilesystem does, or the Proxy object that represent that folder :-) Regarding file permissions we would need first to define what we want to control first, but it could be feasable too (Node.js fs API has support for it).

[iefserge]

My only concern with Node FS API is that it almost directly maps to UNIX syscalls, which I think doesn't make sense for the system like runtime (at least not as the lowest level exposed by the system).

[piranna]

They are mostly basic operations, but if you want to go really low level, a CRUD API (Create, Read, Update & Delete) directly on the block device is what you are looking for, the other operations can be build on top of them. A filesystem is only a virtual abstraction, there are systems like PhantomOS that don't have a filesystem at all, they are all in-memory objects that can get swapped to disk for persistence...

[iefserge]

Block devices are on a separate level from the filesystem, they'll need a separate API (mostly for drivers).

[iefserge]

Some Node APIs return or take a file descriptors, there are also error codes like ENOENT.

fs.fstat(fd, callback)# Asynchronous fstat(2). The callback gets two arguments (err, stats) where stats is a fs.Stats object. fstat() is identical to stat(), except that the file to be stat-ed is specified by the file descriptor fd.

IMO they're mostly bindings instead of the first-class API designed for JavaScript use.