Feaure: Filebased device driver ;use LocalStorage/BrowserFS ; a sketch

[angrave]

Some initial thoughts on how to implement a browser-backed filesystem. i.e. files stored in Linux are just file entries stored inside the browser.

Use BrowserFS that amongst other things, wraps LocalStorage and provides a nodejs-style filesystem API https://github.com/jvilk/BrowserFS/blob/master/README.md http://nodejs.org/api/fs.html BrowserFS supports directories, lazy-loading from a remote server, basic inode properties, filesystem operations (lstat,fstat,mkdir,...) https://github.com/jvilk/BrowserFS/wiki/Using-BrowserFS

Sync and Async versions of most fs calls are available.

How? We need two portions- A linux device driver that talks to a custom device a new filesystem memory-mapped device (similar to ata.js / uart.js) that calls out to BrowserFS.

The device driver could be written starting from wrapfs. Not sure if wrapfs from 1999 is still relevant or useful but (from first glance) it appears to implement some of the heavy lifting.

• or we just program directly against linux vfs API

http://wrapfs.filesystems.org/ http://git.fsl.cs.sunysb.edu/?p=wrapfs-latest.git;a=blob;f=Documentation/filesystems/wrapfs.txt http://www.linuxjournal.com/article/6485?page=0,0

Useful - http://www.filesystems.org/docs/wrapfs/wrapfs.pdf

We will need to define the device interface and need to decide if we want to support async operations. Here's a rough sketch of a ticket-based system (this is to allow eventual async operations)

+0 Ticket-Command +4 Operation (0=read,1=write,2=...) +8 Status (0=Queued,1=Complete,2=Panic/Abnormal abort) +C API result return_val_a (value or address if result data >32 bits) +10 -1f unused +20 param32_a (value or real-address) +24 param32_b (value or address) +28 param32_c (value or address) +100-10ff Reserve 4K Page for big i/o

If the device is sync, then we could assume the results are immediately available for reading in the return val register. For results that return more than 32 bits, this value could point to a struct (possibly inside the device driver memory space or inside kernel ram)

For async operation - Ticket-Command could include a ticket command[8bits] and a new ticket number [24 bits] There are only Four ticket commands - Device Reset New-Ticket ; Status-Check - want to read the status/output of an existing ticket Retire Ticket nnnn; no longer care about ticket nnnn; can discard results.

For sync commands you can declare that tickets are automatically retired when a new ticket is created.

So an idea to explore: pointers would be pass real-address rather than copying data into the memory space of the device. Anyway, this is just a sketch. I look forward to developing this further.

Background reading on Linux VFS http://www.win.tue.nl/~aeb/linux/lk/lk-8.html http://www.tldp.org/LDP/khg/HyperNews/get/fs/vfstour.html http://www.tldp.org/LDP/lki/lki-3.html

[s-macke]

I think there are more ways to provide a filesystem.

1. We can use fuse and implement a special emulator opcode to copy the data. This would be easier to program than a kernel module.
2. NFS filesystems can be already mounted very easily over a ethernet connection. Drawback: the user must have access to the server to copy files into the nfs folder.
3. We can also program a TCP/IP stack in Javascript and an nfs implementation. But I think, this would be a little bit too complicated.
4. We can use the current hidden terminal to copy data back and forth. Drawback: The user must control outside of the emulator which data to copy to the emulator and back.

I don't know how the current virtual servers implement this.

[benjamincburns]

I've been working on this problem. I've got persistent storage working using socket.io in a way that's sort of analogous to how iSCSI works. It's totally transparent to the machine, handles writes well, can be made to work over a high latency connection, and works with any filesystem type.

I'm going to be working on componentizing and cleaning it up tomorrow. I'll push it to my fork and submit a pull request when I'm done.

[ysangkok]

Did you see the implementation in https://github.com/ozaki-r/arm-js ?

[s-macke]

Yes, some time ago. I have checked my kernel configuration. Looks like I can compile it. I will give it a try.

[s-macke]

The first huge step is done. I can boot from a virtio/9p filesystem which is implemented fully in Javascript in form of a tmpfs type filesystem. It is currently in the branch "transferto9p" but this will be standard in future, after all features are implemented and all bugs removed. The ata-device is no longer used. You can find a test website under www.jor1k.com/jor1k/ It boots much faster asynchronously and finally works again for the IPad 2. The other programs will be loaded on demand and no longer before boot. The initial traffic is around 3MB and no longer 9-12MB. A browserfs (onyl chrome?) and a fast and an easy sync feature (with tar.gz files) is finally possible.

[s-macke]

The first version based on this new filesystem is finally online. In the last two months, almost everything of the code changed. Try it. I need bug reports :)

[benjamincburns]

@s-macke Can maybe close this one?