Browser filesystem solutions

[lavelle]

Working on Swing support has left me wanting a better development workflow for the browser interface to Doppio – most Swing-related classes throw a HeadlessException somewhere, leaving the browser the only place they can be debugged.

The browser filesystem needs to be improved before I can work on improving the development tools though – for example adding support for -Xcore-dump requires more than 5mb for the ability to dump a stack of arbitrary size, and even then to save it permanently would require copying and pasting.

This issue outlines the options available for improving the filesystem in the browser.

Browser APIs

There are 4 main APIs available for persistent storage in modern web browsers:

1. Web Storage (aka localStorage). This is what is currently used and, while having a very simple API, is limited to 5mb of storage in total.
2. Web SQL Database. This would provide a more powerful API than Web Storage, but has been abandoned, so we shouldn't use this.
3. IndexedDB. Unlike Web SQL, this is an actively developed API, however it is not supported in Opera and Safari, which would harm Doppio's current cross-browser support. A polyfill is available which could help solve this. No hard limit on the quota, can use up to 20% of the browser's total storage, which is half of the available space on the hard drive (at least in Chrome) – this should be more than enough for Doppio's uses.
4. FileSystem. Only available in Chrome, with a webkit prefix. Lets the application request an arbitrary amount of data, that the user is prompted to allow or deny. Despite being Chrome-only, I think this is worth considering. Chrome is the most popular browser, access to arbitrary storage would make development a lot easier, and we could still fall back to localStorage on other browsers. This has the advantage of being the only browser API that actually writes files in their original state to disk, making it the easiest to access from outside the browser.

   Other options

5. Running a local server. There's already a WEBrick server running in development, so it wouldn't be too much overhead to add an endpoint for the browser demo to POST some data to be saved locally, perhaps in a browser_fs directory created in the project root. This would only work for development though, obviously.
6. Cloud storage APIs. As discussed in #41, this would be a viable solution for production, and would effectively remove the quota limit. I've used dropbox.js and it's really easy to get it working.
7. Browser extensions. Slightly out-there, just trying to exhaustively list every possibility. Chrome extensions can use a permission that allows for unlimited data storage, so we could provide a version of Doppio that runs as a browser extension and has a more powerful filesystem.

[perimosocordiae]

This is a good run-down of the issues and options. See https://github.com/jvilk/BrowserFS/ for our current progress. :+1:

(Pull requests are welcome there as well!)

[jvilk]

This is an area that I heavily worked on before the summer in BrowserFS, and I hope to get back to. We're fairly close to having enough functionality to switch over to using it in Doppio.

The main issue right now is that I need to add support to some synchronous fs functions that Doppio uses (we perform synchronous writes to files in Doppio, and BFS is currently 100% async). Once we add optional sync API support and add it to the localStorage FS in BrowserFS, we can iterate on moving toward IndexedDB.

Also, BFS will allow us to degrade to the 'best' persistent storage API available. So if we add Web SQL support, we'll be able to use that when IndexedDB isn't available.

[jvilk]

Also, as a hacky workaround, we support saving files in-memory without persisting to localStorage in the browser. It's an extra flag to one of the fs commands... I know we use it for preloading files.

[jvilk]

Ha, we already use the hack to ensure that written files don't hit localStorage! :smile:

https://github.com/int3/doppio/blob/master/src/runtime.coffee#L227

I'll commit something that exposes this option to the browser console.

[jvilk]

Darn. I need to sleep now.

In my local copy of Doppio, I added the following to the top of the browser java command:

    jvm.dump_state = false
    # XXX: dump-state support
    for i in [0...args.length]
      if args[i] is '-Xdump-state'
        jvm.dump_state = true
        args.splice i, 1
        break

However, it doesn't appear to trigger state dumping. :frowning:

Here's my test class that I used that triggers state dumping on the console but not in the browser:

class Test {
  public static void main(String[] args) {
    throw new Error("Hey");
  }
}

I'll try to do this tomorrow if no one fixes it before me.

[jvilk]

@lavelle: I've added -Xdump-state support to the browser. it doesn't persist the dump, so you'll need to edit the file to copy+paste it locally to use it with the state viewing tool we have. I hope this makes things a little easier.

[jvilk]

Also, as for the Webrick hook, I already created a mechanism for piping stdout through POST messages for the benchmark build. I haven't tested it or checked it recently, so who knows if we've broken it.

Feel free to hack in a handler for POSTing arbitrary files through Webrick. You could detect the browser environment by checking if node is a global variable, and use that to channel any needed debug info through Webrick.

Please let me know if there is anything specifically that you'd like us to do. I'm planning to reserve part of the weekend for Doppio/BrowserFS work. And feel free to use cycles on improving the in-browser development experience; it's basically a requirement for you to be productive on the Swing project, as you've mentioned.

[perimosocordiae]

re: dump-state, it would be super nice to integrate the core viewer that Jez wrote into the frontend.

[jvilk]

Closing this for now, as we have existing issues that encapsulate current filesystem issues (asynchronicity / cloud storage / indexedDB support).