Export necessary interfaces to enable client-implemented FS's

[vadimkantorov]

Hi, I'm looking to teach BrowserFS understand JupyterLab's API for fetching files from the server (at the end of the day I'd like to mount this FS to Emscripten's FS).

JLab exposes an API endpoint that lets you request:

• a file contents by sending a GET to http://localhost:8888/api/contents/FILE_PATH.json?format=text&type=file&content=1&1505946940755 with a response similar to:

  {
   "name": "package.json", 
   "path": "package.json", 
   "last_modified": "2017-09-20T21:56:34.420983+00:00", 
    "created": "2017-09-20T21:54:40.952607+00:00",
    "content": "FILE_CONTENT", 
    "format": "text", 
    "mimetype": "text/plain", 
    "writable": true, 
    "type": "file"
  }

• a directory listing by sending a GET to http://localhost:8888/api/contents/DIRECTORY_PATH?content=1&1505946940778 with a response similar to:

  {
  "name": "DIRECTORY_NAME", 
  "path": "DIRECTORY_PATH", 
  "last_modified": "2017-09-20T22:29:06.138319+00:00", 
  "created": "2017-09-16T20:10:03.288946+00:00", 
  "content": [
      {"name": "FILE_NAME_1", "path": "FILE_PATH_1", "last_modified": "2017-09-20T22:26:56.788539+00:00", "created": "2017-09-16T20:10:03.293950+00:00", "content": null, "format": null, "mimetype": "text/plain", "writable": true, "type": "file"}, 
      {"name": "FILE_NAME_2", "path": "FILE_PATH_2", "last_modified": "2017-09-20T22:26:56.785554+00:00", "created": "2017-09-16T20:10:03.289965+00:00", "content": null, "format": null, "mimetype": "application/javascript", "writable": true, "type": "file"}
  ], 
  "format": "json", 
  "mimetype": null, 
  "writable": true, 
  "type": "directory"
  }

So far my understanding is that BrowserFS (HTTPRequest in particular) lets you specify a full root directory tree at FS creation time and does not allow to provide a callback to process the server response to extract the file contents. Is my understanding correct?

Would you have an advice how to expose JLab remote file system to BrowserFS in the easiest way? Which existing backend would be the right base to build upon?

As I am coding a JLab extension, I have access to JLab's Contents.IManager directly. Should I rather implement the FileSystem interface directly like the Dropbox backend does?

Thank you!

[vadimkantorov]

If I go the latter path following Dropbox backend example, is there a way to access/extend BaseFileSystem (I am consuming BrowserFS from my own TypeScript code) which seems not exported from browserfs.ts?

[jvilk]

Hey @vadimkantorov,

I think implementing a FileSystem directly is the right way to go. I am probably not exporting all of the types / classes you need.

I'll accept a PR that exports BaseFileSystem from browserfs.ts. Perhaps we should have a top-level BrowserFS.Abstract property that contains all of the abstract classes needed in an external codebase (BaseFileSystem, PreloadFile, BaseFile, etc)?

[vadimkantorov]

I've clone this repo, ran npm install and got an error:

src/generic/key_value_filesystem.ts(936,34): error TS2345: Argument of type '{ [name: string]: string; } | undefined' is not assignable to parameter of type '{}'.
  Type 'undefined' is not assignable to type '{}'.

Is there something trivial I missed? (new to TypeScript and modern JS)

[jvilk]

It's possible NPM pulled in a newer version of TypeScript that doesn't work. Have you tried using yarn install instead? That will pull in the exact version of each package that is known to work.

[vadimkantorov]

yarn install has worked, then I changed directory to my project dir and ran npm install ../browserfs which worked as well. But then TypeScript fails to build my code with:

node_modules/browserfs/dist/node/backend/Dropbox.d.ts(16,13): error TS2503: Cannot find namespace 'DropboxTypes'.
node_modules/browserfs/dist/node/core/util.d.ts(40,73): error TS2304: Cannot find name 'SharedArrayBuffer'.
node_modules/browserfs/dist/node/core/util.d.ts(63,62): error TS2304: Cannot find name 'SharedArrayBuffer'.

npm install @types/dropboxjs --save-dev didn't help

[jvilk]

You're using BrowserFS from master, which uses the latest Dropbox JS SDK. The latest Dropbox JS SDK ships types directly (which I wrote myself! :-) ), so you need to do npm install dropbox --save-dev. @types/dropboxjs contains types for the old SDK (which the stable version of BrowserFS currently uses), so you should remove that dependency to avoid conflicts.

SharedArrayBuffer is a new JS feature, so TypeScript only ships it in the es2017 lib I believe. You can modify your lib property in tsconfig.json like I did to tell TypeScript to enable typings for experimental browser features, or I think adding a *.d.ts to your project with the following would also work (I haven't tested; just postulating):

declare type SharedArrayBuffer = ArrayBuffer;

[vadimkantorov]

Thanks for the suggestion. ES2017 helped for the SharedArrayBuffer!

npm install dropbox --save-dev unfortunately didn't help for the Dropbox error (I've also removed old package from node_modules manually).

Btw the Dropbox npm package pulls another 20 packages as dependencies. It would be cool if the Dropbox backend / dependency was optional.

[vadimkantorov]

I checked the source of dropbox sdk (npm package dropbox), it contains export = DropboxTypes.Dropbox (if I understand well, underlying DropboxTypes isn't available for import at all).

The backend, dropbox_bridge somehow use DropboxTypes without it being explicitly imported. Can this work? Maybe the Dropbox sdk has been updated in the meanwhile?

[james-pre]

Going off d2aad42 (the latest commit at the time of writing), all of the necessary classes, variables, and types are exported.