web ide: Support for File System Access APIs as a Storage Provider?

abose commented 3 years ago

Hi all,

We are working on porting brackets text editor to be natively browser-based as part of the phoenix project. Filer looks like a good fit for our use case. But it does not seem to support the native File System Access API yet.

Does filer support the new File System Access API? I checked through the issues and pull requests and could not find any issues that add support for access to the native file system via the API.
Is it under consideration/any opinions on including it as a filer backend?
Are there any use cases or performance considerations we should keep in mind while using filer for projects with a large number of files and intensive file operations?

Thanks in advance, Arun.

humphd commented 3 years ago

Hi @abose,

Nice to hear from you. Funnily enough the reason I got involved with Filer in 2015 was because my team also ported Brackets to the web as part of Mozilla's Thimble editor. We went on to use Filer+Brackets as part of Thimble with Mozilla for many years, and it worked great. I've also used filer as a backing filesystem for a Linux VM running in the browser, and it works great there too, see https://humphd.github.io/browser-shell/.

At this point, Filer is stable and not actively being extended. We haven't added File System Access API support, no. It could be done by someone, but I'm not going to have time to do it this year.

If you're interested in hacking on/with Filer, I can answer questions and do reviews.

abose commented 3 years ago

Hi, @humphd Thanks for the quick response! I was reading thimble and closely followed the bramble project a while back. Bramble gave us confidence that phoenix could be done as it was already done once before and bramble also contributed changes to brackets at the time. Hopefully, we can port in the live preview implementation from thimble to phoenix.

Will evaluate if we can contribute the feature from our side to filer. would help to get clarity on a few questions:

Can filer work with indexedDB in a separate web worker.
Can a web worker and main thread simultaneously access the same indexed DB source. We are planning to offload certain code intellisence/background-related file access functions(mostly read-only) into separate web workers. Will it create problems with filers current architecture?
is there any performance issues or throttling issues when a large number of file operations are done with indexedDB backend.
Finally, What would be the high-level tasks that should be done to add the native fs backend to filer? How complex is the task to add a new backend?

Thanks again for the help!

humphd commented 3 years ago

1. Can filer work with indexedDB in a separate web worker.

Yes, you can run it in workers. I'm running it in a Service Worker with https://github.com/humphd/nohost, but it works fine in any type of worker.

2. Can a web worker and main thread simultaneously access the same indexed DB source. We are planning to offload certain code intellisence/background-related file access functions(mostly read-only) into separate web workers. Will it create problems with filers current architecture?

Yes, it will work fine. This isn't specific to Filer, just how indexeddb works in general. It's easy to work with the same DB across widows or worker contexts.

3. is there any performance issues or throttling issues when a large number of file operations are done with indexedDB backend.

I haven't run into major issues, but as always, it will depend on what you mean by "large."

4. Finally, What would be the high-level tasks that should be done to add the native fs backend to filer? How complex is the task to add a new backend?

I haven't read the spec for it, so I have no idea.

abose commented 3 years ago

Cool, looks like this is just the lib that is needed. We will contribute to adding this feature in filer. Hoping to get support for our dev https://github.com/tapanmanu to implement the change :)

abose commented 2 years ago

Being implemented in fork, for updates: https://github.com/aicore/filer/projects/1

abose commented 2 years ago

@humphd Update on the current state of things. Have implemented the base support as part of filer. Still, a lot more work to be done. [unit tests, API completeness]

All the code changes done till now can be found here.
FS access apis does not have rename/move support yet. raised an issue in WICG/FS access for the same.

API spec

The entry point is to call fs.mountNativeFolder() API which will mount the file system to an existing filer context under the /mnt/<unique folder name> folder. The changes will be broadcasted to all filer instances in the domain(including web workers) and persisted across browser tab closes(Though it will prompt for permissions as required).
After that all existing filer fs/shell APIs should work as usual.

Trying it out

Phoenix editor has been integrated with the new Filer APIs and we have got partial functionalities restored with native file access in chrome/edge. Try out in this link.

use file> open folder menu in phoenix to open a folder in the local machine.

Queries

Would filer like to get the upstream merge only after we have all changes done and stabilized in our fork?
There are significant feature gaps in the current fs access APIs, so most APIs except the basic RW APIs will throw Errors.ENOSYS('Filer native fs function not yet supported.'); . We can only add support incrementally to these new APIs as the standard evolves.

abose commented 2 years ago

All changes that is required to add fs access apis moved to phoenix for better integration. Please see https://github.com/aicore/phoenix/tree/main/src/phoenix for implementation details if anyone is interested. Closing issue as phoenix used a different way to implement fs native access compatibility with nodejs fs apis.

filerjs / filer