feat: Add a userspace filesystem backed by OpenDAL

[Inokinoki]

Personally, I was working on a prototype of userspace filesystem (DalFs) based on fuser. This fs can leverage the data access capabilities of OpenDAL, to mount a OpenDAL service as a local filesystem.

It is a standalone application. The process keeps blocking, which receives the filesystem access request from the kernel, and replies with the file/dir information or content from OpenDAL. It consists of a simple inode management module from Netfuse project, and the actual callbacks implemented in fuser Filesystem trait.

It is glad for me to improve it and try to integrate it with upstream OpenDAL. There are several feasibilities:

1. Integrate the application as a subcommand of oli, which could act like the executable of DalFs (blocking and serving the kernel).
2. Integrate it as another executable, that can be integrated to the distributions and mounted using mount subcommand (promising).

As for the functionality, DalFs now is able to provide basic filesystem operations. However, there are still a few improvements, like IO performance, permission (ideally can cooperate with OpenDAL by adding some ACLs from the services). We can create more sub-issues further.

[Xuanwo]

Maybe we can start via kicking off the project layout first?

[Xuanwo]

I'm guessing we can add a fuser-opendal lib under integrations?

[Inokinoki]

I'm guessing we can add a fuser-opendal lib under integrations?

IMO, the fuse impl is not general enough. It only contains the fs operation-specific code, such as: "when user want to move a file, what should we do".

I am not a big fan to make it a lib, instead, it is much like an application, using opendal as a lib, and acting as a bridge between the kernel and opendal. WDYT?

[Xuanwo]

I am not a big fan to make it a lib, instead, it is much like an application, using opendal as a lib, and acting as a bridge between the kernel and opendal. WDYT?

Let me make my statement more clear. My plan is to have the following parts:

• fuser-opendal lib in integrations which serve as a lib that implements fuser::Filesystem based on opendal.
• ofs in bin which is a binary that can be used to mount any storage services.
• Also, oli can reuse fuser-opendal inside to provide oli mount but I'm not sure if we should do this.
• And other application who want gain the same feature can use fuser-opendal directly.

[Xuanwo]

• fuser-opendal lib in integrations which serve as a lib that implements fuser::Filesystem based on opendal.

I want this since I'm guessing it will make our test it more easier.

[Xuanwo]

As the owner of the code, you have the ultimate authority to decide its implementation. If you choose to only work on bin/ofs, that's fine by me.

[Inokinoki]

I am not a big fan to make it a lib, instead, it is much like an application, using opendal as a lib, and acting as a bridge between the kernel and opendal. WDYT?

Let me make my statement more clear. My plan is to have the following parts:

• fuser-opendal lib in integrations which serve as a lib that implements fuser::Filesystem based on opendal.
• ofs in bin which is a binary that can be used to mount any storage services.
• Also, oli can reuse fuser-opendal inside to provide oli mount but I'm not sure if we should do this.
• And other application who want gain the same feature can use fuser-opendal directly.

In terms of testing, I totally agree that we need to test the modules separately. However, the fuser::Filesystem is deeply coupled with the cli (maybe the future ofs) and with the fuser.

I think that we can have a better abstraction to decouple it:

1. fuser::Filesystem provides full functional callbacks for fuser, which calls fs manipulation functions from an abstraction layer and utility functions to manipulate inode/path.
2. the implementation of the layer can be provided in a lib (maybe opendal-fs-bridge? also for the future ACL<->fs permission mapping);
3. inode and path name maintenance could be in another lib opendal-fs-util.

In this way, other promising libs such as opendal-kio could also leverage 2 maybe and 3 to provide the similar functionality. For the others who wants to implement another fuse backed by OpenDAL, they are also eligible to the libraries other than fuser. I had the concern that DalFs was like a toy project and relied too much on fuser.

WDYT?

[Xuanwo]

• fuser::Filesystem provides full functional callbacks for fuser, which calls fs manipulation functions from an abstraction layer and utility functions to manipulate inode/path.
• the implementation of the layer can be provided in a lib (maybe opendal-fs-bridge? also for the future ACL<->fs permission mapping);
• inode and path name maintenance could be in another lib opendal-fs-util.

I feel like we are going to far by adding too much crates. How about starting with the simplest setup: a project called ofs under bin which produces a binary. We can decide whether or not to split it in the future.

2. the future ACL<->fs permission mapping);

Personally, I don't care about this feature. Since most storage backend don't respect this. We can expect that users always mount fs as allow_others. And I can also imagine that users use ofs just for data reading and writing like mountpoint-s3.

However, this is the design details. We can discuss about this in the future.

I had the concern that DalFs was like a toy project and relied too much on fuser.

That's OK. We don't need to put too much burden on ourselves for a project that just started. We can improve this part later on.

[Inokinoki]

Sry, I am too busy to work on this. Will have time working on it across Xmas and new year ;)

@ho-229 is also an active developer contributing to DafFs. We will keep working on this.

[Xuanwo]

Is @ho-229 interested in starting a layout for the project in OpenDAL?

[ho-229]

Yes I am glad to

[bokket]

Hi I am also interested in.Can we break down the tasks so that someone can be assigned?@Inokinoki

[Inokinoki]

Hi I am also interested in.Can we break down the tasks so that someone can be assigned?@Inokinoki

I appreciate and it would be great! I will try to see what to do.

[oowl]

Actually, I have mentioned this feature before https://github.com/apache/incubator-opendal/issues/2625. And I am happy to contribute my time to help us to integrate it into our repo if you guys need some help.

[Xuanwo]

Let's get started! Feel free to submit a PR to add an empty rust project in the bin/ofs directory. We can start a tracking issues after than.

[Xuanwo]

Work tracked at https://github.com/apache/incubator-opendal/issues/3782