apache / opendal

Apache OpenDAL: One Layer, All Storage.
https://opendal.apache.org
Apache License 2.0
3.45k stars 482 forks source link

new feature: direct IO support in fs #5125

Open sundy-li opened 2 months ago

sundy-li commented 2 months ago

Feature Description

Support direct io mode in fs backend

Problem and Solution

By using direct io, we can directly write/read into file system without through page-cache, which is useful in DBMS to control the memory.

Additional Context

No response

Are you willing to contribute to the development of this feature?

Lzzzzzt commented 3 weeks ago

I'd like to implement this.

But should this feature be a directory level or file level?

I mean the direct-io flag should apply to the whole root dir or the specific file? If the file, it seems currently no way to pass the flag, should add it to OpRead/OpWrite/...?

Also, the read/write file using the internal FsReader/FsWriter which have internal buffer, is this a little bit conflict?

Xuanwo commented 3 weeks ago

But should this feature be a directory level or file level?

I believe it should at Service level, users should specify this flag while configuring this backend.

Also, the read/write file using the internal FsReader/FsWriter which have internal buffer, is this a little bit conflict?

It's still meaningful since we can avoid in-kernel memory cache.

Lzzzzzt commented 3 weeks ago

I believe it should at Service level, users should specify this flag while configuring this backend.

It's still meaningful since we can avoid in-kernel memory cache.

OK, got it.