search

yadayada / acd_cli

An unmaintained command line interface and FUSE filesystem for Amazon (Cloud) Drive
Other
1.35k stars 167 forks source link

Request: Caching algorithm. #199

Open bobloadmire opened 8 years ago

bobloadmire commented 8 years ago

It would be nice to dedicate a few hundred GBs of local storage for the most recently accessed or put data. This would also reduce the amount of bandwidth need to access files since they wouldn't need to be pulled as often. Similar software, Stablebit CloudDrive has this feature, it is very useful. You can set up 500GB of the latest data locally and have lower latency and reduced bandwidth usage. Out of cache data is accessed normally.

acornspaul commented 8 years ago

:+1:

hazcod commented 8 years ago

Would be nice if directory listings were included!

bobloadmire commented 8 years ago

Stablebit caches Directory listings and the files Metadata. This allows programs like Plex to index the folder structure and files as if they were local.

msh100 commented 8 years ago

@bobloadmire acd_cli currently stores all the node data offline.

ratherDashing commented 8 years ago

@msh100 I have about 20,000 music tracks and it takes roughly 6-8 hours to index. off my local hard drive it takes about 30 minutes. not sure why the difference then

msh100 commented 8 years ago

@ratherDashing it's simply less efficient as your host file system. I don't think that's crazy considering how vastly different they are.

That's why this issue exists, to try and optimise it a little bit.

marcelowa commented 8 years ago

Just wanted to request the same feature :+1:

yadayada commented 8 years ago

I will look into whether it's possible to simply use cachfilesd/FS-Cache for (read) caching.

baro77 commented 8 years ago

I think write caching could be a way to mimic non-sequential writes ... could it be?

xelra commented 7 years ago

I've been trying to get non-sequential writes working with the FUSE-mount for some time. The solutions I looked into were bcache and EnhanceIO. Both those solutions are originally intended to have an SSD serve as cache for an HDD. Unfortunately they're made to work with block level devices.

They implement write-back caching, which is exactly what is needed here, just on a higher level in the fs. The thing about write-back caching is that all writes go to the cache (local) and are then written asynchronuously and sequentially to the backend (acd fuse).

Something like this should make the fuse implementation fast (local cache) and working with any type of read or write.

xelra commented 7 years ago

I've just seen that this is already heavily discussed in this PR #374.