search

jotta / jotta-cli-issues

45 stars 1 forks source link

Question: Is there a way to avoid rehashing when you move files to new dirs? #94

Open ziziz opened 4 years ago

ziziz commented 4 years ago

I have folders that need organizing and renaming on a fairly regular basis. Some with hundreds of gigs in them. And doing so jottad rehashes all that is in a new place or have a new name, every time. This is currently eating up my disk i/o.

Is there a way to avoid this for files that you have already hashed?

Kimbsen commented 4 years ago

Not currently.

We do plan to add the use of filesystem events(inotify, fsevents, readirectorychanges) to jottacli and those should detect most moves and thus avoid the io. Until then the io is unavoidable.

bluet commented 3 years ago

@Kimbsen seems you have it implemented?

I'm experiencing this problem, not sure if it's related. https://github.com/jotta/jotta-cli-issues/issues/148

Kimbsen commented 3 years ago

We have added basic inotify support, but we currently are not detecting renames of large folders/trees yet. Thus this issue is still open.

bluet commented 3 years ago

@Kimbsen I see. About how big of a folder tree will be ignored?

Really hope jottacloud can release source code so we can help on improving client software quality, if you are lacking of development resources. Seems only YOU are the cool guy maintaining Linux client? :-)

Kimbsen commented 3 years ago

Sorry, i was unclear. We currently only use inotify to detect where changes occur to avoid scanning the entire tree on interval. Now we only scan the folder where change occured and any new subtrees. But we still only upload/delete files in backup. We will try to use the logic from sync, where we do currently detect renames relibably, but we are not quite there yet. It can be quite heavy to detect renames in large trees. Also the inputs from fsevents, readdirectorychanges and inotify are not complelety reliable when it comes to renames. So, since backup can contain very large trees we are holding off using the sync implementation there. But we'll get there in the end!

Thanks for the kind words, but we are not planning on releasing the source for jotta-cli. Also, i'm not the only one working on it anymore! @coopersimon joined me a while back :)

bluet commented 3 years ago

Congrats @Kimbsen @coopersimon ! Wish I can help, too.