search

cryptomator / fuse-cloud-access-adapter

FUSE access to cloud-access-java
GNU Affero General Public License v3.0
2 stars 1 forks source link

Performance: Cancel superseeded upload tasks #7

Closed infeo closed 3 years ago

infeo commented 4 years ago

Currently, for every scheduled upload a new copy of the to-be-uploaded file is made (by generaring a UUID) and then put into a map with a long value as key.

This can lead to the consumption of a lot of space, when a lot of such small improvements updates are made by the sequence of open(), write() and release(). The amount of space can be reduced by storing the upload tasks by their (unique) cloudpath, storing at most one temp file for each cloud path and if the file is locally updated and then release cancel all "old" upload task and only upload the newest version of the file.

infeo commented 3 years ago

With 89ecad7077a8dfee926d6abca6d090b6a96ac2a4 the following concept is implemented:

Files are uploaded to a temporary directory and afterwards moved to their destined location. During the upload the path can still change, and a path lock needs to be acquired to perform the final move() operation.

Additionally, the class OpenFile which represents a file, has now different states. If the such an OpenFile is uploaded it changes its state to uploading. If during an upload, there is a change to the file, it changes to the state "needs reuploading", which will trigger another upload, if the first one is finished. An upload is only canceld in a case of an unlink() or rmdir() call. Otherwise not, and any change only changes the state of the OpenFile.

Remark: Canceling an upload does not stop the actual upload (can be seen by monitoring the network traffic).