Allow remote-delta on sync files

gadLinux commented 9 years ago

Hi, I want to fix this improvement ASAP. https://github.com/owncloud/client/issues/179

It relates to the remote-delta implementation on client. But I suppose it must be supported on both sides. I saw that latest version of csync inludes a module for owncloud but it differs on current implementation of client on github.

in fact I saw that's using a special module that's not on the normal csync library that's called httpbf. It seems for me that's sending the whole file as PUT http?

So my question is. If you will want to implement rsync how will you implement it?

I was thinking about clone a module on csync and make it transfer using remote-delta (surely using rsync lib) instead of sending the whole file in chunks.

What do you think? Any suggestions?

Using httpbf. I can detect what's changed and send only the blocks that are changed. But this would mean extra work for nothing. Because librsync already implements this kind of processing we are not going to do ti twice.

Best regards,

jospoortvliet commented 9 years ago

@danimo @dragotin @ogoffart @guruz could you guys give some input here perhaps?

ogoffart commented 9 years ago

Hi @gadLinux , thank you for trying to help on this issue.

The client is no longer using httpf. And is using only very few parts of csync.

You need to contribute in the client repository. (and in the core)

You will find the code that uploads files in src/libsync/propagateupload.cpp here you will find the code that split the file in chunks and uploads them. Similarily, the code that download the files is in src/libsync/propagatedownload.cpp (this is not using chunks)

Good luck :-) But be warned that i think it is a difficult issue. I don't really know if it is possible to do that on top of webdav so you will probably need some kind of extensions to the protocol.

dragotin commented 9 years ago

Some brainstorming:

There needs to be a list of blocks with its according checksums stored on the server for every file. It will be accessible through a special route on the server API.

The client will need to fetch the list once it detected that an existing file has changed and is to be uploaded. Once the blocklist has arrived, the client needs to recalculate the blocklist on the new, changed file. After that, the client can compare the lists and identify the blocks that have a changed checksum. These are the blocks that need to be uploaded.

To upload only parts of files, maybe the HTTP PATCH command (RFC) helps. The server needs to be able to handle this command and reassemble the whole file.

For files that appear new on the client, the client will have to calculate the blocklist and send it along the initial upload to the server to avoid having the server to calculate the list. Also, for each uploaded block, the client will send the new checksum along. The server will either recalculate or invalidate the blocklist for files that were changed on third party storage.

The client needs to be able to deal with the fact that the server can not provide a blocklist for a file and will transparently fall back to uploading the entire file.

The kind of checksum is configurable, and will be adjustable by a server configuration later.

danimo commented 9 years ago

A good starting point for a rsync-like approach using HTTP is zsync, aka client-side rsync. I don't think using librsync is feasible, as we want to stick to HTTP(S) as transport protocol.

gadLinux commented 9 years ago

Hi Olivier,

Thank you for the feedback. I will see what parts are in use on csync. Because I have thinking about two approach. One is to implement the rsync protocol to transfer the file, and the second one is use something like bitttorrent. So it can transfer only the parts needed for it but also leave doors open for multiple client sync against same file in a decentralized way.

I suppose that the first way is easier and better for now.

Let me check the code.

What's the most recent branch of development in both projects? I found there are many and many commits on each branch.

Best regards,

El 22/05/15 a las 12:20, Olivier Goffart escribió:

Hi @gadLinux https://github.com/gadLinux , thank you for trying to help on this issue.

The client is no longer using httpf. And is using only very few parts of csync.

You need to contribute in the client repository. (and in the core)

You will find the code that uploads files in |src/libsync/propagateupload.cpp| here you will find the code that split the file in chunks and uploads them. Similarily, the code that download the files is in |src/libsync/propagatedownload.cpp| (this is not using chunks)

Good luck :-) But be warned that i think it is a difficult issue. I don't really know if it is possible to do that on top of webdav so you will probably need some kind of extensions to the protocol.

— Reply to this email directly or view it on GitHub https://github.com/owncloud/core/issues/16162#issuecomment-104609821.

danimo commented 9 years ago

@gadLinux Please stick with HTTP. Other protocols are not usable in certain scenarios (Port 80 and 443 are always open, others often are not). On top of that, another advantages with the HTTP-based zsync approach is that the server can remain mostly passive, which shifts a lot of load to the client. This improves scalability.

gadLinux commented 9 years ago

Hi,

So it's important to reimplement solution over HTTP. Okay, I hope I can do it this weekend or so.

Did you think about using thrift?

Best regards,

El 25/05/15 a las 11:40, Daniel Molkentin escribió:

@gadLinux https://github.com/gadLinux Please stick with HTTP. Other protocols are not usable in certain scenarios (Port 80 and 443 are always open, others often are not). On top of that, there are a number of other advantages with the zsync approach (with sits on top of HTTP), is that the server can remain mostly passive, which shifts a lot of load to the client. This improves scalability.

— Reply to this email directly or view it on GitHub https://github.com/owncloud/core/issues/16162#issuecomment-105184390.

ogoffart commented 9 years ago

It does not have to be HTTP, but it would be better. (The configuration of an owncloud server has to remain simple). Anyway if it doesn't work we can still fallback to the normal method.

Anyway, as I said, we are slowly moving away from csync, and new code on the client should be written in C++ in the libsync directory.

Bittorrent-like protocol would be something different, that opens its own cans of worms (how do you do authantication or security if you are peer to peer?)

danimo commented 9 years ago

and the second one is use something like bitttorrent. So it can transfer only the parts needed for it but also leave doors open for multiple client sync against same file in a decentralized way.

Do not try to solve two problems at once. The basic idea behind ownCloud is that the server stays in control of the data, so p2p would involve getting a server authorization first. This requires oauth, which is scheduled for one of the next ownCloud server versions. The p2p implementation would then be purely client-side, and not involve the server at all, except for the authorization request.

gadLinux commented 9 years ago

and the second one is use something like bitttorrent. So it can transfer only the parts needed for it but also leave doors open for multiple client sync against same file in a decentralized way.

Do not try to solve two problems at once. The basic idea behind ownCloud is that the server stays in control of the data, so p2p would involve getting a server authorization first. This requires oauth, which is scheduled for one of the next ownCloud server versions. The p2p implementation would then be purely client-side, and not involve the server at all, except for the authorization request.

Yo are right, I agree. Let me take a look to the partial file sync problem.

jospoortvliet commented 9 years ago

@gadLinux perhaps it makes sense to have a Skype call about this to avoid a lot of work going in a direction that won't work ;-)

rullzer commented 9 years ago

Ok so I have been looking into this and reading up on zsync (and by extend rsync). And I think this is something that can be done. Luckily the zsync library is available so I'm modifying that to get a POC working. Since at least the upload use case is not what is was designed for.

I'll hopefully soon put that work on github. So stay tuned. But let me here (quickly) write down how I think this should work.

Assumptions

This only covers the upload scenario (download will follow later, but is easier since it is basically the part zsync tries to solve already)
The server has a copy of file F
The client has a modified copy of file F, lets call it F-new
The server already has the checksum file of F, let call it F-zsync

Setup

diagram1

Server

The server has only 2 tasks. Store the -zsync files. And generate the new file. This is to keep things scalable and not have the server do all the checksumming.

We need to come up with a protocol to tell the server how to assemble the new file. I'll be thinking about that.

We do require space for this to work since you need to copy the original file. And only copy it back once everything is done. Probably locking is also a good idea.

Client

In the normal use case of zsync the client figures out which parts are needed from the server. Now we need to find the parts in common with the server. Which parts needs to go. And which parts need to be added. This requires some new code but the info should all be available.

Since the server is not doing much here all the computational load is shifted to the client. Also in the current setup we trust the clients to generate the zsync files (this is hell in PHP) and send them. From my point of view this is fine since any client can just do a PUT request now anyway.

I hope to have a POC (in C so easily portable to the client) soonish.

CC: @danimo

powerpaul17 commented 9 years ago

I think it should work this way both for the upload and the download because we already know which file is the "newer" one. The client calculates the parts which have to be transferred and the direction is then depending on the sync status.

For the upload there should be something like a HTTP PATCH request.

gadLinux commented 9 years ago

Hi,

Looks great! Finally I didn't have time to go into deep. But if you fail or need help please let me know.

Best regards,

El 30/06/15 a las 09:37, Roeland Douma escribió:

Ok so I have been looking into this and reading up on zsync (and by extend rsync). And I think this is something that can be done. Luckily the zsync library is available so I'm modifying that to get a POC working. Since at least the upload use case is not what is was designed for.

I'll hopefully soon put that work on github. So stay tuned. But let me here (quickly) write down how I think this should work.
  Assumptions
This only covers the upload scenario (download will follow later, but is easier since it is basically the part zsync tries to solve already)

The server has a copy of file F

The client has a modified copy of file F, lets call it F-new

The server already has the checksum file of F, let call it F-zsync

Setup

diagram1 https://cloud.githubusercontent.com/assets/45821/8425844/59f2de74-1f0a-11e5-97ba-74444a586d34.jpg
  Server
The server has only 2 tasks. Store the -zsync files. And generate the new file. This is to keep things scalable and not have the server do all the checksumming.

We need to come up with a protocol to tell the server how to assemble the new file. I'll be thinking about that.

We do require space for this to work since you need to copy the original file. And only copy it back once everything is done. Probably locking is also a good idea.
  Client
In the normal use case of zsync the client figures out which parts are needed from the server. Now we need to find the parts in common with the server. Which parts needs to go. And which parts need to be added. This requires some new code but the info should all be available.

Since the server is not doing much here all the computational load is shifted to the client. Also in the current setup we trust the clients to generate the zsync files (this is hell in PHP) and send them. From my point of view this is fine since any client can just do a PUT request now anyway.

I hope to have a POC (in C so easily portable to the client) soonish.

CC: @danimo https://github.com/danimo

— Reply to this email directly or view it on GitHub https://github.com/owncloud/core/issues/16162#issuecomment-117040384.

jospoortvliet commented 9 years ago

Just curious - does this play into this and might it be interesting to make sure it can work with it?

https://dragotin.wordpress.com/2015/06/22/owncloud-chunking-ng/

DeepDiver1975 commented 9 years ago

@dragotin and myself have been talking about delta-sync as well - there are 2 more blog posts to be expected ...

rullzer commented 9 years ago

@powerpaul17 sure it should work like this both ways. But the download case is easier. Since the zsync file tells you what has to be done. you can download using http range requests. And all is well. Some form of HTTP PATCH would indeed be best for the upload case.

@jospoortvliet this should be independant of that. Altough it could (and should) still make use of chunking if there are a lot of changes.

@DeepDiver1975 awesome looking forward to that.

So my code now "works"... at least on the limited set of tests I've thrown at it. It is inefficient. But works. And hopefully will be documented enough soonish to be available for discussion.

warnerbryce commented 9 years ago

Hello, i'm really interested about testing your code @rullzer
Would you like to share it with me about having more feedback and testing ?

rullzer commented 9 years ago

Sadly no reply yet from the zsync devs (regarding the licence). But here is some code.

A very simple POC owncloud app: https://github.com/rullzer/deltasync A very simple (yet messy) C++ application to upload a file using deltasync: https://github.com/rullzer/deltasync_client

If I have some time soonish I'll update with steps on how to test.

gadLinux commented 9 years ago

Hi,

How do you avoid the deftasync file to be syncronized?

Best regards,

El 30/07/15 a las 10:27, Roeland Douma escribió:

Sadly no reply yet from the zsync devs (regarding the licence). But here is some code.

A very simple POC owncloud app: https://github.com/rullzer/deltasync A very simple (yet messy) C++ application to upload a file using deltasync: https://github.com/rullzer/deltasync_client

If I have some time soonish I'll update with steps on how to test.

— Reply to this email directly or view it on GitHub https://github.com/owncloud/core/issues/16162#issuecomment-126223245.

rullzer commented 9 years ago

O you don't currently. As I said just a POC.

karlitschek commented 9 years ago

@rullzer Very nice. @dragotin @DeepDiver1975 FYI

agowa commented 9 years ago

Wondering if rfc3253 is related to this problem. It describes a WebDAV extension for versioning.

lavrentivs commented 8 years ago

For large amounts of files I believe that something in the line of https://en.wikipedia.org/wiki/Merkle_tree might be a good idea. I had an instance of owncloud running on a really tiny machine (ie, VPS with 512 MB of RAM and 1 core) with a lot of files (160445) and the initiation of the client sync was horribly long. From reading the solutions proposed here I'm afraid that verifying what needs to be updated would still take a long time.

If such a tree structure were used it would suffice to compare two hashes to know that nothing changed. If a single chunk was to changed, it could be found in logarithmic time.

This kind of structure would increase the round-trips so it might need a bit of tweaking. Maybe sending 1k nodes instead of just one would improve it.

gadLinux commented 8 years ago

Hello again. @jospoortvliet I left this because it seems that @rullzer was already implementing something. I'm still interested on it. What's the status of this issue?

@lavrentivs About the merkle tree. We are using this in one of our platforms but maybe this can apply here. I don't know if this would work for a large amount of files or just for big files. I will ask the one tha implemented this here.

Anyway. I saw a lot of improvements in server and client since my last publish but this seems to be still stalled. Should I go or do I wait for @rullzer final solution?

rullzer commented 8 years ago

@gadLinux My approach has a licencing issue wich I'm trying to figure out.

Also implenting this as is is not so trivial since we need proper server side support to do delta uploads/downloads. Which is currently not there. And I prefer to have that eventually in in a nice way instead of hacking it in.

That being said. If you think you have a nice approach please go ahead. :)

gadLinux commented 8 years ago

@rullzer I'm a totally newbie on owncloud specific stuff. So no. I don't have a nice approach. But when I have something cool to implement there's no barriers... :-)

I took a look to your implementation. And got this:

./uploadclient README.md.zsync README.md http://owncloud-dev /files/readme admin admin READING README.md 0 DONE READING []

Started delta sync Violación de segmento (`core' generado)

Maybe doing something wrong...

The licensing issue could be a problem. But I don't understand much about this kind of stuff.

Can I ask why zsync? Why not rsync directly or Unison?

I'm still reviewing architecture. And sincerely I thought you already had the solution for it since it looks nice. But I will tell you if I work on something.

gadLinux commented 8 years ago

@rullzer Let me ask some things. Please. To you and anyone that can point me also to the right place.

Your implementation uses an specific API to upload the zsync. Right? And it's the client the one that manages everything by doing start, move, add, done apis. I suppose that it really does not use owncloud for anything but expose the API and use filesystem utilities. Is correct?

I don't know if there's specific API in owncloud for all this or you are using your own because easier for the POC. Can someone point it for the right way?

gadLinux commented 8 years ago

Just for you to know.

I was taking a look to the whole code. Following directions on this thread. And I can say that my approach should be:

Clients must check first zsync file. Before any operation. Upload/Download doesn't matter. So we must expose this API. Maybe as part as the core of the system. So this file is ignored on server and handled as metadata. In fact. We can create metadata file or use DB.
After blocksum calculation done in client. Compare with zsync file.
Download/Upload differences by chunking blocks over HTTP PATCH.
If zsync file not found proceed as usual.

I was looking at following projects: @rullzer code. Nice. owncloud-client 1.8.0 - 2.1 - Nice since it does chunking already. I realized that client-cmd is there! I will try to modify first here. Then UI client. owncloud-smashbox - Nice to test if server is working well. core 8.2 - Little lost here. Cause I don't know if clients use different api for sync. Or webdav or what... Investigating... The zsync lib approach looks good for me but the license problem doesn't look nice. But anyway the only important think is to do chunking and checksum right. Cause transfer will be done usual way. So why not to get rsync on this part? Is redo the wheel but anyway we cannot use that code on zsync.

Sincerely. Maybe I'm loosing something. Cause I don't see much complicated.

The client is already sending some specific headers on each transfer, so why not to put there the chunk we want to get and all the stuff for r/zsync calculations.

They also have a similar metadata file that's handled in a different way .sys.admin#recall# so let's follow that code and do the same.

I have not much time but it looks this feature is waiting to much time. And owncloud is a great tool. So... I will still keep trying.

rullzer commented 8 years ago

well zsync is essentially rsync but now the client does the expensive calculations instead of the server. Because letting the server do all thos calculations won't scale.

Basically the simple setup I had was

client calculates changes to orignal file using zsync file
clients starts the delta sync with a call to the server
client sends all the moved blocks
clients sends all the new data
client send finish to the server.

However this all sounds simple but we need to take serveral thing into consideration.

We need long time file locking. Or some solution to that. Since we need to have quarantees that the file did not change in between our calculations and upload etc. (POSSIBLE DATA LOSS)
We need to bind the zsync file to original file. As we need to be sure that they are always in sync. ( POSSIBLE DATA LOSS)

And I'm sure there is more than that I forgot. The thing is we need a proper solution that is also maintainable since it is a very complex feature.

gadLinux commented 8 years ago

Ok. Thank you for comments. What I show in the client is that it does not do locking. Just stores a modification time to see if at the end of the process it was modified. I think for now it should be enough to be safe.

For the binding of zsync file. I proposed not just sync this file, but a metadata file for each synchronized file. So we can add more features in the future. I want to see how versioning works at owncloud cause maybe something can be learn.

And I insist about zsync lib I don't think we need it. Just take what we need from rsync and reimplement or use here. Grant credits and license and since all code is published no more cries on that. Let's see. I also wanted to take a look to the Merkle Tree because we use it internally to see that nothing was modified. And can be of help here.

gadLinux commented 8 years ago

Hello,

I want to talk about @lavrentivs comment about Merkle tree. As I said we already implemented this to check if the files under one of our applications are changed and where. Yesterday the guy that did it updated me.

It's useful to know exactly where the file was modified by growing continuous blocks. So it will allow us to send a contiguous block of file that changed. The cons is that you have to build a binary tree of hashes so the calculations grows fast with the number of pieces from the file. He used to use 1024 chunk size. That I think is not good choice for our purpose.

I wonder if it's not a better approach just split the file by N parts (N can be calculated based on size of the object), and just upload what it changed. This way you will end with N+1 hashes for every file (N parts + 1 for the whole file). And not 2 sqrt(N+1)−1 hashes.

I hope I have soon time to address it.

rullzer commented 8 years ago

@gadLinux that will only work in very specific cases. You need a rolling checksum (like rsync (and thus also zsync)) do it. Most likely files that are changed do not have their changes contained to one block.

Assume:

|aaaaa|bbbbb|ccccc|

Now what happens if we have a d to the start of the file. All the blocks change. Which then results in reuploading all parts anyway. To do true delta sync you also want to detect moves within te file etc.

gadLinux commented 8 years ago

You are right. So it doesn't make sense to implement Merkle tree...

TrueOsiris commented 7 years ago

Normally, I wouldn't bother devs on matters as complex as these (unless I know what I'm talking about), but I wonder if this is still being investigated. Usecase: we would for example share a 50GB .vmdk or .vid file between colleagues. As it happens, the dropbox competition also causes files like this to get out-of-sync quite often.

butonic commented 7 years ago

Delta sync is only possible in a very limited scenario: files must be under exclusive ownCloud access. Delta sync with external storages, eg smb would require periodic indexing of the complete storage to update the current chunk hashes.

Even if the above is true inserts or deletes in the middle of a file will trigger a change of all subsequent chunks, unless the client fetches a list of known chink hashes and then tries to find the chunks in the file on disk, burning quite some CPU in the process. This might be solvable by starting to search for chunks in multiple places or prefixing the chunk hash with 32 bytes of actual data to better find chunk boundaries.

IMO chunks should be stored under their hash in a content addressed storage for ownCloud exclusive access... with deduplication and tracking of unneeded chunks at least on ownCloud exclusive storage.

Then we can think about delta sync again.

butonic commented 7 years ago

I did a quick research on how rsync detects the file changes:

Basically one side splits the file into multiple fixed size chunks (500-1000 bytes is reported to be a good value) and then calculates an adler32 as well as md4 for each of them. The other side uses a rolling version of adler32 to find every chunk. Not only at the exact byte offset as the calculated hash, but also on every other byte. The rolling nature makes that a cheap operation. Details here

cc @DeepDiver1975 we should store two checksums for file chunks: adler32 and ... sha256? well, prefix it so we can change it later. This would allow the clients to download a list of adler32 hashes and search for the matching chunk locations in their local version of the file. Then do the delta sync. The clients could also cache the adler32 hashes to detect changes on disk.

DeepDiver1975 commented 7 years ago

cc @DeepDiver1975 we should store two checksums for file chunks: adler32 and ... sha256? well, prefix it so we can change it later. This would allow the clients to download a list of adler32 hashes and search for the matching chunk locations in their local version of the file. Then do the delta sync. The clients could also cache the adler32 hashes to detect changes on disk.

if we would store file chunks: yes - but we dont.

gadLinux commented 7 years ago

I don't know if this makes sense anymore since owncloud 9 has evolved. And I think there are few fork projects. Does it still have interest?

guruz commented 7 years ago

Linking https://central.owncloud.org/t/gsoc-2017-allow-remote-delta-on-file-synchronization/6375/7

gadLinux commented 7 years ago

Hi Ahmed,

Yeahhh... well done. If you finish the upload it will be soooo great....

On 16/10/17 18:56, Ahmed Ammar wrote:

Hi guys, as you can see from the WIP commits referenced above I've started to work on this. A quick summary on the approach is based on what @dragotin https://github.com/dragotin said right at the top of this thread. Adding to that using zsync instead of some custom blacklist implementation. I have only implemented download-mode. To test, some manual labour is needed:

Create some test-files: |dd if=/dev/urandom of=1g.img bs=1024 count=$[10241024]| |dd if=/dev/urandom bs=1024 count=$[1001024] >> 1g.rand.plus100m.img| |dd if=/dev/urandom bs=1024 count=$[1001024] seek=$[1231024] of=1g.rand.mod200m.img conv=notrunc| |dd if=/dev/urandom bs=1024 count=$[1001024] seek=$[5321024] of=1g.rand.mod200m.img conv=notrunc| |dd if=1g.rand.img bs=1024 count=$[2001024] of=1g.rand.mov200m.img oseek=$[4181024] iseek=$[118*1024] conv=notrunc|

After building a local version of zsync, create the .zsync files, e.g: |zsyncmake 1g.rand.mov200m.img|

Upload the data file to oC server: |curl -X PUT --data-binary "@1g.rand.mov200m.img" -u admin:password http://localhost/~aammar/ownCloud/remote.php/webdav/1g.rand.mov200m.img|

Upload the .zsync file to oC: |curl --data-urlencode "content@1g.rand.mod200m.img.zsync" -u admin:password http://localhost/~aammar/core/index.php/apps/files/api/v1/zsync/1g.rand.mod200m.img|

Now that the files are all in place you can test the zsync delta-sync download by doing a simple test. Copy 1g.img (the original random 1G file) to /tmp/test as 1g.rand.mov200m.img and sync: |cp 1g.img /tmp/test/1g.rand.mov200m.img| |owncloudcmd -u admin -p password --exclude /dev/null /tmp/test/ http://localhost/~aammar/ownCloud/remote.php/webdav/|

Example output: https://pastebin.com/uxWyyi8S

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/owncloud/core/issues/16162#issuecomment-336951510, or mute the thread https://github.com/notifications/unsubscribe-auth/AAOO9KxAE3tTt1YSBe1BTP2IRFpTyioaks5ss4q5gaJpZM4ESbAF.

dragotin commented 7 years ago

The zsync lib is licensed under Artistic License 2 which is compatible with GPL according to https://www.gnu.org/licenses/license-list.de.html#GPLCompatibleLicenses

ahmedammar commented 7 years ago

@gadLinux yes that's the plan, was just finishing off download so it works as expected and ensuring the UI is also working as expect. I'll move to upload soon, giving devs a chance to give me any feedback in the meantime.

ahmedammar commented 7 years ago

@dragotin so that's all good from the licensing side?

dragotin commented 7 years ago

@ahmedammar yes, looks like, but @hodyroff has the final take

hodyroff commented 7 years ago

Thanks a lot for this effort. Yes, licensing is all good. zsync will be a new 3rd party lib under the Artistic 2 license (which is BSD-style = non copyleft).

hodyroff commented 7 years ago

Before its merged it will need proper testing effort of course, maybe it can be done in a way that it can be switched on/off in the gui and we can promote it as a technology preview for a start? Understanding that the bounty depends on merging ...

ahmedammar commented 7 years ago

@hodyroff absolutely! will add that 'gui option' when it's complete, currently still have to do upload mode.

felixboehm commented 7 years ago

Let me add some infos related to delta sync:

ahmedammar commented 7 years ago

OK, upload path redone and is working now, had to make some changes to zsync (ahmedammar/zsync@6a04a908481427773814d53ee77a7f1da1afa3d9) to supported what was needed for uploading (instead of the zsync code's expectation of downloading remote blocks).

I am re-uploading even moved blocks for now, just to make implementation easier, but that's an area that can be revisited in the future.

Core implementation is now complete, code re-factoring in certain areas is next step and some small features like UI option for enable/disable and file-size thresholds.

Please review and test.

N.B. ~~Only tested that it compiles on a macOS based machine.~~ Do not use with any critical data yet. 🔥

owncloud / core