Growing files without re-uploading

[havardgulldahl]

JottaCloud supports partial uploads using the HTTP Content-Range header.

jottalib needs to add support for this to fix #10 and hopefully #8 as well.

[havardgulldahl]

Some clarification:

We can upload files in a stream ( #18 ), and hopefully resume uploads ( #22 ), so this bug is about uploading arbitrary chunks of data to an existing file.

If someone is able to do this with the official client, please drop the HTTP log (headers only) here.

[jkaberg]

@havardgulldahl I'm looking into this now, any suggestions on how to simulate this? I tried shutting down the client while uploading but that restarts the entire upload from scratch (when starting the client again)

EDIT: https://gist.github.com/jkaberg/f387a9de553f374a202c (let me know if this is wrong)

EDIT2: One more, this one is a lot smaller aswell https://gist.github.com/jkaberg/a20b80291eb041801540 . This time I paused the upload, exited the client, started up the client and unpaused the client which should have the effect you desire

[nuth]

UPDATE: I see somebody already did this...

I've been writing my own client in java (lost the code, but still). I was thinking of splitting up the file client side and then uploading it chunks. Of course it wouldn't help with re uploading only the changed chunk later (maybe somebody can think of something clever here), but it would help in terms of md5ing large files (and I believe memory usage) and I suppose you could even upload in parallel.

In addition you won't have to upload the whole file again, if corrupted data is transferred.

Downside is this client would have to be used to automatically merge it back together. Still worth some thoughts.

[havardgulldahl]

Don't you have a pause button? Or in the context menu from the tray?

Sendt fra min mobiltelefon

Den 28. mai 2015 kl. 09.19 skrev Joel Kåberg notifications@github.com:

@havardgulldahl I'm looking into this now, any suggestions on how to simulate this? I tried shutting down the client while uploading but that restarts the entire upload from scratch

— Reply to this email directly or view it on GitHub.

[jkaberg]

@havardgulldahl See my post above, I revisited it with 2 logs

[havardgulldahl]

Thanks, @jkaberg , these logs clearly show that resuming upload is doable, so good news for #22

However, for this bug, which is about growing an existing file without re-uploading (i.e. appending data at the end of the file), this doesn't get any further. I'm not sure the Jottacloud API needs to support it, so it may not be possible.

[jkaberg]

@havardgulldahl Not sure on how I can emulate this? The 2nd logs in my previous post is when I use pause/resume in the Jotta client

[havardgulldahl]

@jkaberg Yeah, I don't know if their official client would ever need to do this. As long as they transfer file by file, and reupload the whole file if it's changed, it seems we're out of luck. What we're looking for here, is some support for patching a file by pushing diffs.

Could you capture a log from the websocket chatter, maybe there is something interesting there?

[jkaberg]

@havardgulldahl The websocket only ever gets upgraded, nothing more is happening on that part (atleast of what Fiddler can decipher, and afaik it supports WS fully)

Seems we're out of luck

[havardgulldahl]

Removing milestone. I no longer think this is possible to solve, because this is my current understanding of the protocol:

The md5 checksum is how the JottaCloud servers differentiates between file revisions. So you can't keep growing a file, because with each iteration the file would have a new checksum, and thus JottaCloud would see a new file every time.