muxinc / upchunk

Uploads Chunks! Takes big files, splits them up, then uploads each one with care (and PUT requests).
MIT License
340 stars 46 forks source link

Use Range response header to ensure all bytes were received. #125

Closed skidder closed 8 months ago

skidder commented 8 months ago

This PR resolves issue #124 by ensuring all intermediate chunks in uploads are fully received. It introduces a check for the Range header in HTTP 308 responses, enabling the detection and re-upload of partially transmitted chunks. This fix specifically targets the completeness of intermediate chunk uploads, and does not address the need for a verification of the HTTP response status code for the last chunk upload.

skidder commented 8 months ago

Also, it looks like the Range response header represents all bytes sent successfully so far, not just the bytes sent in the present request. So it's more appropriate to just compare the end-byte of the Range to the end-byte sent with the chunk. I'll update the PR with that approach.

Example

Uploading a file that's 9116371 with a 1048576 chunk-size produced the following request/response values:

Chunk 1:

Request

Content-Range: bytes 0-1048575/9116371
Content-Length: 1048576

Response

HTTP/2 308 
range: bytes=0-1048575
x-range-md5: 65a98e22d8c5da280684d5ba87930b0b

Chunk 2:

Request

Content-Range: bytes 1048576-3145727/9116371
Content-Length: 2097152

Response

HTTP/2 308 
range: bytes=0-3145727
x-range-md5: d9396997b399232c5d3f32fc3fbb373c

Chunk 3:

Request

Content-Range: bytes 3145728-7340031/9116371
Content-Length: 4194304

Response

HTTP/2 308 
range: bytes=0-7340031
x-range-md5: 89b94c380e2346f9deb9606973efbe0c

Chunk 4:

Request

Content-Range: bytes 7340032-9116370/9116371
Content-Length: 1776339

Response

HTTP/2 200 
etag: "7e4ac71428f2449dccfff8bd2a3fe52a"
x-goog-stored-content-length: 9116371
x-goog-stored-content-encoding: identity