Closed nkemnitz closed 3 years ago
Yeah, somehow those files must have been uploaded as multipart... not sure why. ETag calculation is a bit annoying: the number after the dash is the number of multiparts for that object. With that and the size of the full object, one can guess the size of a each multipart and calculate the MD5 for each part + the final combined MD5.
That's odd, if these were created using CloudFiles. I wasn't aware it could even do that. Did you use s3cli or gsutil? Those can sometimes automatically switch to multipart.
I am fairly sure that I did those with CloudVolume - it's the sharded version of Flywire. But I had to rework some files a while ago - maybe I used gsutil there, don't remember.
The real issue is that I can't read them with CloudFiles, though. For now, I set etag = None
in the interfaces.py for those etags that contain the multipart suffix, so that my transfer finishes.
I'm surprised CF can download the multipart files without special logic. I was considering the multi-part logic in #7. I guess downloads are a lot easier.
In any case, this is a good feature. I'll look into it some more. Thanks for the test case and info.
Some of the files on the Cloudian (and probably other S3 backends) seem to have some kind of multipart identifier attached to the ETag - which is weird by itself, because I thought I didn't upload them as multipart... Anyway, the suffix confuses the checksum calculation.
Example:
has a checksum of
cd8d2616dfa6cc80a06a846d3b3f6f30-14
. ~Ignoring the-14
part results matches the expected checksum.~ Was wrong about that.