[Remote: Hubic] Failed to copy: Object Corrupted on segmented files

[hfat]

It seems rclone can't work with segmented files uploaded to the service with Hubic's official client. These files are associated with many smaller files located in a directory named with a "_segments" postfix. Here's the end of a --dump-headers copy command (which works with non-segmented files uploaded with hubicfuse):

... 2016/03/16 12:24:49 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 2016/03/16 12:24:49 HTTP RESPONSE 2016/03/16 12:24:49 HTTP/1.1 200 OK Connection: close Content-Length: 13430784 Accept-Ranges: bytes Content-Type: application/octet-stream Date: Wed, 16 Mar 2016 11:24:41 GMT Etag: "ac951f76e3d45d2edbca0f65f61c2a24" Last-Modified: Mon, 24 Aug 2015 08:10:45 GMT X-Object-Meta-Hubiclocalcreationdate: 2015-08-24T07:41:33 X-Object-Meta-Hubiclocallastmodified: 2015-08-24T07:41:33 X-Static-Large-Object: True X-Timestamp: 1440403844.31143 X-Trans-Id: txcfcd403220784ba88ed31-0056e94272

2016/03/16 12:24:49 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 2016/03/16 12:24:50 Transferred: 29548067 Bytes ( 368.20 kByte/s) Errors: 0 Checks: 0 Transferred: 2 Elapsed time: 1m18.3s Transferring:

• i150821.tar.bz2.enc: 20% done. avg: 2570.2, cur: 2573.0 kByte/s. ETA: 4s

2016/03/16 12:25:27 i150821.tar.bz2.enc: Failed to copy: Object Corrupted 2016/03/16 12:25:27 Attempt 3/3 failed with 1 errors

Transferred: 40292352 Bytes ( 341.74 kByte/s) Errors: 1 Checks: 0 Transferred: 3 Elapsed time: 1m55.1s

version/OS information:

rclone --version

rclone v1.28

cat /etc/centos-release

CentOS release 6.7 (Final)

cat /proc/version

Linux version 2.6.32-573.18.1.el6.x86_64 (mockbuild@c6b8.bsys.dev.centos.org) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) ) #1 SMP Tue Feb 9 22:46:17 UTC 2016

[ncw]

This is because rclone is checking the md5sum, but ignoring the X-Static-Large-Object: True flag. rclone (or actually the swift library) would normally expect to see an X-Object-Manifest header on a large object.

I made an issue here about it ncw/swift#67

[ncw]

I've fixed this and put a beta for you to try here which should fix the problem above.

http://pub.rclone.org/v1.28-9-g9dccf91%CE%B2/

Could you also try rclone md5sum on a segmented file? This should return a blank for any segmented objects.

Thanks

Nick

[hfat]

No, it keeps returning the wrong hash and the downloads keep failing:

2016/03/16 19:08:15 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2016/03/16 19:08:15 HTTP REQUEST 2016/03/16 19:08:15 GET /1.0/account/credentials HTTP/1.1 Host: api.hubic.com User-Agent: rclone/v1.28 Authorization: Bearer ... Accept-Encoding: gzip

2016/03/16 19:08:15 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2016/03/16 19:08:15 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 2016/03/16 19:08:15 HTTP RESPONSE 2016/03/16 19:08:15 HTTP/1.1 200 OK Transfer-Encoding: chunked Access-Control-Allow-Origin: * Connection: keep-alive Content-Type: application/json; charset=utf8 Date: Wed, 16 Mar 2016 18:08:06 GMT Server: nginx/1.4.7 Set-Cookie: api.hubic.com=...

2016/03/16 19:08:15 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 2016/03/16 19:08:15 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2016/03/16 19:08:15 HTTP REQUEST 2016/03/16 19:08:15 HEAD /v1/AUTH_... HTTP/1.1 Host: lb9911.hubic.ovh.net User-Agent: rclone/v1.28 X-Auth-Token: ...

2016/03/16 19:08:15 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2016/03/16 19:08:26 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 2016/03/16 19:08:26 HTTP RESPONSE 2016/03/16 19:08:26 HTTP/1.1 200 OK Connection: close Content-Length: 13430784 Accept-Ranges: bytes Content-Type: application/octet-stream Date: Wed, 16 Mar 2016 18:08:17 GMT Etag: "ac951f76e3d45d2edbca0f65f61c2a24" Last-Modified: Mon, 24 Aug 2015 08:10:45 GMT X-Object-Meta-Hubiclocalcreationdate: 2015-08-24T07:41:33 X-Object-Meta-Hubiclocallastmodified: 2015-08-24T07:41:33 X-Static-Large-Object: True X-Timestamp: 1440403844.31143 X-Trans-Id: txfe4047ecfa01470daa5b3-0056e9a106

2016/03/16 19:08:26 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 2016/03/16 19:08:26 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2016/03/16 19:08:26 HTTP REQUEST 2016/03/16 19:08:26 HEAD /v1/AUTH_... HTTP/1.1 Host: lb9911.hubic.ovh.net User-Agent: rclone/v1.28 X-Auth-Token: ...

2016/03/16 19:08:26 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2016/03/16 19:08:39 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 2016/03/16 19:08:39 HTTP RESPONSE 2016/03/16 19:08:39 HTTP/1.1 200 OK Connection: close Content-Length: 13430784 Accept-Ranges: bytes Content-Type: application/octet-stream Date: Wed, 16 Mar 2016 18:08:29 GMT Etag: "ac951f76e3d45d2edbca0f65f61c2a24" Last-Modified: Mon, 24 Aug 2015 08:10:45 GMT X-Object-Meta-Hubiclocalcreationdate: 2015-08-24T07:41:33 X-Object-Meta-Hubiclocallastmodified: 2015-08-24T07:41:33 X-Static-Large-Object: True X-Timestamp: 1440403844.31143 X-Trans-Id: txf8acafb29a2945bca3daf-0056e9a111

2016/03/16 19:08:39 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< "ac951f76e3d45d2edbca0f65f61c2a24" i150821.tar.bz2.enc

Transferred: 0 Bytes ( 0.00 kByte/s) Errors: 0 Checks: 1 Transferred: 0 Elapsed time: 23.9s

With the beta, the error is different though:

2016/03/16 19:11:20 i150821.tar.bz2.enc: Corrupted on transfer: MD5 hash differ "\"ac951f76e3d45d2edbca0f65f61c2a24\"" vs "4d5563962d5de13efe3bdeee43cee647" 2016/03/16 19:11:20 Attempt 3/3 failed with 1 errors

[ncw]

I'll try again in a moment - it should be easy to fix.

Thanks

[ncw]

Here is another beta for you to try

http://pub.rclone.org/v1.28-10-gbd27473%CE%B2/

Hopefully that will fix it!

Thank for testing

Nick

[hfat]

Yeah, the download isn't aborted with that beta. And no md5sum is returned by rclone md5sum. Note that I haven't tried with very large (>5G) files because I've split all the very large files I've uploaded.

[ncw]

Thanks for testing the beta.

Nick