aws / aws-cli

Universal Command Line Interface for Amazon Web Services
Other
15.53k stars 4.13k forks source link

S3 Sync - corrupted files #2459

Closed aamnah closed 7 years ago

aamnah commented 7 years ago

I'm trying to sync a bucket locally aws s3 sync s3://my-site.com . and it finishes successfully every time. But the files downloaded are corrupted. The bucket i'm trying to sync has 120 objects.

# aws --version
aws-cli/1.11.52 Python/2.7.10 Darwin/16.4.0 botocore/1.5.15

An .html file ends up looking like this:

‹½XýŽÛ6ÿŸb¢ù@Ci?’´ëµl“4I‘Ëå²)zE´DKÜ•D•¤ìuÜkÜëÝ“Ü)ieË´wi‚$–f†óÅá̏šÞyöקï}ûr[ñÁÔÿLsÁSzÀÇRXIεv4vɾ†¬ÜÚš‰ß¹šg?Ÿ³§ª¬¹•‹B¨ÊŠ ×½z>i&¶VV¼³`%źVÚ„×2µù,+™æ^‚¬¤•¼`&á…˜uŠ¬´…ˆW›¢€ÂZYepa¹¶"Fžë%Y]Å,0vS“6s-–³€b˜D‘-RV+e…    U†æ„­ÅÂH+Xc˜àƲ£—ü“ªøÚ‰DÜ`^L”UJ—¼ŸDXÊ*D :9½Ã|%ûFpäÞ00öÕâvz1%³õ&+±\ð¥1WU–ªjÿg|-Œ*{ž„‡Î³!yà–÷‹ âåû¿¼~¹,W)¼¦VU^xõü{0MMÔ²…p‘:áR¤’ÃoÐR˜›m&µ¿œ¿{óêÍ‹ÉPaª„©îYX+}r    Õ9°¹€šgß8,e!0![ê> taÑ#8ýè©H7‰–µ£ŸEƒ«”1aɯ“´r¢&òØär„ß…Ç7ï.—˜‰iäÕü­Ú‡…Pgûv‹Æ面Jåò£guílºPé7â`Zñ$næ,ÀÔ6h…m%@²Ì°y©Ê•Ñ4•[bŒZ—•Ð]•  •© ‰ÿÿ¥Äï_?{ÏÄ ÞкiÄ[«š;à<¤²7_Ô Y¥â:¤]À5ÔYð“.;ËK¬ ÎrOd½«_ƾPÝváœHcC{C>LMÍ«œ–½|i¯°ÛÊ•ØñËÇžñž1pÎÓþ÷XÛ¾ý€”±c‰J·½"±?Ç¥vBm9uáhc·LG¿qÌ‹îºÖŸ†öaá)¦C~0<Û'wÈ©•±{Ž³sïæ åGqŸèÒá –y¼ÁŒÇ>®[ÁG—QËÐg÷ÜÎ…zè#€tcœîØ)·èŒv±PÉSíù;>…ó&ƒããG]ººµ‡½u7’ÐA™t@‰\‰¿ñAx™i~2EX¦tZcç5L–±Å¿ å-Qà•£`€Ç´¥â)ÕU¬›ÅýŸ‰Ç•€{šÀ¢´Yl•x›^Ódâ…6“Io&ÔMuÿèf.=èíìnfÏ  jŸs^V<’„ë.žVbȬ1sNÐ1K¡“£‚g*=<,½LNu1T S±äúŠ¢‚ö‚3/ìÀO¶!ðS±”×溞{>fœWsœël…é<±ºž­J7ô½0bcdJ»•{Ø–gýÊ=ìRi1׉ÐõÀñm6¯’\ùî°±Þm3PïÙ‚ºYÒäÁǃ>¸‡çÐUÖVUщ«"«üya~ûqÈ)pèj.?ÁC\ÇI V9˜â%ñ €rø»TE*tØçM­á†¯zaÆÖÜ&ùÈ‹ºnù-¤ÑÕ¸‰h‰x¯Èº7‡?š®AU&<ã ‚N„ x)643Í7Å/¢6kisœ>såˆ,°X VÕNÃ0jàv§.¦6Ufƒ=ëšå2ËüGí$ˆ/^ˆ“>gà—O‡íF¹jÁ3^V€]@*–¼A¬Ç–î&n?÷AíVížÌ¾&ÈÝîYçIêì³ »h­(ë¹ÿù׿1ñˆÎBc—Iïß\VIѤÂ8ìæ:kk¯ß÷øúsZa&=uwNÑ¢Ð6ÖÔE㧾àÁá…‰Cí»íC,) ;><}ü$ˆ&F_8hvµú¦ú¥záô¦*1Qß$£q{z¯®±VK™5Ú!QTôtø¾WÖRr¥VB/µvŠð†`HÜDO¾{rtzzõuĨ0»°±Y·¿Ìݐ—‘}¹Jî+K·[௸‹þ‚aü”ëKÔO–ƒáÌsc– œîlsoOaj’8²ïVSŸÝ‚{±ìøÁ åÝ Ê'¦ÆÎ7¼,æVÍë ̪‡ ¸G+©o±»Oà¥ZS½¶‹€N]ND¿Ò½îAR¹XiU1ïÄ¿SWfFè`¨¦½ø|¤û‘Ii%®må>nq†9FmÔ­wÛøÿs÷•ý{ÛßÿÒÍö¼âÅKÉgÈ°ê°m ˆ ^[– ±¨ŽûÌÜÿz‰!°JT½icxÚ=Â]¢ž9 ¬»¨Æ[É_q»™ò Ì Â‹õ3ìk÷œeri~ňQ̉‡¸I?6EA´]yì °p­qÜï–nIõwÞÜÁ¥RÀY…{‚·mUá¶ÑHRaEå3‡hl·ux\äàL|îžáüJór€ý[s„#oË‚ ú<'1Ñ&ˆ÷Wgwµ·kòH»6å-:ƒðÂ΂÷ž;>^¶gЕ`Û»=V\¿Ï”Ê á,};6õÂq¿›òË©ø½æ2<"ïá-Òlm8ú}¬}7ÕƒóC/]>(]þ­z÷+³Ðp|$¸ò «y}ÏÀO|Å/ügŒ*C,õàækϾï/ü’_·éãµlo¼H‹ ¹0Ñ%}€ÚDGáþmßnýÃØØD7™Û¥¤Ý?2§óp¬ìóŠ]ÈÝw£ž„ÇáatinHÔáÿåÊÛ}0¼ì¿Ïîjù/SS÷µ,þ/T±¹™

Ideas? Am i missing something or is it an issue with the tool?

kyleknap commented 7 years ago

I am not sure why this is happening. Are all of the files corrupted? Could you run md5 on those files and compare them with the ETag of the objects (assuming the object was not uploaded in multipart or encrypted server side)?

Sambego commented 7 years ago

I an having the same problem when uploading a .dmg file to S3.

I'm using this command to upload a file to S3:

aws s3 cp "path/to/file" "s3://bucket/file" --grants read=uri=http://acs.amazonaws.com/groups/global/AllUsers

I'm using this version of the aws cli:

aws-cli/1.11.55 Python/2.7.10 Darwin/16.3.0 botocore/1.5.18
Sambego commented 7 years ago

Nevermind my previous comment, I made a mistake in the upload script and was uploading a .zip file and renaming it as a .dmg by accident. Uploading the correct file works like a charm!

JordonPhillips commented 7 years ago

@aamnah can you confirm whether or not the checksums match?

yesharhadi commented 5 years ago

I'm having this issue when downloading HTMLs using the AWS CLI. :(