search

laurilehmijoki / jekyll-s3

Push your Jekyll Blog to Amazon S3!
182 stars 20 forks source link

Filey::Filey::InvalidMd5Error #18

Closed dguido closed 11 years ago

dguido commented 11 years ago

jekyll-s3 errors out when run with no options. This is the second time that I'm sync'ing to my S3 bucket and it should be performing an update of content already there. I'm on OS X 10.8 and I just installed jekyll-s3 2.2.2.

/Library/Ruby/Gems/1.8/gems/filey-diff-1.0.0/lib/filey-diff/filey.rb:9:in `initialize': Filey::Filey::InvalidMd5Error (Filey::Filey::InvalidMd5Error)
    from /Library/Ruby/Gems/1.8/gems/filey-diff-1.0.0/lib/filey-diff/data-sources/aws_sdk_s3.rb:20:in `new'
    from /Library/Ruby/Gems/1.8/gems/filey-diff-1.0.0/lib/filey-diff/data-sources/aws_sdk_s3.rb:20:in `do_internal_load'
    from /Library/Ruby/Gems/1.8/gems/aws-sdk-1.8.0/lib/aws/s3/bucket.rb:408:in `map'
    from /Library/Ruby/Gems/1.8/gems/aws-sdk-1.8.0/lib/aws/core/collection.rb:50:in `each'
    from /Library/Ruby/Gems/1.8/gems/aws-sdk-1.8.0/lib/aws/core/collection.rb:50:in `each'
    from /Library/Ruby/Gems/1.8/gems/aws-sdk-1.8.0/lib/aws/core/collection/with_limit_and_next_token.rb:61:in `_each_batch'
    from /Library/Ruby/Gems/1.8/gems/aws-sdk-1.8.0/lib/aws/core/collection.rb:82:in `each_batch'
    from /Library/Ruby/Gems/1.8/gems/aws-sdk-1.8.0/lib/aws/core/collection.rb:49:in `each'
    from /Library/Ruby/Gems/1.8/gems/aws-sdk-1.8.0/lib/aws/s3/object_collection.rb:282:in `each'
    from /Library/Ruby/Gems/1.8/gems/filey-diff-1.0.0/lib/filey-diff/data-sources/aws_sdk_s3.rb:11:in `map'
    from /Library/Ruby/Gems/1.8/gems/filey-diff-1.0.0/lib/filey-diff/data-sources/aws_sdk_s3.rb:11:in `do_internal_load'
    from /Library/Ruby/Gems/1.8/gems/filey-diff-1.0.0/lib/filey-diff/data-sources/data_source.rb:8:in `get_fileys'
    from /Library/Ruby/Gems/1.8/gems/filey-diff-1.0.0/lib/filey-diff/comparison.rb:28:in `select_in_outer_array'
    from /Library/Ruby/Gems/1.8/gems/filey-diff-1.0.0/lib/filey-diff/comparison.rb:11:in `list_changed'
    from /Library/Ruby/Gems/1.8/gems/jekyll-s3-2.2.2/bin/../lib/jekyll-s3/diff_helper.rb:8:in `resolve_files_to_upload'
    from /Library/Ruby/Gems/1.8/gems/jekyll-s3-2.2.2/bin/../lib/jekyll-s3/uploader.rb:24:in `upload_files'
    from /Library/Ruby/Gems/1.8/gems/jekyll-s3-2.2.2/bin/../lib/jekyll-s3/uploader.rb:11:in `run'
    from /Library/Ruby/Gems/1.8/gems/jekyll-s3-2.2.2/bin/../lib/jekyll-s3/cli.rb:14:in `run'
    from /Library/Ruby/Gems/1.8/gems/jekyll-s3-2.2.2/bin/../lib/jekyll-s3/cli.rb:7:in `run'
    from /Library/Ruby/Gems/1.8/gems/jekyll-s3-2.2.2/bin/jekyll-s3:7
    from /usr/bin/jekyll-s3:23:in `load'
    from /usr/bin/jekyll-s3:23
laurilehmijoki commented 11 years ago

Hello dguido

The stack trace implies that the length of the MD5 checksum string of your S3 object is not 32. Currently, filey.rb requires this.

A valid hex-encoded MD5 checksum should always have the length of 32.

Could you debug the case a bit? You can easily add print statements into the filey.rb by modifying your local file /Library/Ruby/Gems/1.8/gems/filey-diff-1.0.0/lib/filey-diff/filey.rb. Please add puts md5 right above the row 9 in filey.rb and run jekyll-s3. Then you can paste the output of the puts statement here. After that we can hopefully figure out what's wrong with the MD5 checksum.

P.s. I was able to sync an already existing file with the version 2.2.2, which means that this problem does not affect every use case.

dguido commented 11 years ago

I added a puts md5 + " " + name right above the length check: 

jekyll-s3 
Deploying _site/* to www.xxx.com
28cad361988e6078364be03be97b4b21 404.html
7f93b81b789992462e67723fe122cb72 index.html
f26610a32d62062b8fdeecb16773affb footer.js
5989e0f1b96be8ec20de673d4ae84a9e header.js
2b40642e78c6d369af5c6bcfa0c1720b iecss.css
c3fbfd9f11fa8252478c7eb056c80a87 iejs_footer.js
b8b4c5273f21738e89074b3746feda30 iejs_header.js
bad91d529ba1986819428a11b85f97bc redirect.js
70da773d8375e0344be558ed424f98b2 screen.css
67d4344c7ce77bb9fdec4f438cd59bf2 MHH.README.txt
29797619b047bb4511c4beaacf705cf7 MacHackersHandbook-code.zip
31f24732574acb40d4f0b9a1e80224c4 ida-x86emu-objc.zip
82ddff8ca4be361ece2bfa71be2e4e22 index.html
5584ade2dc16d88ca8cde7f2dbf85e9a base.css
9f4cac7e8decf5bd9388c96904a2e6a3 flexslider.css
e13223a0be6cf879187a8f188236cfa9 ie7.css
dee67df72949588088efcdae16151feb layout.css
a916e52dd603dc4c8deb1f2264e938d1 skeleton.css
dd6b7b0bf5c3af22499abc0a9ee1e1b2 ajax-loader.gif
cd0e756779096853316e3a3a62a914e1 bkg_smoky_cords.jpg
b57f557acd652b040e42a2dd742554cd bkg_smoky_noise.jpg
4abff055bc502c80ba5e4e81d49d3d14 stripes_05.png
c186570098dc23a3e111b089a7bd4be3 stripes_15.png
5e4caaf35d2e44644c45dacd2a72f0b2 stripes_small_05.png
35b282dbe93fcbdd5ad0317194c466f9 stripes_tiny_05.png
148b9c1200f6305f9f82619382c81c7e stripes_tiny_08.png
dfb16088e95c37cbb24a5096afb5ac4d stripes_tiny_10.png
3b9fbaa032a294ee39b37107140b7209 fleuron.png
33730e7eed96b918f8a96e46e480e772 arrow_right.png
9727a5fbb49dc9d4d2fe31dd9ac09169 calendar_1.png
6f561e85ecce0fc4b2b0aa127a262c26 camera_4.png
0235ab30c820f7e1678205a3397edacc chat_2.png
03f97b11b530fe05ed9e05377f9f8044 cross.png
09b5765fd83bac644555ed54a06bb407 link_1.png
b30271a1425d47d0f68f61f3fed836ac pause.png
6824421aa2a3d8e118bd4d4889e2a782 pause_12.png
9caeb7cc348781322d2407a425cbc261 pinboard.png
59d928300e7ea2dcb6cf5748bebb73df play_12.png
2b642637b4813399cdb206232e645f07 tag_1.png
35abf69fdf0b1db8e23f8d9e17840931 tag_3.png
a642081b7444a021e8968b3008d63815 tick.png
483c3ca2621a8085e35b0c9078eb5dbe tick_green.png
72714510425216388ed2eeb68c544a71 twitter.png
148e32cf5680ed93141aa2044a46b5c4 twitter_2.png
be49ba14efa12633a4375ba093327ee5 video_recorder.png
956c4eb13180b0e22e9e4d9a7e943c7e vimeo.png
c8800f330157202a34f78f6bd56ce4b4 warning.png
07d15abd3d424e145e97f4246e558921 zoom_in.png
0cf523d94067de432cceb112f4a2372b slider_arrows.png
bbfd7b49dc892a72a8a87d8d1ae3e4ee spacer.gif
5c33dafff50d6bcc61d65e2a73ee061f Lockup_78_Bottom.png
21af155563411956f19d1370d6730ab8 Lockup_78_Middle.png
c18a7b6e23ee57e108fa11c2988d9af3 Lockup_78_Middle_BLUE.png
e1b18b9b39ba4cbe4eb8fc7dcfc8cb24 Lockup_78_Top.png
461137180024583705f996c270f3dc10 Lockup_BOTTOM.png
c772009b56eea31a637eaf7868d0ea07 Lockup_MIDDLE.png
43a7db2d6d898dc5f2a65c81ba0acd64 Lockup_TOP.png
6bad567eddc3bf63a673eca7d769ea20 bottom-products.jpg
e5fd48b2e41df67d0576ebd1f2bc767d bottom-products_opt.jpg
a5704dadb06892383e84733828ed0c77 bottom-research.jpg
40e036d522fe2dadecc48051d781fcb4 bottom-research_opt.jpg
d6d282ca5b0d7da5d949357590bdb976 bottom-services.jpg
bb6189f422339848c3653756e58259a6 bottom-services_opt.jpg
c2cc7933db420f025c78bffbad9a99de bottom-training.jpg
dace8bd2b9a058e3e937de81093e8014 during-conference.jpg
f4f4fccfa6dfb9a315f121f5fd6a3166 ios_hackers_handbook.jpg
b3020211eecc56ddb97d7bfe0a89202a logo_blue.png
4201edf7c53e8b66823726203167746d logo_blue_small.png
6861b6c473f9dd95e79427ab8574167a main.jpg
3e2313d8d885873f2f07b00dadce9f06 main_opt.jpg
d755f6a0f9ec56acebb6b56dca516223 the_art_of_software_security_testing.jpg
f22056bb3d54562bc38f8a86f0e107ae the_mac_hackers_handbook.jpg
e424dc49bfd9614630213d4105a739d9 twitter_newbird_blue.png
00e926f996e8938e5211e74846983fe1 twitter_newbird_blue_opt.png
ca022be4a1c56c770b9700df99473d07 index.html
cf4655ad04ecd51a7427f2e69de39886 agility.js
326ec53d831b20d32374ee3cbcf46483 fix-ie.js
d5e37d8ee1a4ce73ad47f1614adce449 html5.js
bd5f108bb81229ae39c1cfacba888f0b jquery.flexslider-min.js
c6ff16a5818580a2cfba46aca7849aee jquery.flexslider.js
9118381924c51c89d9414a311ec9c97f jquery.min.js
b10a9579ee00f935a90438a3eb0b1eb3 modernizr.js
8ff319e85318c07ee2017bbf2e4b213a tabs.js
a79a56216a7e076cb85f4f093f6f9750 TOB_Blue.eps
ded4fac72c6d52d77297a06df3829edf TOB_Blue.jpg
dc9c8050945d33f83f5d58376a8152d3 TOB_Grey.eps
659e26101bf217531c4815701624d7fa TOB_Grey.jpg
f392c1602cd1d9a0176853c41b366db5-5 digital_underground_97.mp3
/Library/Ruby/Gems/1.8/gems/filey-diff-1.0.0/lib/filey-diff/filey.rb:10:in `initialize': Filey::Filey::InvalidMd5Error (Filey::Filey::InvalidMd5Error)
    from /Library/Ruby/Gems/1.8/gems/filey-diff-1.0.0/lib/filey-diff/data-sources/aws_sdk_s3.rb:20:in `new'
    from /Library/Ruby/Gems/1.8/gems/filey-diff-1.0.0/lib/filey-diff/data-sources/aws_sdk_s3.rb:20:in `do_internal_load'
    from /Library/Ruby/Gems/1.8/gems/aws-sdk-1.8.0/lib/aws/s3/bucket.rb:408:in `map'
    from /Library/Ruby/Gems/1.8/gems/aws-sdk-1.8.0/lib/aws/core/collection.rb:50:in `each'
    from /Library/Ruby/Gems/1.8/gems/aws-sdk-1.8.0/lib/aws/core/collection.rb:50:in `each'
    from /Library/Ruby/Gems/1.8/gems/aws-sdk-1.8.0/lib/aws/core/collection/with_limit_and_next_token.rb:61:in `_each_batch'
    from /Library/Ruby/Gems/1.8/gems/aws-sdk-1.8.0/lib/aws/core/collection.rb:82:in `each_batch'
    from /Library/Ruby/Gems/1.8/gems/aws-sdk-1.8.0/lib/aws/core/collection.rb:49:in `each'
    from /Library/Ruby/Gems/1.8/gems/aws-sdk-1.8.0/lib/aws/s3/object_collection.rb:282:in `each'
    from /Library/Ruby/Gems/1.8/gems/filey-diff-1.0.0/lib/filey-diff/data-sources/aws_sdk_s3.rb:11:in `map'
    from /Library/Ruby/Gems/1.8/gems/filey-diff-1.0.0/lib/filey-diff/data-sources/aws_sdk_s3.rb:11:in `do_internal_load'
    from /Library/Ruby/Gems/1.8/gems/filey-diff-1.0.0/lib/filey-diff/data-sources/data_source.rb:8:in `get_fileys'
    from /Library/Ruby/Gems/1.8/gems/filey-diff-1.0.0/lib/filey-diff/comparison.rb:28:in `select_in_outer_array'
    from /Library/Ruby/Gems/1.8/gems/filey-diff-1.0.0/lib/filey-diff/comparison.rb:11:in `list_changed'
    from /Library/Ruby/Gems/1.8/gems/jekyll-s3-2.2.2/bin/../lib/jekyll-s3/diff_helper.rb:8:in `resolve_files_to_upload'
    from /Library/Ruby/Gems/1.8/gems/jekyll-s3-2.2.2/bin/../lib/jekyll-s3/uploader.rb:24:in `upload_files'
    from /Library/Ruby/Gems/1.8/gems/jekyll-s3-2.2.2/bin/../lib/jekyll-s3/uploader.rb:11:in `run'
    from /Library/Ruby/Gems/1.8/gems/jekyll-s3-2.2.2/bin/../lib/jekyll-s3/cli.rb:14:in `run'
    from /Library/Ruby/Gems/1.8/gems/jekyll-s3-2.2.2/bin/../lib/jekyll-s3/cli.rb:7:in `run'
    from /Library/Ruby/Gems/1.8/gems/jekyll-s3-2.2.2/bin/jekyll-s3:7
    from /usr/bin/jekyll-s3:23:in `load'
    from /usr/bin/jekyll-s3:23
laurilehmijoki commented 11 years ago

I wonder why Amazon sends a 34-characters long MD5 checksum.

laurilehmijoki commented 11 years ago

Hi dguido, I just released the version 2.2.3, which contains the fix for your problem.

I did the fix like this: https://github.com/laurilehmijoki/filey-diff/commit/943a8a0eb7717916651351dd3d18d3a003724e07.

dguido commented 11 years ago

Hm, now every time I run jekyll-s3 it thinks that the files with the weird MD5s are out-of-date and need to be uploaded. I put those debug lines back in filey-diff, ran jekyll-s3 and the weird MD5s come up the same each time for the same two files.

laurilehmijoki commented 11 years ago

Do you know who generates those faulty MD5s?