search

fredjean / middleman-s3_sync

The gem that tries really hard not to push files to S3.
MIT License
271 stars 75 forks source link

Gzipped files don't seem to be uploaded #47

Closed manuelmeurer closed 10 years ago

manuelmeurer commented 10 years ago

I just started using middleman-s3_sync instead of middleman-sync for the same reason you started this project, to "sync" instead of "upload everything every time". :smile:

It seems like my gzipped files are not uploaded though.

$ bundle exec middleman build
      create  build/images/projects/productwidgets-2bccdd77.png
      create  build/images/projects/karmabox-3f10d448.png
      create  build/images/projects/periphereswissen-e84e2afc.png
      create  build/images/glyphicons-halflings-84f61363.png
      create  build/images/imprint_de-df22fd9d.png
      create  build/images/imprint_en-96cd7768.png
      create  build/images/glyphicons-halflings-white-a25c4705.png
      create  build/images/projects/carfinder-a85e7694.png
      create  build/images/logo-0c359c3e.jpg
      create  build/images/face-3f2b032a.jpg
      create  build/images/blank-54a69da1.gif
      create  build/fonts/zocial-regular-webfont-55d26298.svg
      create  build/fonts/zocial-regular-webfont-a7dc3960.woff
      create  build/fonts/zocial-regular-webfont-b0e173c9.ttf
      create  build/fonts/zocial-regular-webfont-bc2accd3.eot
      create  build/javascripts/site_head-5ce02500.js
      create  build/javascripts/libraries/site_head/modernizr-47817d41.js
      create  build/javascripts/site-daf3f650.js
      create  build/javascripts/libraries/site/jquery-89a6b84f.js
      create  build/javascripts/libraries/site/jquery.lazyload-9f33a126.js
      create  build/stylesheets/site-80d930b3.css
      create  build/projects/index.html
      create  build/blog/page/2/index.html
      create  build/blog/index.html
      create  build/contact/index.html
      create  build/blog/feed.xml
      create  build/blog/2014/03/06/load-remote-rails-links-and-forms-in-bootstrap-modals/index.html
      create  build/blog/2014/03/06/get-query-parameter-from-current-url-with-javascript/index.html
      create  build/blog/2014/01/10/how-to-drop-your-postgres-database-with-rails-4/index.html
      create  build/blog/2013/12/17/how-to-access-your-google-drive-files-with-ruby/index.html
      create  build/blog/2013/11/19/how-to-set-up-automatic-retweeting-using-a-google-apps-script-yahoo-pipes-and-twitterfeed/index.html
      create  build/blog/2013/10/29/the-joy-of-ruby-and-small-incremental-improvements/index.html
      create  build/blog/2013/10/06/making-php-development-bearable-for-ruby-developers-using-coffeescript-sass-and-haml/index.html
      create  build/blog/2013/08/13/bootstrap-navbar-helpers-for-rails-and-middleman/index.html
      create  build/imprint/index.html
      create  build/index.html
      create  build/blog/2013/05/22/new-gem-gemconfig-a-nifty-way-to-make-your-gem-configurable/index.html
      create  build/blog/2013/03/06/new-project-carfinder-dot-io-find-your-closest-carsharing-car/index.html
      create  build/sitemap.xml
      create  build/robots.txt
      create  build/blog/2013/01/11/new-gem-skimlinks/index.html
      create  build/blog/2012/12/19/new-gem-tries/index.html
      create  build/blog/2012/11/14/create-a-cache-key-from-method-argument-values/index.html
      create  build/blog/2012/10/07/how-to-make-everything-background-processable-through-sidekiq/index.html
      create  build/blog/2012/09/30/the-road-ahead-for-showspace/index.html
      create  build/blog/2012/08/22/how-to-install-and-secure-mysql-5-5-from-source-with-sprinkle/index.html
      create  build/blog/2012/06/17/how-to-upload-your-gmvault-backups-to-s3/index.html
      create  build/blog/2012/03/27/how-to-compile-custom-sass-stylesheets-dynamically-during-runtime/index.html
        gzip  build/blog/2012/03/27/how-to-compile-custom-sass-stylesheets-dynamically-during-runtime/index.html.gz (6.2 KB smaller)
        gzip  build/blog/2012/06/17/how-to-upload-your-gmvault-backups-to-s3/index.html.gz (4.4 KB smaller)
        gzip  build/blog/2012/08/22/how-to-install-and-secure-mysql-5-5-from-source-with-sprinkle/index.html.gz (4.6 KB smaller)
        gzip  build/blog/2012/09/30/the-road-ahead-for-showspace/index.html.gz (7.2 KB smaller)
        gzip  build/blog/2012/10/07/how-to-make-everything-background-processable-through-sidekiq/index.html.gz (5.3 KB smaller)
        gzip  build/blog/2012/11/14/create-a-cache-key-from-method-argument-values/index.html.gz (4.9 KB smaller)
        gzip  build/blog/2012/12/19/new-gem-tries/index.html.gz (4.1 KB smaller)
        gzip  build/blog/2013/01/11/new-gem-skimlinks/index.html.gz (4.5 KB smaller)
        gzip  build/blog/2013/03/06/new-project-carfinder-dot-io-find-your-closest-carsharing-car/index.html.gz (4.4 KB smaller)
        gzip  build/blog/2013/05/22/new-gem-gemconfig-a-nifty-way-to-make-your-gem-configurable/index.html.gz (3.8 KB smaller)
        gzip  build/blog/2013/08/13/bootstrap-navbar-helpers-for-rails-and-middleman/index.html.gz (4.2 KB smaller)
        gzip  build/blog/2013/10/06/making-php-development-bearable-for-ruby-developers-using-coffeescript-sass-and-haml/index.html.gz (9.5 KB smaller)
        gzip  build/blog/2013/10/29/the-joy-of-ruby-and-small-incremental-improvements/index.html.gz (7.2 KB smaller)
        gzip  build/blog/2013/11/19/how-to-set-up-automatic-retweeting-using-a-google-apps-script-yahoo-pipes-and-twitterfeed/index.html.gz (6.9 KB smaller)
        gzip  build/blog/2013/12/17/how-to-access-your-google-drive-files-with-ruby/index.html.gz (10.3 KB smaller)
        gzip  build/blog/2014/01/10/how-to-drop-your-postgres-database-with-rails-4/index.html.gz (4.3 KB smaller)
        gzip  build/blog/2014/03/06/get-query-parameter-from-current-url-with-javascript/index.html.gz (4.3 KB smaller)
        gzip  build/blog/2014/03/06/load-remote-rails-links-and-forms-in-bootstrap-modals/index.html.gz (4.2 KB smaller)
        gzip  build/blog/index.html.gz (11 KB smaller)
        gzip  build/blog/page/2/index.html.gz (8.6 KB smaller)
        gzip  build/contact/index.html.gz (2.4 KB smaller)
        gzip  build/imprint/index.html.gz (12.5 KB smaller)
        gzip  build/index.html.gz (1.7 KB smaller)
        gzip  build/javascripts/libraries/site/jquery-89a6b84f.js.gz (59.7 KB smaller)
        gzip  build/javascripts/libraries/site/jquery.lazyload-9f33a126.js.gz (2.2 KB smaller)
        gzip  build/javascripts/libraries/site_head/modernizr-47817d41.js.gz (2.8 KB smaller)
        gzip  build/javascripts/site-daf3f650.js.gz (64.2 KB smaller)
        gzip  build/javascripts/site_head-5ce02500.js.gz (2.8 KB smaller)
        gzip  build/projects/index.html.gz (6.7 KB smaller)
        gzip  build/stylesheets/site-80d930b3.css.gz (126.4 KB smaller)
        gzip  Total gzip savings: 401.2 KB
     s3_sync  Gathering the paths to evaluate.

     s3_sync
All S3 files are up to date.

None of the .gz files are present in my S3 folder though.

fredjean commented 10 years ago

Can you point me to your website? I'll check on the headers to see what is currently stored in S3.

Sent from my iPhone

On Mar 8, 2014, at 3:52 AM, Manuel Meurer notifications@github.com wrote:

I just started using middleman-s3_sync instead of middleman-sync for the same reason you started this project, to "sync" instead of "upload everything every time".

It seems like my gzipped files are not uploaded though.

$ bundle exec middleman build create build/images/projects/productwidgets-2bccdd77.png create build/images/projects/karmabox-3f10d448.png create build/images/projects/periphereswissen-e84e2afc.png create build/images/glyphicons-halflings-84f61363.png create build/images/imprint_de-df22fd9d.png create build/images/imprint_en-96cd7768.png create build/images/glyphicons-halflings-white-a25c4705.png create build/images/projects/carfinder-a85e7694.png create build/images/logo-0c359c3e.jpg create build/images/face-3f2b032a.jpg create build/images/blank-54a69da1.gif create build/fonts/zocial-regular-webfont-55d26298.svg create build/fonts/zocial-regular-webfont-a7dc3960.woff create build/fonts/zocial-regular-webfont-b0e173c9.ttf create build/fonts/zocial-regular-webfont-bc2accd3.eot create build/javascripts/site_head-5ce02500.js create build/javascripts/libraries/site_head/modernizr-47817d41.js create build/javascripts/site-daf3f650.js create build/javascripts/libraries/site/jquery-89a6b84f.js create build/javascripts/libraries/site/jquery.lazyload-9f33a126.js create build/stylesheets/site-80d930b3.css create build/projects/index.html create build/blog/page/2/index.html create build/blog/index.html create build/contact/index.html create build/blog/feed.xml create build/blog/2014/03/06/load-remote-rails-links-and-forms-in-bootstrap-modals/index.html create build/blog/2014/03/06/get-query-parameter-from-current-url-with-javascript/index.html create build/blog/2014/01/10/how-to-drop-your-postgres-database-with-rails-4/index.html create build/blog/2013/12/17/how-to-access-your-google-drive-files-with-ruby/index.html create build/blog/2013/11/19/how-to-set-up-automatic-retweeting-using-a-google-apps-script-yahoo-pipes-and-twitterfeed/index.html create build/blog/2013/10/29/the-joy-of-ruby-and-small-incremental-improvements/index.html create build/blog/2013/10/06/making-php-development-bearable-for-ruby-developers-using-coffeescript-sass-and-haml/index.html create build/blog/2013/08/13/bootstrap-navbar-helpers-for-rails-and-middleman/index.html create build/imprint/index.html create build/index.html create build/blog/2013/05/22/new-gem-gemconfig-a-nifty-way-to-make-your-gem-configurable/index.html create build/blog/2013/03/06/new-project-carfinder-dot-io-find-your-closest-carsharing-car/index.html create build/sitemap.xml create build/robots.txt create build/blog/2013/01/11/new-gem-skimlinks/index.html create build/blog/2012/12/19/new-gem-tries/index.html create build/blog/2012/11/14/create-a-cache-key-from-method-argument-values/index.html create build/blog/2012/10/07/how-to-make-everything-background-processable-through-sidekiq/index.html create build/blog/2012/09/30/the-road-ahead-for-showspace/index.html create build/blog/2012/08/22/how-to-install-and-secure-mysql-5-5-from-source-with-sprinkle/index.html create build/blog/2012/06/17/how-to-upload-your-gmvault-backups-to-s3/index.html create build/blog/2012/03/27/how-to-compile-custom-sass-stylesheets-dynamically-during-runtime/index.html gzip build/blog/2012/03/27/how-to-compile-custom-sass-stylesheets-dynamically-during-runtime/index.html.gz (6.2 KB smaller) gzip build/blog/2012/06/17/how-to-upload-your-gmvault-backups-to-s3/index.html.gz (4.4 KB smaller) gzip build/blog/2012/08/22/how-to-install-and-secure-mysql-5-5-from-source-with-sprinkle/index.html.gz (4.6 KB smaller) gzip build/blog/2012/09/30/the-road-ahead-for-showspace/index.html.gz (7.2 KB smaller) gzip build/blog/2012/10/07/how-to-make-everything-background-processable-through-sidekiq/index.html.gz (5.3 KB smaller) gzip build/blog/2012/11/14/create-a-cache-key-from-method-argument-values/index.html.gz (4.9 KB smaller) gzip build/blog/2012/12/19/new-gem-tries/index.html.gz (4.1 KB smaller) gzip build/blog/2013/01/11/new-gem-skimlinks/index.html.gz (4.5 KB smaller) gzip build/blog/2013/03/06/new-project-carfinder-dot-io-find-your-closest-carsharing-car/index.html.gz (4.4 KB smaller) gzip build/blog/2013/05/22/new-gem-gemconfig-a-nifty-way-to-make-your-gem-configurable/index.html.gz (3.8 KB smaller) gzip build/blog/2013/08/13/bootstrap-navbar-helpers-for-rails-and-middleman/index.html.gz (4.2 KB smaller) gzip build/blog/2013/10/06/making-php-development-bearable-for-ruby-developers-using-coffeescript-sass-and-haml/index.html.gz (9.5 KB smaller) gzip build/blog/2013/10/29/the-joy-of-ruby-and-small-incremental-improvements/index.html.gz (7.2 KB smaller) gzip build/blog/2013/11/19/how-to-set-up-automatic-retweeting-using-a-google-apps-script-yahoo-pipes-and-twitterfeed/index.html.gz (6.9 KB smaller) gzip build/blog/2013/12/17/how-to-access-your-google-drive-files-with-ruby/index.html.gz (10.3 KB smaller) gzip build/blog/2014/01/10/how-to-drop-your-postgres-database-with-rails-4/index.html.gz (4.3 KB smaller) gzip build/blog/2014/03/06/get-query-parameter-from-current-url-with-javascript/index.html.gz (4.3 KB smaller) gzip build/blog/2014/03/06/load-remote-rails-links-and-forms-in-bootstrap-modals/index.html.gz (4.2 KB smaller) gzip build/blog/index.html.gz (11 KB smaller) gzip build/blog/page/2/index.html.gz (8.6 KB smaller) gzip build/contact/index.html.gz (2.4 KB smaller) gzip build/imprint/index.html.gz (12.5 KB smaller) gzip build/index.html.gz (1.7 KB smaller) gzip build/javascripts/libraries/site/jquery-89a6b84f.js.gz (59.7 KB smaller) gzip build/javascripts/libraries/site/jquery.lazyload-9f33a126.js.gz (2.2 KB smaller) gzip build/javascripts/libraries/site_head/modernizr-47817d41.js.gz (2.8 KB smaller) gzip build/javascripts/site-daf3f650.js.gz (64.2 KB smaller) gzip build/javascripts/site_head-5ce02500.js.gz (2.8 KB smaller) gzip build/projects/index.html.gz (6.7 KB smaller) gzip build/stylesheets/site-80d930b3.css.gz (126.4 KB smaller) gzip Total gzip savings: 401.2 KB s3_sync Gathering the paths to evaluate.

 s3_sync

All S3 files are up to date. None of the .gz files are present in my S3 folder though.

— Reply to this email directly or view it on GitHub.

manuelmeurer commented 10 years ago

http://www.krautcomputing.com/ Shouldn't I see the gzipped files in S3 though?

fredjean commented 10 years ago

It looks like the content that is in the S3 bucket isn't gzipped. I would try using the force option to push the gzipped content up.

S3 doesn't support content and encoding negotiations. As a result, we push the gzipped files up using the original name. Due to some GZIP issues, we are storing and comparing the fingerprint of the content before it is compressed. As a result, turning on compression and trying to update existing, non compressed content doesn't work.

I'm pondering a way to get this to work without a force. I just haven't thought of a good way to do it yet.

fredjean commented 10 years ago

Actually, I might just have thought of a way to do so... I'll play with the idea and might be able to implement it tonight.

manuelmeurer commented 10 years ago

Awesome, cheers! :+1:

fredjean commented 10 years ago

I just pushed version 3.0.25 of the gem. The only change (beside the version number) is that s3_sync now check whether the content encoding matches what we are expecting. If not, s3_sync will update the file.

You will still not see the .gz extension attached to the file on S3, but you should see a Content-Encoding header set to gzip when you look at the files headers.

Let me know if this works for you.

manuelmeurer commented 10 years ago

Great, seems to work, although the output says all files were ignored. The S3 files as well as the response headers on my website indicate that the files are gzippped though. :+1: 

     s3_sync                                                                                                                                                                                                                                 |
Ready to apply updates to krautcomputing-web.
     s3_sync  Ignoring blog/2012/03/27/how-to-compile-custom-sass-stylesheets-dynamically-during-runtime/index.html (alternate encoding)
     s3_sync  Ignoring blog/2012/06/17/how-to-upload-your-gmvault-backups-to-s3/index.html (alternate encoding)
     s3_sync  Ignoring blog/2012/08/22/how-to-install-and-secure-mysql-5-5-from-source-with-sprinkle/index.html (alternate encoding)
     s3_sync  Ignoring blog/2012/09/30/the-road-ahead-for-showspace/index.html (alternate encoding)
     s3_sync  Ignoring blog/2012/10/07/how-to-make-everything-background-processable-through-sidekiq/index.html (alternate encoding)
     s3_sync  Ignoring blog/2012/11/14/create-a-cache-key-from-method-argument-values/index.html (alternate encoding)
     s3_sync  Ignoring blog/2012/12/19/new-gem-tries/index.html (alternate encoding)
     s3_sync  Ignoring blog/2013/01/11/new-gem-skimlinks/index.html (alternate encoding)
     s3_sync  Ignoring blog/2013/03/06/new-project-carfinder-dot-io-find-your-closest-carsharing-car/index.html (alternate encoding)
     s3_sync  Ignoring blog/2013/05/22/new-gem-gemconfig-a-nifty-way-to-make-your-gem-configurable/index.html (alternate encoding)
     s3_sync  Ignoring blog/2013/08/13/bootstrap-navbar-helpers-for-rails-and-middleman/index.html (alternate encoding)
     s3_sync  Ignoring blog/2013/10/06/making-php-development-bearable-for-ruby-developers-using-coffeescript-sass-and-haml/index.html (alternate encoding)
     s3_sync  Ignoring blog/2013/10/29/the-joy-of-ruby-and-small-incremental-improvements/index.html (alternate encoding)
     s3_sync  Ignoring blog/2013/11/19/how-to-set-up-automatic-retweeting-using-a-google-apps-script-yahoo-pipes-and-twitterfeed/index.html (alternate encoding)
     s3_sync  Ignoring blog/2013/12/17/how-to-access-your-google-drive-files-with-ruby/index.html (alternate encoding)
     s3_sync  Ignoring blog/2014/01/10/how-to-drop-your-postgres-database-with-rails-4/index.html (alternate encoding)
     s3_sync  Ignoring blog/2014/03/06/get-query-parameter-from-current-url-with-javascript/index.html (alternate encoding)
     s3_sync  Ignoring blog/2014/03/06/load-remote-rails-links-and-forms-in-bootstrap-modals/index.html (alternate encoding)
     s3_sync  Ignoring blog/index.html (alternate encoding)
     s3_sync  Ignoring blog/page/2/index.html (alternate encoding)
     s3_sync  Ignoring contact/index.html (alternate encoding)
     s3_sync  Ignoring imprint/index.html (alternate encoding)
     s3_sync  Ignoring index.html (alternate encoding)
     s3_sync  Ignoring javascripts/libraries/site/jquery-89a6b84f.js (alternate encoding)
     s3_sync  Ignoring javascripts/libraries/site/jquery.lazyload-9f33a126.js (alternate encoding)
     s3_sync  Ignoring javascripts/libraries/site_head/modernizr-47817d41.js (alternate encoding)
     s3_sync  Ignoring javascripts/site-daf3f650.js (alternate encoding)
     s3_sync  Ignoring javascripts/site_head-5ce02500.js (alternate encoding)
     s3_sync  Ignoring projects/index.html (alternate encoding)
     s3_sync  Ignoring stylesheets/site-80d930b3.css (alternate encoding)
     s3_sync  Updating sitemap.xml
fredjean commented 10 years ago

Excellent! Is it OK if I close this issue?

manuelmeurer commented 10 years ago

Sure, just thought maybe the "Ignoring ..." output is still wrong?

fredjean commented 10 years ago

It's certainly more verbose than it could be... I have created issue #49 as a reminder to tone this down a bit...