search

rstacruz / sinatra-assetpack

Package your assets transparently in Sinatra.
http://ricostacruz.com/sinatra-assetpack/
MIT License
541 stars 97 forks source link

Encoding::UndefinedConversionError from UTF-8 to US-ASCII while minifying JS in production #76

Open kgrz opened 11 years ago

kgrz commented 11 years ago

The assets part of my app:

assets {                                                                                                                                                                                       
  serve '/javascripts', from: 'public/javascripts'
  serve '/stylesheets', from: 'public/stylesheets'
  serve '/images', from: 'public/images'

  js :foundation, '/javascripts/foundation.js',[
    '/javascripts/foundation/*.js'
  ]

  js :application, '/javascripts/application.js',[
    '/javascripts/application.js'
  ]

  css :application, '/stylesheets/application.css', [
    '/stylesheets/app.css',
  ]

  js_compression :yui
  css_compression :yui

I'm using slim as my templating engine. The app is a "modular" Sinatra app. Not sure if these change the behaviour.

The offending part is at the line https://github.com/rstacruz/sinatra-assetpack/blob/master/lib/sinatra/assetpack/package.rb#L101

This is unrelated to the yui compressor since changing that to the default compressor also triggers this error. The issue with this error is when compiling/compressing the assets on a per-request basis, the javascripts files won't get sent to the browser:

Screen Shot 2013-02-04 at 2 18 14 PM

Changing the entire method to the below snippet corrects this behaviour. 


if result.body.respond_to?(:force_encoding)
  result.body.force_encoding("ISO-8859-1").encode("UTF-8")  if result.status == 200                                                                                    
else
  result.body  if result.status == 200
end

Tested this with 1.8.7 (Mac system ruby), 1.9.3-p327, rubinius 1.2.4. Here's the commit in my repo https://github.com/kgrz/sinatra-assetpack/commit/4810b71d58b6087c5a28b8e5a0bf36a409a19bba Will send a PR if this gets tested thoroughly. The tests pass without any error but there are failures which were there before changing this method.

I'm guessing the reason for this happening only in production is that the combined method is not used in other environments.

Another point to note is that this setup ( assets in public directory) cannot used if you want to build the assets first since by default, the built assets get copied into public directory.

Here's the stacktrace:

Encoding::UndefinedConversionError - U+FEFF from UTF-8 to US-ASCII:
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-assetpack-0.1.2/lib/sinatra/assetpack/package.rb:101:in `encode'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-assetpack-0.1.2/lib/sinatra/assetpack/package.rb:101:in `block in combined'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-assetpack-0.1.2/lib/sinatra/assetpack/package.rb:97:in `map'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-assetpack-0.1.2/lib/sinatra/assetpack/package.rb:97:in `combined'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-assetpack-0.1.2/lib/sinatra/assetpack/package.rb:83:in `minify'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-assetpack-0.1.2/lib/sinatra/assetpack/class_methods.rb:28:in `block (3 levels) in add_compressed_routes!'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/tilt-1.3.3/lib/tilt.rb:127:in `fetch'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-assetpack-0.1.2/lib/sinatra/assetpack/class_methods.rb:27:in `block (2 levels) in add_compressed_routes!'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:1293:in `call'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:1293:in `block in compile!'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:860:in `[]'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:860:in `block (3 levels) in route!'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:876:in `route_eval'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:860:in `block (2 levels) in route!'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:897:in `block in process_route'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:895:in `catch'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:895:in `process_route'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:859:in `block in route!'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:858:in `each'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:858:in `route!'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:963:in `block in dispatch!'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:946:in `block in invoke'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:946:in `catch'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:946:in `invoke'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:960:in `dispatch!'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:794:in `block in call!'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:946:in `block in invoke'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:946:in `catch'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:946:in `invoke'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:794:in `call!'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:780:in `call'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/rack-protection-1.3.2/lib/rack/protection/xss_header.rb:27:in `call'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/rack-protection-1.3.2/lib/rack/protection/path_traversal.rb:16:in `call'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/rack-protection-1.3.2/lib/rack/protection/json_csrf.rb:17:in `call'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/rack-protection-1.3.2/lib/rack/protection/base.rb:48:in `call'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/rack-protection-1.3.2/lib/rack/protection/base.rb:48:in `call'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/rack-protection-1.3.2/lib/rack/protection/xss_header.rb:27:in `call'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/rack-1.5.1/lib/rack/nulllogger.rb:9:in `call'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/rack-1.5.1/lib/rack/head.rb:11:in `call'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/sinatra-1.3.4/lib/sinatra/base.rb:124:in `call'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/connection.rb:81:in `block in pre_process'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/connection.rb:79:in `catch'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/connection.rb:79:in `pre_process'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/connection.rb:54:in `process'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/connection.rb:39:in `receive_data'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/eventmachine-1.0.0/lib/eventmachine.rb:187:in `run_machine'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/eventmachine-1.0.0/lib/eventmachine.rb:187:in `run'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/backends/base.rb:63:in `start'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/thin-1.5.0/lib/thin/server.rb:159:in `start'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/rack-1.5.1/lib/rack/handler/thin.rb:16:in `run'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/rack-1.5.1/lib/rack/server.rb:264:in `start'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/rack-1.5.1/lib/rack/server.rb:141:in `start'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/lib/ruby/gems/1.9.1/gems/rack-1.5.1/bin/rackup:4:in `<top (required)>'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/bin/rackup:23:in `load'
    /Users/kashyapkmbc/.rbenv/versions/1.9.3-p327/bin/rackup:23:in `<main>'
j15e commented 11 years ago

Thanks for the detailed report.

I think the encoding issue might be related to querying via http to get assets, we should get them from the disk directly when possible. Might be related to #68 too

j15e commented 11 years ago

(and or detect & deal with encoding properly, I think your fix will work only for files encoded in ISO-8859-1)

kgrz commented 11 years ago

ASCII-8BIT to UTF-8 conversion can be done this way without errors. So, files encoded in ISO-8859-1, US-ASCII, ASCII-8BIT and UTF-8 can be converted to UTF-8 encoding without errors.

j15e commented 11 years ago

Unfortunately force_encoding is not (1.8.7 compatible](http://ruby-doc.org/core-1.8.7/String.html) and I doubt doing this is the real solution as force_encoding does not change anything. We should add a test I adress this issue with a cleaner solution.

shioyama commented 11 years ago

Just bit by the same bug. Any progress?

j15e commented 11 years ago

I haven't work on this issue, but would sure accept a pull request on it. I doubt replacing the package snippet with force_encoding would solve all cases.

doits commented 11 years ago

just stumbled upon this right now, too. why is it trying to convert from utf-8 to ascii anyway? everything is written in utf-8 on my server (hopefully all vendor js, too), output to clients is utf-8. why does it want to convert from utf-8 to ascii?

kgrz commented 11 years ago

@doits I suppose this has to do with backwards compatibility. (@j15e mentioned this earlier :)

j15e commented 11 years ago

I do not have any update on this issue but it should indeed be resolved and we have to find a clean solution for this issue.

doits commented 11 years ago

I could get rid of the error by setting the correct LANG env var. By doing LANG=en_US.UTF-8 it changed Encoding.default_external to UTF-8. Therefore, assetpack does not try to convert things to ASCII, but to UTF-8 and everything works.

I could do this for apache with phusion passenger by SetEnv LANG en_US.UTF-8.

Soulou commented 10 years ago

I'm also encountering this issue, is there something new? (Setting the env var LANG didn't work in my case.

jarredholman commented 10 years ago

Can an option be provided to specify which encoding should be used instead of forcing it to the Encoding.default_external?

willmoss commented 10 years ago

I encountered this issue today, it would be nice to get a fix thanks

j15e commented 10 years ago

Could you report your full stacktrace @willmoss ? Is it exactly the same as the original issue?

papanikge commented 7 years ago

Just encountered this in production as well.

j15e commented 7 years ago

@papanikge unfortunately, this repo is not maintained anymore. see #197 there is many other better up to date assets manager for sinatra