Asset pipeline: locally precompiled md5 fingerprints don't match with the ones generated by rails helpers in production (rails 3.1 rc6, ruby 1.9.2)

[ncri]

I precompile assets using rake assets:precompile RAILS_ENV=production locally on my development machine.

The problem is the generated md5 fingerprints in the precompiled filenames don't match the ones generated by the rails helpers (like asset_path) in production.

Any clues why this is?

I'm on Heroku, so precompiling on the server is no option, as they have a read only file system.

[tenderlove]

@josh ping

[josh]

@hone is working on having a precompile step on heroku. Not sure if its ready yet.

Until then you can have your assets served dynamically and cached by something like rack-cache.

[guilleiguaran]

Looks like you have same problem described in #2383. I was trying to reproduce the bug with @spastorino but we couldn't

[ncri]

Ah, okay, sounds promising, thx. Dynamically served assets is okay in general, however I have an issue with getting a gem to work with it: PDF kit. I used to feed it absolute filesystem image paths, as it doesn't like relative web paths. However, I can't do that anymore now without precompilation, as the actual files don't exit on the disk otherwise. Well, for now I found a hack, but it's a bit ugly. The issue is actually not in PDFKit itself, but with the wkhtmltopdf tool pdfkit uses. It can only deal with absolute paths.

[pixeltrix]

Are you developing on windows by any chance? The different line endings could be changing md5 fingerprints.

[tenderlove]

@pixeltrix if that is true, then it sounds like a bug. We should open the files with binary encoding so that the checksums are platform independent.

Can someone point me to where the checksums are calculated? It worries me that a checksum would change depending on platform. It kind of defeats the purpose of a checksum. :-)

[kamui]

I'm having a similar issue, except I'm using the Cedar stack, which apparently has a write-able file system. Heroku will even detect Rails 3.1 apps and automatically run the assets:precompile rake task on slug compilation (although to not scare developers with error messages, it won't actually show you if it was successful or not when you "git push heroku"). If it works successfully, you can "heroku run bash" and "ls ~/public/assets" and see all your compiled assets.

In my case, the assets are all there, but the md5 fingerprints don't match the asset urls rendered in the view. You just get a bunch of 404's on every asset.

[tenderlove]

@kamui can you provide a sample app with repro steps? I'd love to track this down.

[kamui]

@tenderlove sure I'm putting it together now.

[jmazzi]

@tenderlove md5 hashes bytes, not characters. Are you using UTF-8 for the calculations?

[tenderlove]

@jmazzi if the files are opened and read in binary format, it shouldn't matter.

[ncri]

Ah @kamui, that seems actually exactly my issue! I didn't know about that's automatic precompilation on cedar. And, no, I'm not on Windows, but on a Mac. So this seems a Heroku issue. Just wonder what happens with the assets I precompile locally. Does Heroku delete them? Where did you see that Heroku does the precompilation? Is it in the slug compilation output? In that case I always overlooked it... ;-)

[ncri]

Oops, sorry, accidentally closed the issue. Wonder if the precompilation on Heroku is happening in this step: -----> Preparing app for Rails asset pipeline ?

[guilleiguaran]

There are some details about get Rails 3.1 on Heroku Cedar stack: http://devcenter.heroku.com/articles/rails31_heroku_cedar

[tenderlove]

@ncri when you push your app, heroku will do the pre-compilation. I don't know what happens to your checked in assets. Theoretically if they hashed the same, nothing would happen.

[pixeltrix]

@tenderlove it may not make a difference. If the git config autocrlf is true then any text file checked out on a windows box will have CRLF endings and when checked out on the server it will have LF endings and the bytes will be different.

[kamui]

I had a long standing issue with heroku support with 3.1.0.rc5 and assets not precompiling. So I learned a bit about how cedar works with the 3.1 asset pipeline. "rake assets:precompile" is run when the slug is compiled. Yeah, I think that line is when the rake task is being run. Support said they used to output failures, but this scared people deploying so they removed any sort of notice about whether or not the task was successful or not. I think for now the only way to check is to "heroku run bash" and take a look to see if the assets are there.

It's not a huge problem, since if the asset precompile fails, Rails will just compile the assets on the first request to a url to ~/tmp/cache/assets. Since 3.1 enables Rack::Cache, I think Rack::Cache will cache these assets in your Rails cache_store, which was one of my issues since redis is my cache_store, I had tons of static assets taking up redis memory.

As noted in the link @guilleiguaran posted, if you're using MongoDB, the precompile will fail b/c the MongoDB connection string is unavailable at slug compile time. Short term fix was to hard code the connection string instead of using ENV['MONGOHQ_URL'].

[kamui]

@tenderlove when you run "rake assets:precompile", I think it runs "rake assets:clean" first no? I think any assets you already compiled and checked in would get wiped out on the clean task.

[ncri]

Wondering if different ruby versions make a difference with fingerprint generation?? I use 1.9.2 here, Heroku cedar uses 1.9.1?

[guilleiguaran]

@ncri Heroku Cedar uses 1.9.2 also

[ncri]

Well, thats what it says in the docs: but i think its wrong. my path points to a 1.9.1 folder:

GEM_PATH => vendor/bundle/ruby/1.9.1

[stevenharman]

@ncri Heroku Cedar is 1.9.2p180, as can be seen by heroku run bash and then ruby --version

[jmazzi]

@ncri sprockets lets you use sha1. Might wanna try that out to see if it can't at least get you up and running. Not sure if rails lets you specify that tho.

[ncri]

@stevenharman: yep, thx, can confirm that. Strange, why are all ruby path's in my environment having 1.9.1 in there? I think they must have recently upgraded and not have changed the paths... Hmmm, wonder which ruby is then used with my app in actuality.

[kamui]

@ncri 1.9.2 has some concept of a "library compatible version." 1.9.2 is library compatible with 1.9.1, so bundler. and even the standard library, puts it in a 1.9.1 directory.

[ncri]

Okay, Heroku doesn't precompile my assets. Or it fails doing so, as they are not in the public folder. Locally I can just compile them fine.

[ncri]

@kamui: yes, I can confirm, assets seem to end up in my memcached cache store....

[ncri]

Ah, okay kamui, that is confusing ;-)

[tenderlove]

@ncri is it possible to give me access to your application?

[kamui]

@tenderlove: Odd, I've tried to replicate the problem with a very simple sample Rails app on Heroku, but it's compiling the assets just fine and the md5 fingerprints match up. There must be something else going on with my more complex real world app. Maybe some other gem is causing the issue. My sample app is here:

https://github.com/kamui/omfgapp http://omfgapp.herokuapp.com/

All I did was take the default Rails index.html and added a controller/view, moved out the js and css into static assets (with js/scss manifest files). There should be 3 static assets, rails.png, application.js, and application.css.

[ncri]

@tenderlove, sorry, would love to, but an nda would be needed

[ncri]

Ah, okay @kamui, that observation could also be true for our app, which is rather complex and uses many gems...

[tenderlove]

@kamui A few questions:

• How many assets do you have in your Real World app?
• Can you move your assets to the sample app, and see if it fails to push?
• Then start removing assets until it is successful again?

I know it's time consuming, but that's what I would do if I had access to your app.

My hunch is that somewhere we are doing a Dir glob. Dir globs do not guarantee a particular order is returned, and this order can change from file system to file system. But it's possible that a small enough number of files will return the same order between FSs.

Edit: this would also explain the difference with JRuby. It's quite likely that dir globs return different order on JRuby.

[tenderlove]

@ncri I'm willing to sign an NDA to get this issue fixed.

[kamui]

@tenderlove:

There's some 687 assets in my app/assets folder.

I'll try that and see what I can find. Would that be a bug that was introduced in rc6? My rc5 app didn't have this issue.

[tenderlove]

@kamui If my hunch is correct, then it's random. The filesystems could randomly return the same list, and we can't tell the version where the problem started until we actually track it down.

[daliborfilus]

Mmm.. Interesting issue. @ncri do you have the same sprockets gem version on heroku as on your mac? (see https://github.com/rails/rails/issues/2383#issuecomment-1782975 )..

@pixeltrix yep, that can be it too .. there can also be some hook which .strips blank lines?

@ncri try to replace require_tree with literal list of included files, as @tenderlove says (yep, Dir[] should return files in order which they were created on a fs, not by filename.., and mostly it's random)

Issue with utf-8 has a lowest chance to be true, but there can be issue with mac's utf-8 with BOM (which macs hpfs adds automatically to every filename (which breaks svn btw), so if you have any non-ascii chars in asset filenames/pathnames, please remove them) (p.s. ignore that if sprockets don't use filename in hash generation)

This whole can also be caused by uglifier gem? The hashes are computed after uglifier/other filter does his job or before?

[hone]

@josh we run the precompile task now when people push their apps.

[kamui]

@tenderlove: Number of assets doesn't seem to matter. I dumped all 687 assets into the sample app and I referenced 4 of those assets (2 js, 2 css) in my sample view and they load fine with the correct md5 fingerprints. I bash'd into heroku and it looks like all the assets were compiled. I can only verify 4 of those assets have the correct fingerprint since I only have 1 view referencing them.

@NoICE Would that issue also affect image assets? They don't run through the uglifier/minification. In my app, images also have mismatching fingerprints.

[tenderlove]

I don't understand why fingerprints would be different when you generate them on your local machine vs generating on the server. Especially if you're using the same RAILS_ENV. It seems like rake assets:precompile would result in the same thing no matter which machine / VM.

@josh can you please help with this? We need an expert with the asset pipeline, and I think that's you.

[h0jeZvgoxFepBQ2C]

I have the same issue... I don't run it on heroku.. It worked fine when using rc5 -> switch to rc6 -> not working.. When downgrading it works again... Behaviour is the same on local machine(OSX) and server (debian) (both in production mode)..

[kamui]

Well, in my case I rely on heroku to precompile my assets, I don't do it on my development machine. So the same VM that's rendering the views, is also precompiling the assets. And yet the fingerprints still don't match.

[metaskills]

Just a thought. Perhaps this has something to do with ExecJS and it autodetecting the best runtime, in this case therubyracer which I believe is on the heroku stack? I remember bundling therubyracker when I first started playing with sprockets 2 and the pipeline since I wanted to locally have the same thing on heroku and let ExecJS auto detect the same runtime.

[h0jeZvgoxFepBQ2C]

It doesn't have anything to do with heroku from my point of view... As I said, I don't use heroku and have the same problem..

[metaskills]

@lichtamberg True, but the pattern holds true if it is something with uglifier => execjs => autodetected runtime being different from development to production. Just a thought.

[metaskills]

More spitballing, perhaps your compressor is different in development vs production?

config.assets.js_compressor  = ...
config.assets.css_compressor = ...

[metaskills]

More spitballing, config.assets.compress = true is set for production only. So it would make sense that production fingerprints would be different than local when you run the rake task which would be in default development mode?

[tenderlove]

I think @nzkoz may have found the issue, and it's likely what I suspected: file ordering.

Let's take a "hike" through our stack (LOL :heart: :heart:):

• hike contains an entries method that gets a list of files and does not sort them
• sprockets contains an entries method that calls in to trail. The entries are still not sorted.
• In sprockets file digest method, the entries method is called, joined, and added to the digest, but is still not sorted.

It seems possible that depending on your filesystem and ruby implementation, the digest could be different since the paths are never sorted.

The End.

/cc @josh @sstephenson

[metaskills]

Asset finger prints are the same no matter compress nor compressor. Duh! https://gist.github.com/1153023

Learned a lot tho :)

[josh]

@tenderlove sounds smelly, lets ensure those are sorted.