Inability to compile nondigest and digest assets breaks compatibility with bad gems

[gwcoffey]

For instance, the jquery_mobile-rails gem does not use image-url in its CSS. This is a bug really, but it "worked fine" in Rails 3. I assume they will fix the gem eventually. But in the meantime, in Rails 4, it is broken and as far as I can tell there's no way to work around it. The simple short term fix is to compile both digest and nondigest assets for this gem, but I don't see any way to do that. I can apparently only turn digest off or on globally for the application.

A rake assets:precompile:nondigest task or something similar would provide a workaround. But it seems to be gone now.

Is there a good way to deal with this, or do I just have to drop the gem and manage jquery mobile manually outside the asset pipeline, which is a pain?

[ryana]

Well that escalated quickly. Must have been one hell of an IRC exchange to get people this heated :) I would encourage everyone to set a calendar reminder for like a month from now and come back and re-read this whole thread noting the timestamps. There are probably some valuable lessons to be learned in how to better manage this from all sides in the future. I'll certainly be thinking about it.

Give that this is not going to make it's way in sprockets, a summary of available options:

• You can use this rake task to copy assets: https://github.com/rails/sprockets-rails/issues/49#issuecomment-20535134
• I wrote a monkeypatch (but is locked to a specific minor rev): https://gist.github.com/ryana/6049833
• You can use redirects: https://github.com/sikachu/sprockets-redirect
• You can use a gem: https://github.com/alexspeller/non-stupid-digest-assets

Happy hacking ;)

[guilleiguaran]

TL;DR: Digests are used for cache busting. Remember that if you use the non-digest assets and serve them with far-future expires headers, you will cause problems with cached assets if the contents ever need to change.

Rails embraces good defaults and in my book that doesn't count as a good default.

Anyway Rails is not stopping you if you want to re-add it using third party gems (like the gems suggested in this thread) or add a custom rake task to your app (as the custom task suggested in this thread).

[PhilipAnd]

I have, what might be a related issue.

I have a Sass file which references som SVG-images. All the SVG images are getting a hash appended to the file (as expected) when precompiling. The issue arises, since the compiled CSS file is referencing the file without the hash. Shouldn't the Sass compiler use the digest version of the images, which are referenced in the sass file, or should I e.g. use the gem https://github.com/alexspeller/non-stupid-digest-assets, to create two versions?

[amacneil]

+1

[Jell]

+1 on the issue, although I don't think the title is appropriate. I'm coming here from https://github.com/alexspeller/non-stupid-digest-assets

We're building an API and chose to prefix our assets with a version number (i.e v2/my_asset.js). This was working fine with Rails 3 but is broken in Rails 4 because non-digest versions are no longer generated.

I feel that generating both digest and non-digest versions of the assets should be straightforward and cost very little work or maintenance, if that is the case would you please reconsider your decision? I know too little about the codebase to do a PR myself.

[gwcoffey]

Jean-Louis:

Are you using the asset-url helper in your CSS files? What you describe should work fine with digested assets, and is a good example of why this change is probably beneficial. If your code doesn't work with digested assets, then you have a problem in your code, and it no longer "seems to work."

[Jell]

@gwcoffey like I'm saying, we're building an API and what we compile is a javascript and CSS which is used OUTSIDE our application. We absolutely need non-digest assets, because the clients that use our javascript are not in our control.

The version you see in the link is the version of our API, not the version of the asset.

[Jell]

@gwcoffey now that I'm re-ready my comment, I see how my message might have been confusing :p sorry about that

[nicholas-johnson]

Oh hey there, I'm a little late to the party here. It should probably be possible to have the option to expose non-digest assets as well as the digest versions via a config setting. I need to refer to assets externally, as I have several syndicated JavaScript widgets. I'd be happy to submit a patch if other people agree :)

[ryana]

@forwardadvance I think several dozen people agree, but unfortunately not the gatekeepers. Their advice is to fork sprockets and maintain your own distribution of it. There is a list of available options in this comment above: https://github.com/rails/sprockets-rails/issues/49#issuecomment-26020304

Have a good one!

[tilsammans]

http://codetunes.com/2013/we-released-rails-assets/

[igravious]

Hi there Rails Core Team. This just bit me. You got this one wrong. I find it hard to see how this wouldn't affect any decent sized project to be honest.

I use two different JQuery plugins. Both have CSS and use images.

In development mode, no problem. Images from vendor/assets/stylesheets/images work

In production mode, they get digested (once you learn the magic config.assets.precompile += %w[*.png *.jpg *.jpeg *.gif] incantation).

This places them in public/assets/images which is the right place but they are now fingerprinted so the CSS references break.

It is not an option to use image_tag because these are 3rd party libs and it's a royal pain. It is not an option to copy them to public/images because that's not really where they should be living.

Either we need a way to a) selectively copy over these types of assets without digests or b) ???

There's no way this issue should be closed.

[sevenseacat]

The solution AFAIK has always been to modify the plugins to update the asset references. It's a bit annoying, but it works fine.

[gwcoffey]

On Dec 17, 2013, at 6:31 AM, igravious notifications@github.com wrote:

It is not an option to use image_tag because these are 3rd party libs and it's a pain anyway. It is not an option to copy them to public/images because that's not really where they should be living.

Why not? If app/assets is for the asset pipeline, and you don't want to update the jquery library to use the asset pipeline, then I think they DO belong in public. In many cases, putting even the CSS or JS into the pipeline will still require you make changes to the library because they use relative paths that depend on their own directory structure. So in general, either a library IS or IS NOT asset-pipeline ready.

There are really a few ways to solve your issue:

1: Put the requisite parts into public just like we always did before the asset pipeline.

2: Use a gem someone else maintains. There are gems for, ie, jquery-mobile and jquery-ui. They do run a bit behind so if having the latest release right away is important for you this may be a problem.

3: Update the library to use asset-url. I do this semi-regularly now and it is usually very easy. Just search the css for "url" and revise. It is quite rare to have any asset dependency in the javascript.

3 seems totally stupid at first but then you do it a few times and realize it isn't that hard, it doesn't happen often, and once it is done, you're actually using the asset pipeline (with fingerprinting etc...) properly. So when you DO update the library, you're users will see the new assets right away without caching issues, which is one of the primary reasons the asset pipeline exists.

I opened this issue for exactly the reasons you describe, but I've been convinced that for the use case you're describing, I was wrong. For those running APIs, I think there's a legitimate concern. But for normal rails app development, it is better to fail fast when assets aren't being pipelined properly.

That said, I would very much prefer they didn't work in development either, to save the last-minute-oops-sorry-client hustle.

Geoff

[gwcoffey]

On Dec 17, 2013, at 5:32 PM, Andrew Kaspick notifications@github.com wrote:

@gwcoffey Regarding #3: Can you clarify how you're using asset-url. I have plain css files in vendor/assets/stylesheets that reference images that are digested through the pipeline. Are you renaming all of your css files to be sass files instead in order to get asset-url working correctly with image references? This is what I'm hung up on currently.

I only do this maybe every six months, but if I recall the general steps are:

1: Put the CSS file(s) into /app/assets/stylesheets (possibly in a sub-directory) and rename them .css.scss

2: Put the images into /app/assets/images/{whatever}

3: Search the CSS files for "url" and change them to asset-url calls. Ie:

url(../images/foo.png) => asset-url("/{whatever}/foo.png")

I think in most cases I've dealt with, that's all there is to it. If you have @imports, you may want to change them too, so they get compiled into the application.css. Also, don't forget the gem route. For instance, https://github.com/joliss/jquery-ui-rails is well maintained, and they do all this work for you.

Geoff

[akaspick]

I was under the assumption that all 3rd party libs should go into vendor/assets (that's been what we've been "told" to do since rails 3... I thought).

Since I'm trying to follow "conventions" (which are now different?), I've got everything working in vendor/assets with one extra addition... needing to do the following...

config.assets.precompile += %w(third_party.css {whatever}/*.png)

Is the recommended approach now to move 3rd party libs that we've made "pipeline ready" back into app/assets?

[gwcoffey]

For what it's worth, I put them in /vendor/assets as well. I was remembering wrong.

On Dec 18, 2013, at 7:14 AM, Andrew Kaspick notifications@github.com wrote:

I was under the assumption that all 3rd party libs should go into vendor/assets (that's been what we've been "told" to do since rails 3... I thought).

Since I'm trying to follow "conventions" (which are now different?), I've got everything working in vendor/assets with one extra addition... needing to do the following...

config.assets.precompile += %w(third_party.css {whatever}/*.png) Is the recommended approach now to move 3rd party libs that we've made "pipeline ready" back into app/assets?

— Reply to this email directly or view it on GitHub.

[stefanhendriks]

+1 for this issue. Wanted to make my 404/500/422 pages look the same as in the rest of the app. We use custom fonts and our app uses haml/sass everywhere. Some issues are fixed by using the asset-path within our sass/haml which is fine. However, I still need to reference from my static page to the end result (with digest).

I have put the files now in my rails app and let exceptions route through it.

[akaspick]

@stefanhendriks Put non-digested files in your public dir. If you need the special casing for your static pages, then that's where you should put them. Otherwise there are the other proposed gems/options provided in this discussion that should help.

[stefanhendriks]

@akaspick Actually I am using now dynamic pages and it works fine. The only remaining problem I have now is rspec complaining about missing templates once a 404 or a 500 error occurs. Even though I set config.consider_all_requests_local = false within test.rb.

Any idea? generating static versions of them (as Ryan does in his screencast) works (as the template exists again, and the test is passing), but the whole point is to do dynamic pages, so this solution is not sufficient.

[bennick]

Over the past year it seems that front-end libraries have been rapidly adopting the bower package manger. For may reasons bower is awesome. Mainly because it is similar to ruby's gem system. You can configure bower to install packages directly into the pipeline. Updating packages just takes altering a bower.json file and running bower install. Front end dependency life is just grand.

This change to sprockets-rails makes any images these bower installed packages use unfindable because they reply on non digested relative paths. Updating the packages directly just wrong and using a gem wrapper is, imo, a way of the past.

I will be using the rake task listed above but it would be nice if sprockets-rails could support this common scenario. As others have suggested having the ability to whitelist files or directories to keep non-digest assets, like a config.assets.nondigest option, would solve this.

Just some thoughts. Thanks to those who contribute to this project. I appreciate it.

[shioyama]

Completely agree with @bennick and strongly recommend having a look at using bower for managing front-end assets. I wrote this blog post on using bower with rails that might be useful to some people interested in doing that.

[gkop]

+1 that this issue ought to be reopened.

[guilleiguaran]

@alexspeller what if I rename my rake task to assets:precompile?

namespace :assets do
  task :precompile => :environment do
    assets = Dir.glob(File.join(Rails.root, 'public/assets/**/*'))
    regex = /(-{1}[a-z0-9]{32}*\.{1}){1}/
    assets.each do |file|
      next if File.directory?(file) || file !~ regex

      source = file.split('/')
      source.push(source.pop.gsub(regex, '.'))

      non_digested = File.join(source)
      FileUtils.cp(file, non_digested)
    end
  end
end

Then your argument about not working with heroku/capistrano, etc aren't valid anymore

[testingbot]

Thanks for the gem @alexspeller

[907th]

+100500

[breadnotes]

I have the same use case others have mentioned with error pages that are broken with the removal of non digest assets.

[TheR2]

Just started to convert my applications and stumbled on this problem too. So I thought I could put my 2 cents on problem.

We have this wonderfull mechanism of ruby gems, to include external ruby code in our projects and use them as engines in rails applications. But world is moving to more and more code being written in javascript and with this sprockets move it has become even harder to include third party javascript code transparently in our projects.

Moving static javascript to public folder, for using projects like ck_editor, is probaly OK if you use library only for your own, single project. But what if same code is beeing used by multiple projects. Than copying same code all over again becomes pain. That's why gems were invented for. Problem is that gems don't handle putting external javascript code into right destination directory. But sprockets know everything about it.

Funny thing is that code placed into assets folder works perfectly OK when in development, but in production and test nothing works at all, because the code doesn't get copied to destination folders.

I was already suggesting new directive eg. 'copy_directory' which would copy everything what is found in that directory to /assets/diretory when assets:precompile is run. This would help us a lot to gemify third party javascript libraries and include them into rails projects.

Or have I missed something that is already build into rails?

by TheR

[alexspeller]

@TheR2 gems work fine with the asset pipeline. However rubygems are not the best solution for packaging javascript libraries. Have a look into bower and bower-rails for packaging javascript libraries, they work quite well together.

[righi]

@guilleiguaran Thanks for the Rakefile solution, but unfortunately I found a problem with it after using it for a few days.

If you keep old copies of files in public/assets (the default behavior in Rails 4, even after an asset:clean) then you risk having old versions of your assets ending up in the non-digest files.

For example, imagine on Monday assets/javascripts/application.js compiles to public/assets/application-39f56e9c8b27cc62488e84f9f5cf7788.js, and after making changes on Tuesday it compiles to public/assets/application-1274e865b362e15aa2a29b1c9dbdf025.js. During precompile, both application-39f56e9c8b27cc62488e84f9f5cf7788.js and application-1274e865b362e15aa2a29b1c9dbdf025.js will get copied to application.js, with one clobbering the other. If you're lucky, the most recent file will be the one that wins. I wasn't so lucky today in a production environment. :-)

I switched to using https://github.com/alexspeller/non-stupid-digest-assets and that's working better for me.

[907th]

@righi @guilleiguaran I faced with that problem too. non-stupid-digest-assets gem helps with it.

[juniorjp]

+1, I'm using the provided rake task to solve the problem, but it should be a Rails default, this is a pain specially for beginners where things should work smoothly by default, it's the Rails philosophy.

If you do not want non-digested files to be generated you should config not to generate them, instead of forcing the rest of us to do the opposite, since we are like 99% of the cases.

[abarnet]

I think it's a good default not to generate non-digest because non-digest requires doubling the number of files written. However, there should be a config option to either generate all non-digest or s subset of non-digest files.

I think for a lot of people (myself included), there are a few files like generating error pages that need non-digest versions but all the rest get served via digest assets.

[juniorjp]

Everybody I know uses some files that references non-digested assets. I should not force the front-end developers of my project to always use rails helpers in their layouts to reference correctly digested assets, this violates Rails convention over configuration pattern.

There was no single project in rails 4( even my coming soon page) where this was not an issue.

[karmajunkie]

+1 on Rails core getting this wrong and not providing an officially supported route (via config option or official rake task) for remediating the problems caused by their heavy-handedness.

[willglynn]

+1. This needs to be a config option.

[wlipa]

Another use case to consider is HTML emails. There may be many emails sitting in people's mail readers that were sent before this change was made. Those emails probably use non-digested asset references because they were the stable choice for long lived documents, if you're willing to commit to not deleting or changing the assets. It's impossible to go back and change these emails because they were already sent.

I appreciate the hard-core digest only default, but I'd like to be able to make a hand tuned exception for certain assets.

[bcjordan]

+1. Also bit by the use case @wlipa mentioned.

[aggrhm]

+1 Also here just to complain. Seems very short-sighted to assume all 3rd-party assets will use the Rails pipeline hooks like 'asset-url' when referencing images/fonts. Why not just provide both?

Please make a config option so 'non-stupid-digest-assets' is not basically a requirement for anyone building a Rails 4 app.

[celsodantas]

Question for the team: How can you build a JS API with Rails 4? It doesn't seem possible without generating a non-digested version of the file.

[alexspeller]

@celsodantas I think you are confusing the asset pipeline with json responses to controller actions. Nothing in this discussion is relevant to building APIs.

[celsodantas]

I'm not talking about controllers responding to JSON, I'm talking about writing JS code to publish to clients of my application to use (like a SDK). In these case we need to give the clients a fixed path.

Example: http://docs.shopify.com/embedded-app-sdk/initialization

[alexspeller]

@celsodantas oh, I see what you mean. The techniques to achieve this are discussed in great detail above.

[celsodantas]

@alexspeller, I saw the "solutions"/workaround. I just think it's not a wise idea to not add an option to add the non digested files back.

[ryana]

I find it "ironical" that this issue just bit me again 8 months after I first expressed my opinion that this should be changed.

I'm using one of Symbolset's awesome font icon sets: https://symbolset.com/icons/gizmo

I copied their webfonts and CSS into my assets directory. Here is a segment of the stylesheet that accompanies the font:

stylesheets/ss-social.css:15:  src: url('/assets/webfonts/ss-social-circle.eot');
stylesheets/ss-social.css:16:  src: url('/assets/webfonts/ss-social-circle.eot?#iefix') format('embedded-opentype'),
stylesheets/ss-social.css:17:       url('/assets/webfonts/ss-social-circle.woff') format('woff'),
stylesheets/ss-social.css:18:       url('/assets/webfonts/ss-social-circle.ttf') format('truetype'),
stylesheets/ss-social.css:19:       url('/assets/webfonts/ss-social-circle.svg#SSSocialCircle') format('svg');

ss-social.css is then imported into my application.css.scss. Now, it's a pretty safe bet these assets are never going to change. But sure enough, after upgrading to Rails4, all my social icons go bye bye because the ss-social-circle.(eot|woff|ttf|svg) files no longer exist in my asset directory.

I know how to fix this. I think we are saying that the best practice here is to go into ss-social.css and change the call from url() to asset-url(). Ok. That's fixes it. Ignoring the fact it's annoying to have to do that an already minified CSS file (which this isn't, but others are), it does solve the problem.

But that's not what concerns me here. What concerns me here is that we've now built something into Rails that by default works correctly in the development environment and breaks in the production environment, and there is literally no way you would know it until you run the app in the production environment.

This doesn't just happen on Heroku btw. It also happens on dokku/docker since the assets directory starts fresh every time a container is started. On deploys directly to a linux box (via Capistrano et al), these issues may be hidden if the assets directory is old and still contains non-digested assets from before this change was instituted.

What's maybe even worse is that this default also works in the test environment, so if you're going out of your way to generate screenshots in Poltergeist to ensure stuff looks good, you're still going to get bit once you go to production.

Opinions about caching best practices and performance aside, I think we can all agree that having a default that introduces an error when going from dev -> production is not something that can stand.

I think the cleanest way to address this is to take all the assets generated by the asset compilation (with their digests) and simply make copies of them without digests in the name. We don't need to recompile. The copied files will still contain references to digested assets, but that's ok I think. This should be the default behavior.

If you're a contributor on this project, please let me know if you'd accept a pull request of this nature.

Thanks for reading this whole thing!

[bennick]

@ryana I agree the default should be to have non digest assets with an option to exclude them. To me this change feels like a big case of premature optimization.

[karmajunkie]

@ryana @bennick It doesn't bother me that the default is for digested assets. I think convention over configuration is perhaps Rails greatest contribution to the web development zeitgeist, and if most people need digests only, great, I have no problem making a slight modification to my environment settings upon upgrade.

What I find incredibly irksome, heavy-handed, and a bit arrogant is that the maintainers of the gem have determined that they know what's better for my use-case than I do, and that there is in fact no reason for me to have an officially supported method of generating non-digested assets. The hacks detailed in this thread are just that: hacks. If it works, and its apparently easy enough to do that nobody on the maintenance team feels the least bit troubled by requiring everyone in this thread (realizing that the commenters are likely the tip of the iceberg) to use them, then why such resistance to adding it as an officially supported option so that we have an interface we can rely on in future versions of the framework? This isn't merely a matter of a vocal minority clamoring for a feature no one needs. This is a feature which was official, and as such relied upon, by many people, and its summary removal (without even a deprecation process, AFAICT—correct me if i'm wrong) was absolutely the wrong move, and I'm flabbergasted at the resistance to adding it back when there's zero impact to anyone else.

[gwcoffey]

I completely agree that it is bad and broken that things work in development and not in production. It makes sorting out asset problems a big pain and it needs to be fixed. Although I think the fix should be that they don't work in development, not that they do in production.

Bear in mind in your use case with a minified webfont CSS, you have another option you didn't mention: Put the assets in public.

By putting them into the asset pipeline, it is reasonable for you to assume that you are getting the actual primary benefit of the pipeline, which is fingerprinting to ensure proper assets are shown with each version of the site + excellent cache support. In the same way that it is wrong for things to work in development and not in production, it is wrong for them to appear to work in production when they really aren't working as expected.

If you want fingerprinted safe assets, put them in the asset pipeline and adjust the code as needed so that it works.

If you want to simply drop the assets into your project and not worry about versioning, then put them in public.

I suspect in rails 3 there are thousands of sites out there with willy-nilly asset handling that doesn't really do what the pipeline promises because people running the site don't even realize they aren't doing it right and it "seems fine" to them. It is better to fail fast in this case in my mind.

NB: None of this addressed the other use case people have mentioned where they want combined, minified JS and/or CSS for use in non-rails pages. I'm not sure how best to deal with that, and it would probably be good if Rails had a simple sanctioned way to do it.

Geoff

[ryana]

@gwcoffey yeah good points. I would support a solution that made this not work in development. Of course, that would probably break a lot more applications :)

Perhaps we make the asset pipeline throw an error saying "Hey man you need to use asset-url() here" when it encounters a "url()" call to a relative path in an asset directory, that might be the right way to handle this.

[bennick]

@ryana A problem with that approach, as discussed above, is that you should not be modifying 3rd party front end libraries. These libraries also conflict with @gwcoffey idea of putting assets into public unless you are will to store all of your 3rd part library code directly in public. This, however, defeats much of the purpose of using the asset pipeline.