Error with CacheBusterFilter at End of Pipeline

[logicaltext]

Perhaps I'm doing this wrong, but I get an error with the following Assetfile:

output "public"
input "assets", "**/*.js" do
  concat "application.js"
  filter Rake::Pipeline::Web::Filters::CacheBusterFilter
end

I get an Errno::ENOENT:

No such file or directory - /.path/to/rake-pipeline-tmp-1/application.js (Errno::ENOENT)

This might be similar to issue #19 whereby the filter only works if the input files exist when the pipeline gets set up, though I can't be sure.

[dudleyf]

I suspect you're right. Filters shouldn't depend on their input files existing when they're instantiated. @jamesarosen can you take a look at this?

[wagenet]

It might also help to know a bit more about your app structure.

[logicaltext]

Sure. It's a simple Sinatra app. I was actually just getting started with Rake::Pipeline and that's when I came across the error. My app directory looks something like this:

    app
    ├── Assetfile
    ├── Gemfile
    ├── app.rb
    ├── assets
    │   └── javascripts
    │       ├── one.js
    │       └── two.js
    ├── config.ru
    └── views
    └── public

When I ran bundle exec rakep build using the filters mentioned earlier, I got the error. Everything was fine when just using ConcatFilter, or using ConcatFilter and another filter (UglifyFilter for example). It was just when using the CacheBusterFilter--after the other filters--that things didn't seem to work.

[jamesarosen]

I'll take a look. Hopefully soon.

[jamesarosen]

The CacheBusterFilter definitely doesn't do anything with its inputs on instantiation. One possibility is that its output_name_generator requires reading the file and that's happening before the file exists.

[jamesarosen]

Yes, the problem is that the filter's output_paths is being called before the filters before it have run. For example:

output "public"
input "lib" do
  match "**/*.rb" do
    concat "rakep-wf.rb"
    filter Rake::Pipeline::Web::Filters::CacheBusterFilter
  end
end

will fail because public/rakep-wf.rb does not exist when the filter's output_files is called (i.e. when Pipeline#generate_rake_tasks calls Pipeline#outputs, which calls Pipeline#output_paths, which calls the filter's generator).

[logicaltext]

Thanks for looking into this!

Yes, the problem is that the filter's output_paths is being called before the filters before it have run.

Is there a way around that? Because I would think that a common use case is to use the CacheBusterFilter as the last filter in a chain.

[jamesarosen]

You're using it exactly how I expected it to work when I wrote it. It's certainly possible that rakep exposes the actual unwritten file contents to the filter somehow, but nothing I've tried works. I'll keep investigating. I'd love any input from @wagenet or @wycats.

[dudleyf]

A filter's input files won't exist before generate_output is called, so there's currently no way to write a filter whose output file name depends on its input file contents. output_name_generator should be a pure function, depending only on the input file name. We need to be able to know what the output files of a filter are so we can construct the dependency tree before actually invoking the filter.

[jamesarosen]

In that case, we should probably remove this filter as it uses the file's contents (or at least mtime) to determine the output name. That's the whole point of the cache-busting.

[dudleyf]

It's a valid use case, and it makes me sad that it doesn't work :(

[logicaltext]

@jamesarosen, @dudleyf: Thanks for looking into this. The behavior makes sense given the nature of Rake's File Tasks. I suppose one way around it is to use multiple pipelines:

output "public"

input "assets" do
  match "**/*.js" do
    concat "application.js"
  end
end

input "public" do
  match "*.js" do
    filter Rake::Pipeline::Web::Filters::CacheBusterFilter
  end
end

And then have some other part of the deployment chain remove the application.js "artifact" from public.

[frodsan]

Any news in this issue?

[dudleyf]

@frodsan I don't see how we can make this work. I think we should remove the CacheBusterFilter after we release 0.6.0, unless someone else has a better idea.

[rlivsey]

Just ran into this myself, I take it nothing has changed in the past year to make this possible?

[rlivsey]

For anyone else who runs across this, I've come at this from a different angle and am generating the concatenated file name based on the git hash of the last change to a directory: http://livsey.org/blog/2013/05/16/cache-busting-with-rake-pipeline/