Bypassing the cache?

[mojodna]

How would I bypass the cache? I can think of 2 cases where I'd want to:

• my app isn't shrinkwrapped and I want to install updated dependencies (especially when pointing at the HEAD branch of a git repo)
• the buildpack made an invalid slug (see #46, but probably for other reasons)

[zeke]

I'm going to assume you're talking about the diet branch, as master currently doesn't do any caching. In the current implementation, the cache can be busted two ways:

1. A change in your package.json
2. A change in the node version that is resolved from your engines.node range, e.g. engines.node is 0.10.x and node stable has changed from 0.10.18 to 0.10.19 since you last deployed.

I'm thinking of changing this to something less magic, and respecting the shrinkwrap file for apps that use it. In this case the cache would be busted by either of the following, in order:

1. A change in npm-shrinkwrap.json
2. A change in package.json

Thoughts?

[mojodna]

Definitely npm-shrinkwrap.json, good call. Not busting the cache when node stable changes would be good too (even if node changes don't have impact, it's an occasion to unexpectedly pick up dependency changes).

Maybe the existence of a .bypass_cache file or something similar (to allow users to opt out entirely). It doesn't handle case-by-case opt outs, but I'm not sure that's possible... (user-env-compile could be added and an environment variable set, but that's a bit weird and has the side-effect of restarting the app before the build occurs.)

What do you think about pre-filling node_modules/ from the cache (if nothing's changed) and then running npm update against it? If it's a shrinkwrapped app, nothing will change, if there wasn't a node_modules/ to begin with (in the repo), the cached version will get updated. If there was a node_modules/ checked in, it might get unexpectedly updated, so maybe check for its existence before pulling from the cache and using that as an indication that npm update shouldn't be run?

[zeke]

I've been pondering the idea of a cache-bypassing file, but it feels a bit hacky. Ideally the cache Just Works™, and you should never need to bypass it. But yeah, you're probably right.

[svperfecta]

I seem to having problems with this today myself. I'm pointing to a specific commit of a Github repo in my package.json file using HTTPS. When I change the commitish, Heroku continues to install the old version; this happens whether I change my project's package.json file or not. I think it should be possible to manually clear the cache, and the cache should be improved to (presumably?) use more than just the version number of the NPM modules since that may stay constant between two different commits.

[danielsantiago]

I'm having the same problem today, and I'm not using module version, I'm using something like this: "module": "git+https://bitbucket.org/company/module.git#master".

I think the problem is related with this: https://github.com/heroku/heroku-buildpack-nodejs/pull/67

[svperfecta]

That's exactly what I was doing.

On Wed, Dec 4, 2013 at 1:28 PM, Daniel Santiago notifications@github.comwrote:

I'm having the same problem today, and I'm not using module version, I'm using something like this: "module": "git+ https://bitbucket.org/company/module.git#master".

— Reply to this email directly or view it on GitHubhttps://github.com/heroku/heroku-buildpack-nodejs/issues/48#issuecomment-29831108 .

[danielsantiago]

@genexp I create another issue (https://github.com/heroku/heroku-buildpack-nodejs/issues/76) because I think the problem we have need to be solve without the "bypassing the cache" feature.

[zeke]

Hey party people, we'll get this worked out eventually but if this is a blocker for you right now then please switch your apps over to the legacy branch for the time being. Instructions here: https://github.com/heroku/heroku-buildpack-nodejs#legacy-compatibility

[jamischarles]

+1 for this. We ran into this with the diet buildpack and stopped using it for that reason.

[zeke]

Good news everyone! As of npm v1.3.16, git URL dependencies are now updated on npm install.

[svperfecta]

Awesome! Thanks all!

On Thursday, December 19, 2013, Zeke Sikelianos wrote:

Good news everyone! As of npm v1.3.16https://github.com/isaacs/npm/releases/tag/v1.3.16, git URL dependencies are now updated on npm install.

— Reply to this email directly or view it on GitHubhttps://github.com/heroku/heroku-buildpack-nodejs/issues/48#issuecomment-30907532 .

Brian Corrigan MadGlory Interactive http://www.madglory.com Tw: @madgloryint Ph: 518.867.1439

[zeke]

This new version of node/npm is now available on Heroku: https://devcenter.heroku.com/changelog-items/374

[joesankey]

I have hit this issue with node 0.10.24. I had a dependency pointing at the HEAD of a git repo, it was not getting updated. The only workaround I had was to temp remove the dependency and push to Heroku to prune it from the cache...

[danielsantiago]

@joesankey you were using credentials in the git repo URL? For example: https://user:password@domain.com/....git? If so there was a problem with this type of URL in the NPM version which come in 0.10.24. It was fixed in the latest version of NPM and node 0.10.25.

[joesankey]

@danielsantiago nope, I was pointing to https://github.com/brunch/brunch/tarball/master

[danielsantiago]

@joesankey Try this: git+https://github.com/brunch/brunch.git#master

[joesankey]

@danielsantiago I've already cleared the dependency from the cache so can't try that out. Maybe if someone else hits this they can try.

[zeke]

I think @danielsantiago is on to something with the git+https suggestion: the recent bugfix in npm addresses the updating of git-based dependencies, but apparently pointing directly to http(s) tarballs may be a problem for npm.

[zeke]

After a few months in production, this doesn't appear to be a common issue for customers. There have also been a few developments that mitigate the problems:

• Automatic npm rebuild on node version change: https://devcenter.heroku.com/changelog-items/418
• git-based dependencies were semi-recently fixed in npm: https://devcenter.heroku.com/changelog-items/374
• The heroku-repo CLI plugin's heroku repo:purge_cache can be used to clear the build cache if absolutely needed.

I'm closing, but feel free to comment if this still feels like an issue.

[mojodna]

heroku-repo looks like it'll do the trick. Thanks for the pointer.

[svperfecta]

Awesome fix On Feb 14, 2014 11:14 PM, "Seth Fitzsimmons" notifications@github.com wrote:

heroku-repo looks like it'll do the trick. Thanks for the pointer.

Reply to this email directly or view it on GitHubhttps://github.com/heroku/heroku-buildpack-nodejs/issues/48#issuecomment-35146922 .