NodeJS buildpack doesn't play nice with multi buildpacks

[hakanw]

I'm using a setup with a .buildpacks file containing the standard nodejs buildpack, and the standard python buildpack.

My python+nodejs environment uses the node binaries during the deploy process, but unfortunately it can't find the node binary, even though it was just compiled by this buildpack. After a lot of debugging, I've nailed down the bug.

The nodejs buildpack is supposed to export its environment (and path to the node executable, node_modules etc) as part of its compile process. However, this is what the $PATH looks like for the python buildpack (after the nodejs buildpack is done). Note that weird tmp dir, which does not exist (and hence no node process can be found)!

'/app/.heroku/python/bin:/app/.heroku/vendor/bin::/usr/local/bin:/tmp/build_43e14da961254e10720c9bcb2f422630/.heroku/node/bin:/tmp/build_43e14da961254e10720c9bcb2f422630/node_modules/.bin:/app/bin:/app/vendor/bundle/bin:/app/vendor/bundle/ruby/2.2.0/bin:vendor/bundle/ruby/1.9.1/bin:/usr/local/bin:/usr/bin:/bin:bin:/tmp/codon/vendor/bin:/tmp/buildpackmCeYE/vendor/bpwatch:/tmp/buildpackmCeYE/vendor/pip-pop'

The fix here should be to somehow let this buildpack add /app/.heroku/node/bin etc to the environment that the next buildpack inherits.

[hunterloftis]

The node.js buildpack is by far the most commonly used buildpack with multi-buildpacks, so I'd be interested in seeing an actual test case demonstrating that node does not already export itself properly to the environment. I'd expect to see several dozen tickets about this if it had suddenly stopped working.

The path in the python buildpack (a temp build directory) is normal and expected; the build is run within a temporary directory, so node adds that directory to the path so that subsequent buildpacks can find the node binary.

I've created a simple test buildpack that you can insert after the node buildpack to verify all of this:

• https://github.com/hunterloftis/buildpack-node-env

[hakanw]

The problem we're experiencing might be a little special; it's that if you have two buildpacks, in this case nodejs and then python:

https://github.com/heroku/heroku-buildpack-nodejs.git
https://github.com/heroku/heroku-buildpack-python.git

And then deploy, the python process cannot find the node process during the deploy step. It needs to do this, because its asset pipeline stuff needs node for example to convert LessCSS => CSS, run uglifyJS and other things.

However, if you later do heroku run python do-the-asset-pipeline-stuff as a one-off command later, it works. In that case if can find node.

So the bug we've verified (which I can produce a minimal testcase if you don't believe me) is that the node process is not found on the $PATH during the deploy/compile step for the buildpacks that come after the nodejs buildpack.

Hope this makes things clearer! Sorry if I was not clear in my first report.

[alimony]

Here is a small node+django test case project that fails in the way described above:

https://github.com/alimony/heroku-multipack-fail

Just follow the README in that repo to try it out.

[hunterloftis]

Thanks for the test case @alimony. I'm also talking about that case - using node in subsequent buildpacks during the build phase.

Can you change the .buildpacks in that test case to inject the test buildpack between node and python? eg:

https://github.com/heroku/heroku-buildpack-nodejs.git
https://github.com/hunterloftis/buildpack-node-env.git
https://github.com/heroku/heroku-buildpack-python.git

If the node buildpack is successfully exporting, you should see the second buildpack (the check buildpack) able to access the node and npm versions as in its readme. If the check buildpack can successfully access these things, but the python buildpack cannot, that suggests that the problem isn't with node's export, but with the python buildpack.

[hunterloftis]

I suspect an external issue because:

1. The test buildpack, entered after the node buildpack, successfully finds and runs both the node and npm binaries (for me - please verify with your test case).
2. This is a very common use case for the node buildpack (compiling assets during build for other languages), and I haven't had any other reports of issues even though it's used thousands of times each day.

[alimony]

Tried this now, and yes the second buildpack is indeed succeeding while the third one fails. I'll continue investigating this on the assumption that something is wrong in e.g. django-pipeline regarding environment and path.

[alimony]

Well, it seems like PATH is actually not inherited by the Python buildpack.

The collectstatic step is run through sub-env, see this line. Looking at that util function, it filters out env vars that are on a blacklist, among them PATH.

There seems to have been problems related to this before, see this issue.

[alimony]

I've opened a new issue here: https://github.com/heroku/heroku-buildpack-python/issues/223