[Feature] Ability to specify Repository URL containing custom node executable

[Essenpreis]

We want to specify the node executable running on our landscape. So we have our own node fork and ideally want to reuse the node buildpack. Ideally we would like the CF node buildpack to support the specification of an external repository where node will be downloaded from. Java buildpack has a similar option where you can specify the JDK to be used.

Idea on implementation by slevine:

I really like the idea of us supporting (4), where you provide a supply.yml that contains overrides and additional values for the dependency-related hashes in the buildpack manifest.yml files, optionally with cached versions of those dependencies.

Relates to

• https://cloudfoundry.slack.com/archives/C02HWMDUQ/p1512827969000044
• https://cloudfoundry.slack.com/archives/C02HWMDUQ/p1512864711000013?thread_ts=1512864711.000013&cid=C02HWMDUQ

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. Unfortunately, the Pivotal Tracker project is private so you may be unable to view the contents of the story.

The labels on this github issue will be updated when the story is started.

[sclevine]

@idoru thoughts about this proposal? Seems like an excellent way to reduce forking to me.

[Essenpreis]

any update on this one?

[sclevine]

I'm still very interested in having this implemented to reduce forking. I'm prioritizing it now, but blocking it on additional discussion with the engineering team.

[Essenpreis]

Great. I see the item is implemented. One question.

As i understand the idea is that we would still supply a buildpack but in this case we would not have to fork but only have to provide an override.yml where we specify the url of our node executable. Correct?

How would I have to specify this executable when I want to achieve that it gets downloaded via an external url?

Is it something like?

nodejs: dependencies: (standard format + file key for caching) default_versions: http://mynodeexecutablehost/nodejs.exe

[heikoettelbruecksap]

Looking at https://github.com/cloudfoundry/nodejs-buildpack/blob/master/manifest.yml, I suspect (=> I hope somebody can confirm or correct this) the following:

default_versions:

• name: node version: my-custom-version (...) dependencies:
• name: node version: my-custom-version uri: https://.../node-custom-linux-x64-123456.tgz sha256: c8323162d0c2bc3d6ef1e9232e74791f45d7b426964b9a60dd7b7176ef6d664f cf_stacks:
• cflinuxfs2

[idoru]

@heikoettelbruecksap I think that looks correct, but the markdown formatting makes it hard to be sure. Here is an example of what override.yml should look like, we use this to test the feature: https://github.com/cloudfoundry/ruby-buildpack/blob/5029c0c076afe3151c50d16cc35be5157e45e9bf/fixtures/overrideyml_bp/override.yml

[heikoettelbruecksap]

@idoru You're right, I didn't realize the formatting issue :-( Thanks for pointing me to this example. (btw: strange to see an example pointing to node.js under the ruby-buildpack. I suspect this is what we usually call "historically grown"?)

[heikoettelbruecksap]

@sclevine Can you already share an idea of when / with which CF version the changes to the Node.js buildpack will be released?

[dgodd]

@heikoettelbruecksap nodejs 1.6.17 which has this new feature. We have not yet documented it well or created tooling to make it easy to use, (which is why we have not yet promoted it), but the feature is in the buildpack and you should feel free to use it and report your experience

[heikoettelbruecksap]

I just learned that I could already deploy an app with 1.6.17 before it becomes available as system buildpack in "my" CF landscape (e.g. by providing option -b with the buildpack's GitHub URL + branch/tag to "cf push"). Being rather new to buildpacks, I hope the following question is rather easy to answer:

Is there a way for us to create a custom buildpack on top of the community Node.js buildpack (specified as custom buildpack, as mentioned above)? The only required changed compared to the community version would be that we need to use another Node.js version as dependency. If someone knows whether this is possible, and also how to achieve it, I would be glad for respective hints. Thanks already in advance!

[dgodd]

@heikoettelbruecksap The new override.yml from a supply file is designed to mean that you don't need a custom buildpack.

So if you create a directory with override.yml file and setup for a supply buildpack in it (example to copyhttps://github.com/cloudfoundry/nodejs-buildpack/tree/master/fixtures/overrideyml_bp)

Build the above buildpack-packager build --cached, and upload it cf update-buildpack [OVERRIDEBP_NAME] -p [ZIP FILE]

Now push your app with

cf v3-push [APPNAME] -b [OVERRIDEBP_NAME] -b nodejs_buildpack

You should see that the values from override.yml are able to be used.

This should make for an easier workflow than using custom buildpacks

[heikoettelbruecksap]

@dgodd Thanks for explaining this. I must admit I'm very new to buildpacks, so apparently I misused the term "custom buildpack" to some extent. Is there any documentation explaining the different buildpack types, so that I understand more about what is a "custom buildpack" (assumption: a complete/independent buildpack), a "supply buildpack" (assumption: always a kind of delta to another buildpack), ...? Specific question: The approach you described implies that (1) the deployer of the app needs to use the "v3-push" (since v2's push doesn't support multiple buildpacks) and especially (2) that he/she needs to know and specify both buildpacks. Is it possible to somehow define the "baseline" buildpack (here: nodejs_buildpack) as part of the supply buildpack, so that the app can also be deployed with a simple cf push [APPNAME] -b [OVERRIDEBP_NAME]?

[dgodd]

@heikoettelbruecksap

Supply Buildpack: Is a buildpack used in a multi buildpack environment to supply binaries and libraries but not change your app, or determine how to make it run (the final buildpack would do that). You are correct, to use multi buildpack support you will need v3-push.

Custom Buildpack: is usually used to mean that someone forked one of our buildpacks and then changed it in some way (eg. changed dependencies or less frequently changed/added behavior)

When you say baseline, I think you mean, is it possible to have a buildpack which is really just two other buildpacks? The answer is sadly no, but has been discussed as a possibility for the future.

If all you want is to change dependencies, then you can fork our buildpack and go from there, but that means you need to maintain your fork. Or you could use the new override.yml support to create a supply buildpack with just your dependency changes in it, and the maintenance cost will hopefully be lower.

[marcrahnsap]

@dgodd

I created a supply buildpack with just one node version (8.1.2). Then a deployed an application using the overridebp & the default nodjs bp. I did not specify the node version in the apps package.json and hence it was deployed with version v8.9.4 and not v8.1.2. So my question is: Is it possible to specify my exchanged node version as default node version in the override.yml?

[dgodd]

@marcrahnsap The override.yml file adds dependencies to the list provided by the other buildpacks. So if you don't specify a version all of the versions provided by both the ovverride buildpack and (in your case) the node buildpack will be used to determine which one you want.

In your case specifying the default version in the override.yml should do the trick. eg. https://github.com/cloudfoundry/python-buildpack/blob/master/fixtures/overrideyml_bp/override.yml

[Essenpreis]

@dgodd. @sclevine
Thx for the lively discussion and support so far. In our use case it would be good however to ONLY allow the node version(s) we specify in the override.yml. All the other node versions coming from the original buildpack should NOT be possible being referenced and an error message should be returned upon push.

What do you think about that?

Background: In our supply buildpack we have added some additional features to node and we want to make sure our users in any case will benefit from these features or get an error if we cannot provide the features in the version they specified.