Allow runtime install of tools

[viceice]

Make all possible tools runtime installable.

TODO

• [x] #121
• [x] #197
• [x] #213
• [x] bundler #251
• [x] cocoapods #251
• [x] composer
• [x] docker #651
• [x] #118
• [x] elixir #376
• [x] #318
• [x] #200
• [x] git-lfs #652
• [x] #653
• [x] #119
• [x] gradle
• [x] hashin (pip_requirements) #639
• [x] #199
• [x] java
• [x] #178
• [x] nix #573
• [x] #217
• [x] npm
• [x] pipenv #639
• [x] pip-tools (pip-compile) #639
• [x] #120
• [x] pnpm
• [x] poetry
• [x] #633
• [x] #404
• [x] #475
• [x] #571
• [x] sbt
• [x] scala
• [x] #576
• [x] #181, #198
• [x] yarn

[rarkins]

I don't think it really matters much whether it's in /home/user or /usr/local however /usr/local would probably be better for OpenShift, where the UID is random.

[viceice]

We like to implement a new binarySource=install where renovate install tools dynamically at runtime.

Use-case is to have smaller docker image used in kubernets, which can download missing tools on demand.

ref #96

https://github.com/renovatebot/docker-buildpack/issues/144#issue-802657002

[rarkins]

@viceice should we break these into separate issues to research and complete one by one? I'd like to know early if we have any "blockers", too

[rarkins]

Thinking it through, we'll need:

1. OS pre-requisites which need to be built into the base image, mostly stuff that requires apt
2. "prebuilt" versions of most languages (we do not want to compile them at install time) and some tools. For these we do the prebuilding ourselves, currently publishing to GitHub Releases
3. most package manager tools dynamically installable at run-time, from their origin internet sources

So in terms of research, it's a matter of working out which falls in (2) and which in (3).

Pre-built:

• [ ] node?
• [x] php
• [x] python
• [x] ruby

Dynamically-installable:

• ?

(let's update the above lists as we go)

@viceice any good way to build up a definitive list - everything here? https://github.com/containerbase/buildpack/tree/main/src/usr/local/buildpack/tools

[viceice]

No easy way beside to check every tool. I think most tools can be installed as runtime if we use a file download.

The morst rework is needed for erlang and java, as they are installed via apt. For java i already have a solution in mind. erlang needs some research

[rarkins]

Portable java seems possible via https://adoptopenjdk.net/releases.html?variant=openjdk11&jvmVariant=hotspot

Edit: Seems they changed project name and now have binaries in GitHub releases: https://github.com/adoptium?q=binaries&type=source&language=&sort=

[rarkins]

I think erlang is still pretty niche and I could live with it being unsupported for this for a while

[viceice]

Portable java seems possible via https://adoptopenjdk.net/releases.html?variant=openjdk11&jvmVariant=hotspot

Edit: Seems they changed project name and now have binaries in GitHub releases: https://github.com/adoptium?q=binaries&type=source&language=&sort=

I know, Some issues about adopt / adoptium:

• weird versioning scheme: jdk-11.0.9+11, jdk-11.0.9+11.1, jdk-11.0.9.1+1, jdk-11.0.10+9, jdk-11.0.11+9 . JDK8 has a even more weired versioning scheme
• then new adoptium don't provide older releases, so they only have v8, v11 and v16+

So i'm thinking if we should add a adoptium-java datasource and versioning to renovate, so we can directly use the binary releaes / adoptium api?

[viceice]

Maybe we should use the api at https://api.adoptium.net/ to get required version(s)

[viceice]

OK, if we use the api, we can use semver, at the api has a semver v2 field.

https://api.adoptopenjdk.net/v3/info/release_versions?architecture=x64&heap_size=normal&image_type=jdk&jvm_impl=hotspot&os=linux&page=0&page_size=10&project=jdk&release_type=ga&sort_method=DEFAULT&sort_order=ASC&vendor=adoptopenjdk

{
  "versions": [
    {
      "build": 13,
      "major": 8,
      "minor": 0,
      "openjdk_version": "8u181-b13",
      "security": 181,
      "semver": "8.0.181+13"
    },
    {
      "build": 12,
      "major": 8,
      "minor": 0,
      "openjdk_version": "8u192-b12",
      "security": 192,
      "semver": "8.0.192+12"
    },
    {
      "build": 8,
      "major": 8,
      "minor": 0,
      "openjdk_version": "8u202-b08",
      "security": 202,
      "semver": "8.0.202+8"
    },
    {
      "build": 3,
      "major": 8,
      "minor": 0,
      "openjdk_version": "8u212-b03",
      "security": 212,
      "semver": "8.0.212+3"
    },
    {
      "build": 4,
      "major": 8,
      "minor": 0,
      "openjdk_version": "8u212-b04",
      "security": 212,
      "semver": "8.0.212+4"
    },
    {
      "adopt_build_number": 1,
      "build": 10,
      "major": 8,
      "minor": 0,
      "openjdk_version": "1.8.0_222-b10",
      "security": 222,
      "semver": "8.0.222+10.1"
    },
    {
      "adopt_build_number": 1,
      "build": 9,
      "major": 8,
      "minor": 0,
      "openjdk_version": "1.8.0_232-b09",
      "security": 232,
      "semver": "8.0.232+9.1"
    },
    {
      "adopt_build_number": 1,
      "build": 8,
      "major": 8,
      "minor": 0,
      "openjdk_version": "1.8.0_242-b08",
      "security": 242,
      "semver": "8.0.242+8.1"
    },
    {
      "adopt_build_number": 1,
      "build": 9,
      "major": 8,
      "minor": 0,
      "openjdk_version": "1.8.0_252-b09",
      "security": 252,
      "semver": "8.0.252+9.1"
    },
    {
      "build": 10,
      "major": 8,
      "minor": 0,
      "openjdk_version": "1.8.0_262-b10",
      "security": 262,
      "semver": "8.0.262+10"
    }
  ]
}

[viceice]

{
  "versions": [
    {
      "build": 28,
      "major": 11,
      "minor": 0,
      "openjdk_version": "11+28",
      "security": 0,
      "semver": "11.0.0+28"
    },
    {
      "build": 13,
      "major": 11,
      "minor": 0,
      "openjdk_version": "11.0.1+13",
      "security": 1,
      "semver": "11.0.1+13"
    },
    {
      "build": 7,
      "major": 11,
      "minor": 0,
      "openjdk_version": "11.0.2+7",
      "security": 2,
      "semver": "11.0.2+7"
    },
    {
      "build": 9,
      "major": 11,
      "minor": 0,
      "openjdk_version": "11.0.2+9",
      "security": 2,
      "semver": "11.0.2+9"
    },
    {
      "build": 7,
      "major": 11,
      "minor": 0,
      "openjdk_version": "11.0.3+7",
      "security": 3,
      "semver": "11.0.3+7"
    },
    {
      "adopt_build_number": 1,
      "build": 11,
      "major": 11,
      "minor": 0,
      "openjdk_version": "11.0.4+11",
      "security": 4,
      "semver": "11.0.4+11.1"
    },
    {
      "adopt_build_number": 1,
      "build": 10,
      "major": 11,
      "minor": 0,
      "openjdk_version": "11.0.5+10",
      "security": 5,
      "semver": "11.0.5+10.1"
    },
    {
      "adopt_build_number": 1,
      "build": 10,
      "major": 11,
      "minor": 0,
      "openjdk_version": "11.0.6+10",
      "security": 6,
      "semver": "11.0.6+10.1"
    },
    {
      "adopt_build_number": 1,
      "build": 10,
      "major": 11,
      "minor": 0,
      "openjdk_version": "11.0.7+10",
      "security": 7,
      "semver": "11.0.7+10.1"
    },
    {
      "build": 10,
      "major": 11,
      "minor": 0,
      "openjdk_version": "11.0.8+10",
      "security": 8,
      "semver": "11.0.8+10"
    }
  ]
}

sometimes we need to filter duplicates, eg: 11.0.2+7 and 11.0.2+9

[rarkins]

Here's now I'd like to approach this:

• Separate issue for each tool/language
• Issues assigned to whoever is working on them
• Ensure tests are updated for each tool as part of the same PR (i.e. it installs and verifies as non-root)

[viceice]

All tools are now runtime installable. Only git is left. Not possible yet. Will open a new issue for that