Add support for github release fetched from wget/curl in Dockerfiles

[gberche-orange]

• As a docker file maintainer,
• in order to automate bump of dependent github release binaries fetched using wget/curl
• I need github releases fetched from wget or curl to be bumped by dependabot

Sample docker fragment from Dependabot Docker file

# Install Elm 0.18 and Elm 0.19
ENV PATH="$PATH:/node_modules/.bin"
RUN npm install elm@0.18.0 \
  && wget "https://github.com/elm/compiler/releases/download/0.19.0/binaries-for-linux.tar.gz" \
  && tar xzf binaries-for-linux.tar.gz \
  && mv elm /usr/local/bin/elm19 \
  && rm -f binaries-for-linux.tar.gz

# Install Go and dep
RUN curl https://dl.google.com/go/go1.13.4.linux-amd64.tar.gz | tar -xz -C /opt \
  && wget -O /opt/go/bin/dep https://github.com/golang/dep/releases/download/v0.5.4/dep-linux-amd64 \
  && chmod +x /opt/go/bin/dep \
  && mkdir /opt/go/gopath
ENV PATH=/opt/go/bin:$PATH GOPATH=/opt/go/gopath

In the examples above, this feature would only be useful in conjonction with support for other ecosystems in docker files such as #2129 for apt or npm, or go

[thepwagner]

👋 in other repos I've solved this by pushing the version out of Dockerfile and into an ecosystem that Dependabot supports natively:

e.g.

ELM_VER=$(jq -r '.dependencies["elm"]' package.json)
wget "https://github.com/elm/compiler/releases/download/${ELM_VER}/binaries-for-linux.tar.gz"

That gives Dependabot hooks into a supported ecosystem to push updates (package.json), without the mess of extracting installations from the Dockerfile text.

[gberche-orange]

thanks @thepwagner ! Would you have pointers to some public repos as to get deeper inspiration and apply this pattern to my docker files ?

[thepwagner]

It's a contrived example as it's still using pip to perform the install, but https://github.com/thepwagner/docker-awscli/blob/master/Dockerfile is a sample of the idea:

• the version I want to pin is stored in requirements.txt
• dependabot can open PRs updating that, which affect the docker container: https://github.com/thepwagner/docker-awscli/pull/2

Less public usage uses go.mod and projects that release binaries. Not specifically this project, but you can imagine https://github.com/hashicorp/terraform/releases -> https://www.terraform.io/downloads.html URLs.

Definitely not the general solution proposed, but hopefully some inspiration.

[gberche-orange]

thanks for the additional details. Would the following be a fair an accurate rephrasing of your pattern ?

If your dependency installed from Docker file is also managed by another ecosystem, then use that ecosystem to keep the version number updated by dependabot in this ecosystem, and then import the version number in your Docker file to install it, potentially independently of the ecosystem tooling, e.g. using curl and tar

Taking as an example the following Docker image orange-cloudfoundry/orange-cf-bosh-cli/Dockerfile, this would likely require the following ecosystems

• go.mod
  • https://github.com/cloudfoundry-incubator/bosh-backup-and-restore/
  • https://github.com/vmware-tanzu/velero
• go dep
  • https://github.com/cloudfoundry/bosh-cli
  • https://github.com/cloudfoundry/cli/
• some apparently missing ecosystems
  • go-based https://github.com/thomasmitchell/bosh-complete (using version.go
  • C and Makefile https://github.com/stedolan/jq
  • Shell and makefiles https://github.com/rupa/z

[gberche-orange]

@thepwagner I experimented with gomod as a way to fetch github release tags in the following test repo https://github.com/gberche-orange/dependabot-test/. However, dependabot reject tags which don't follow the gomod naming convention: v-X.Y.Z (x.Y.Z being semver), in particular semver compatible tags without the v prefix.

I haven't found another solution for fetching version numbers for github releases such as https://github.com/cloudfoundry/routing-release/releases of format 0.207.0

Did you find a workaround for this use-case ?

[gberche-orange]

Another side effect of the workaround using gomod ecosystem to bump github release in a Docker file is the resource usage that this implies on dependabot side, with associated implications in terms of scalability, economical and environmental costs.

The update scanning of the following 4 dependencies at https://github.com/gberche-orange/dependabot-test/blob/b7ead8a58906c57661efb785cf2818f09edd77c0/go.mod#L5-L18 take about 3 mins elapsed time to dependabot during it triggers 338 github api calls. See full log at https://gist.github.com/gberche-orange/69cd2d1b8f651bed0b229ce8744cf77d

require (
    github.com/cloudfoundry-incubator/bosh-backup-and-restore v1.7.0
    github.com/thomasmitchell/bosh-complete v1.1.0
    github.com/stedolan/jq v0.0.0-20150818062500-a5b5cbefb839
[...]

Hopefully, a more native "github release" ecosystem would be able to just need 1/few github api call for each dependency to lookup newly available github releases/commits.

[johnboyes]

Hi @thepwagner and @gberche-orange, I've been using a similar mechanism to keep those dependencies up to date that Dependabot cannot update automatically. I'll share it with you in case it's useful (feedback welcome, too 🙂 ).

I use a GitHub Action yml file which exists just so that PRs are automatically created by Dependabot when the dependencies have a new version. Then in the same PR that Dependabot creates, I update the version manually in the actual place where it's needed (here's an example). In practice over the last few months I've not found having to do that to be an inconvenience - I make the update in the GitHub editor in the browser, so it's very quick.

Here's my documentation of this approach, FYI.

[thepwagner]

@johnboyes 😍 that's a cool hack! Since the PRs to update your "Actions workflow" are opened by Dependabot, you could even have an Actions workflow triggered to update the version in the actual place(s) - but as you mention it's quick enough to do manually.

Documenting your project's updates, and using scheduled issues as a reminder to refresh are neat patterns too!

---

https://github.com/thepwagner/action-update-dockerurl is an additional unsupported solution to this problem that runs on GitHub Actions, with an emphasis on projects that are hosted on github and attach a binary to their releases. If you're hacking in this space it may be another starting point. It scratched my itch+config, YMMV.

[jeffwidman]

I'm going to close this as the original ask of extracting versions from wget/curl commands in a dockerfile makes sense to a human, but would be painful and very brittle to try to tackle programmatically.

The workarounds mentioned above are definitely better at making it so Dependabot has a clean API for knowing what is bumpable.

[thepwagner]

For anyone searching for more workarounds, I use Renovate's RegEx manager for this now. To resolve the brittleness, it only works for artifacts attached to GitHub releases, not any URL.

Configuration:

    {
      fileMatch: ["(^|/)Dockerfile$"],
      matchStrings: [
        "# renovate: datasource=(?<datasource>.*?) depName=(?<depName>.*?)( versioning=(?<versioning>.*?))?\\sARG .*?_VERSION=(?<currentValue>.*)\\sARG .*?_CHECKSUM=(?<currentDigest>.*)\\s",
      ],
      versioningTemplate:
        "{{#if versioning}}{{versioning}}{{else}}semver{{/if}}",
    },

Sample PR

[jeffwidman]

Oh, that is quite clever!

I can envision us implementing something along these lines at some point, so I'll re-open this.