docker: Replacement causes existing dependencies to use wrong digest

[uhthomas]

How are you running Renovate?

Mend Renovate hosted app on github.com

If you're self-hosting Renovate, tell us what version of Renovate you run.

34.125.1

If you're self-hosting Renovate, select which platform you are using.

Bitbucket Server

If you're self-hosting Renovate, tell us what version of the platform you run.

N/A

Was this something which used to work for you, and then stopped?

I never saw this working

Describe the bug

We're replacing an old image with a new one, but the digest for the new image is different.

This is fine and the replacement PRs look fine too, but Renovate seems to assume the replacements are also aliases.

This means that Renovate is now opening PRs for the old image with the digest for the new image...

To help illustrate this, here are our two images:

• legacy, digest: 44dea30fe85df1f3d554cee602fa69d553da7ab746c8c9f20defb1f1fc3e03b8
• current, digest: e742813303ad00bbe1409ea01eb909cb67818a8fbaae8c07df79036a7dea3e1d

Renovate is opening PRs which look like:

-FROM legacy@sha256:44dea30fe85df1f3d554cee602fa69d553da7ab746c8c9f20defb1f1fc3e03b8
+FROM legacy@sha256:e742813303ad00bbe1409ea01eb909cb67818a8fbaae8c07df79036a7dea3e1d

This is wrong and unexpected. I've read these issues, but they're not completely the same:

• https://github.com/renovatebot/renovate/issues/20128
• https://github.com/renovatebot/renovate/issues/19999

Relevant debug logs

Logs

``` Copy/paste the relevant log(s) here, between the starting and ending backticks ```

Have you created a minimal reproduction repository?

https://github.com/renovate-reproductions/27728-docker-replacement-digest

[uhthomas]

Please see this snippet from the logs, with relevant modifications for privacy.

"deps": [
        {
                "autoReplaceStringTemplate": "example.com/legacy:{{#if newValue}}{{newValue}}{{/if}}@{{#if newDigest}}{{newDigest}}{{/if}}",
                "currentDigest": "sha256:46cea30fe85df1f3d554cee602fa69d553da7ab746c8c9f20defb1f1fc3e03b8",
                "currentValue": "some-tag",
                "currentVersion": "some-tag",
                "datasource": "docker",
                "depIndex": 0,
                "depName": "example.com/legacy",
                "depType": "final",
                "fixedVersion": "some-tag",
                "registryUrl": "https://example.com",
                "replaceString": "example.com/legacy:some-tag@sha256:46cea30fe85df1f3d554cee602fa69d553da7ab746c8c9f20defb1f1fc3e03b8",
                "versioning": "docker",
                "warnings": [],
                "updates": [
                        {
                                "updateType": "replacement",
                                "newName": "example.com/current",
                                "newValue": "some-tag",
                                "newDigest": "sha256:174f813303ad00bbe1409ea01eb909cb67818a8fbaae8c07df79036a7dea3e1d",
                                "branchName": "renovate/example-com-legacy-replacement"
                        },
                        {
                                "updateType": "digest",
                                "newValue": "some-tag",
                                "newDigest": "sha256:174f813303ad00bbe1409ea01eb909cb67818a8fbaae8c07df79036a7dea3e1d",
                                "branchName": "renovate/example-com-legacy-some-tag"
                        }
                ]
        }
],

[github-actions[bot]]

Hi there,

Get your issue fixed faster by creating a minimal reproduction. This means a repository dedicated to reproducing this issue with the minimal dependencies and config possible.

Before we start working on your issue we need to know exactly what's causing the current behavior. A minimal reproduction helps us with this.

To get started, please read our guide on creating a minimal reproduction.

We may close the issue if you, or someone else, haven't created a minimal reproduction within two weeks. If you need more time, or are stuck, please ask for help or more time in a comment.

Good luck,

The Renovate team

[uhthomas]

I tried to create a reproduction repository but Renovate isn't able to create any PRs at all.

https://github.com/uhthomas/renovate20304

https://app.renovatebot.com/dashboard#github/uhthomas/renovate20304/1001308789

WARN: Error updating branch: update failure(branch="renovate/alpine-replacement")

[viceice]

I tried to create a reproduction repository but Renovate isn't able to create any PRs at all.

https://github.com/uhthomas/renovate20304

https://app.renovatebot.com/dashboard#github/uhthomas/renovate20304/1001308789

WARN: Error updating branch: update failure(branch="renovate/alpine-replacement")

provide more context debug logs

[uhthomas]

provide more context debug logs

I really am not sure what more to provide. This is the hosted version of Renovate, so all the debug logs are:

https://app.renovatebot.com/dashboard#github/uhthomas/renovate20304/1001308789

The repository only contains a renovate.json and a Dockerfile which reproduce the issue(s).

https://github.com/uhthomas/renovate20304

The dependency dashboard also shows that it can't make a PR for some reason.

https://github.com/uhthomas/renovate20304/issues/2

[uhthomas]

If it helps, I've copied the output from the linked debug log and put it in this gist.

[viceice]

the logs doesn't contain the warning, so wrong run

[uhthomas]

the logs doesn't contain the warning, so wrong run

Yes they do.

https://gist.github.com/uhthomas/da50bee0a31a3e0755bc5e6644780f34#file-renovate-debug-log-L254

[viceice]

looks like a duplicate of

• https://github.com/renovatebot/renovate/issues/20128

[uhthomas]

@viceice I did see that issue, but it's not quite the same. The replacement rules work as expected in our case, but it just seems that for some reason Renovate will now open PRs with the replacement digest for the original image.

I am struggling to demonstrate this as the hosted version of Renovate is running into a different(?) issue in the debug log.

[uhthomas]

Okay @viceice, I have a full working reproduction now.

Repository: https://github.com/uhthomas/renovate20304/tree/445bbce6bf13dc2da5f4350e10c74d8f9d946601

Logs: https://app.renovatebot.com/dashboard#github/uhthomas/renovate20304/1001783011 (gist).

Here's what Renovate is trying to do:

See: https://github.com/uhthomas/renovate20304/pull/3/files

It's trying to update the original image with a digest from the replacement image...

I can confirm it's incorrect as docker pull does not work, whereas it does work if I use the digest with the replacement image and version.

❯ docker run gcr.io/distroless/base-debian10:debug@sha256:5f1e20553a35537925dafe9b6a4d6805deecfbb76c2e419ef72d9023166ec09d
Unable to find image 'gcr.io/distroless/base-debian10:debug@sha256:5f1e20553a35537925dafe9b6a4d6805deecfbb76c2e419ef72d9023166ec09d' locally
docker: Error response from daemon: manifest for gcr.io/distroless/base-debian10@sha256:5f1e20553a35537925dafe9b6a4d6805deecfbb76c2e419ef72d9023166ec09d not found: manifest unknown: Failed to fetch "sha256:5f1e20553a35537925dafe9b6a4d6805deecfbb76c2e419ef72d9023166ec09d" from request "/v2/distroless/base-debian10/manifests/sha256:5f1e20553a35537925dafe9b6a4d6805deecfbb76c2e419ef72d9023166ec09d".
See 'docker run --help'.
❯ docker run gcr.io/distroless/base-debian11:nonroot@sha256:5f1e20553a35537925dafe9b6a4d6805deecfbb76c2e419ef72d9023166ec09d
Unable to find image 'gcr.io/distroless/base-debian11:nonroot@sha256:5f1e20553a35537925dafe9b6a4d6805deecfbb76c2e419ef72d9023166ec09d' locally
gcr.io/distroless/base-debian11@sha256:5f1e20553a35537925dafe9b6a4d6805deecfbb76c2e419ef72d9023166ec09d: Pulling from distroless/base-debian11
5f80a38cb015: Pull complete 
9fb3436fe506: Pull complete 
4f70717a8bb3: Pull complete 
Digest: sha256:5f1e20553a35537925dafe9b6a4d6805deecfbb76c2e419ef72d9023166ec09d
Status: Downloaded newer image for gcr.io/distroless/base-debian11@sha256:5f1e20553a35537925dafe9b6a4d6805deecfbb76c2e419ef72d9023166ec09d

Is what I've written clear?

[viceice]

please write expected and current behavior to the repo readme, then we'll fork it

[uhthomas]

Will do.

[uhthomas]

@viceice Done!

[viceice]

ok, i see a pin PR but no replacement PR. is this only happen on self hosted renovate or also on the GitHub app?

@rarkins @secustor @JamieMagee it seems the pin PR uses the replacement data for pin and blocks any other PR.

i think the pin update type shouldn't use the replacement rule data.

[uhthomas]

I think this is the same behaviour as self hosted. We saw correct replacement in some repos, but just a pin PR with the incorrect digest in others.

[uhthomas]

Is there anything I can do to help fix this?

[viceice]

disable pin for those does or pin before enable the replacement

[uhthomas]

I believe this issue occurs even if the original dependency was pinned. I can add that to the reproduction repository to prove it if that would be helpful.

[uhthomas]

I tried disabling digest pinning for those deps, but the same thing happened. Have I misconfigured it?

{
  "packageRules": {
    "matchDatasources": [
      "docker"
    ],
    "matchPackageNames": [
      "example.com/old"
    ],
    "pinDigests": false,
    "replacementName": "example.com/new",
    "replacementVersion": "some-tag"
  }
}

As noted in my previous comment, this happens for dependencies which already have digests. So I imagine this probably prevents this for dependencies which don't already have a digest, but this is still a problem for dependencies which do already have a digest.

So, it's not possible to use this feature for what we need unfortunately. Is the demotion from p2 to p3 correct given the potential severity of using bad digests?

[uhthomas]

I was really hoping some of the recent changes in v35 around replacement would have inadvertently fixed this issue, but that does not appear to be the case.

@rarkins @viceice Any ideas on how we can resolve this? We really want to use Renovate to replace older versions of internal debian images (jessie, stretch, buster) with bullseye or bookworm but just can't because Renovate creates PRs to update those old images to bad digests.

[lixdavid94]

We are also seeing this issue. We want to use replacementName to replace our java11 images with java17 ones. While renovate does create a PR to replace the image name correctly, it also creates a PR related to the original java11 image using the java17 image digest. This is an issue for our developers since we're using renovate to open these PRs ahead of time to give time for developers to migrate over. But using replacementName is breaking updates to the original java11 image.

In other words, we're expecting renovate to open java11 image PR updates normally, and also a java11 image replacement PR with java17

[ssegato81]

I'd like to add more information.

I have the following renovate.json file

{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "configWarningReuseIssue": false,
  "extends": [
    "config:recommended"
  ],
  "packageRules": [
    {
      "matchDatasources": [
        "docker"
      ],
      "matchPackagePatterns": [
        "^registry.ms.io/company/node-16.*"
      ],
      "replacementName": "registry.ms.io/company/openjdk-17",
      "replacementVersion": "5.2.148"
    }
  ]
}

The following Dockerfile

FROM registry.ms.io/company/node-16:4.7.100@sha256:5a5fe176b9fa2f706a5b1b09f26137bf69de27bc40627c80ad212d0c6ea9de8b as base

SHELL ["/bin/bash", "-o", "pipefail", "-c"]
USER root

Last node-16 docker image version: 4.7.102 The openjdk docker image only has versions above 5.0.0, so it doesn't have 4.7.102 version

In this case, Renovate should create two PR 1) updates the node-16 image to the latest version (4.1.102) 2) change the docker image to openjdk version 5.2.148 (this is that latest one)

PR 2 is created correctly PR 1 throughs the following error

Renovate failed to look up the following dependencies: Could not determine new digest for update (docker package registry.ms.io/company/node-16).

Files affected: Dockerfile

Looking at the logs I see the following

DEBUG: getManifestResponse(https://registry.ms.io, comany/openjdk-17, 4.7.102, head) (repository=seba/testRenovate)
DEBUG: HEAD https://registry.ms.io/v2/company/openjdk-17/manifests/4.7.102 = (code=ERR_NON_2XX_3XX_RESPONSE, statusCode=404 retryCount=0, duration=734) (repository=seba/testRenovate)
DEBUG: Docker Manifest is unknown (repository=seba/testRenovate)
       "err": {
         "name": "HTTPError",
         "code": "ERR_NON_2XX_3XX_RESPONSE",
         "timings": {
           "start": 1707503296111,
           "socket": 1707503296112,
           "lookup": 1707503296113,
           "connect": 1707503296114,
           "secureConnect": 1707503296568,
           "upload": 1707503296568,
           "response": 1707503296845,
           "end": 1707503296845,
           "phases": {
             "wait": 1,
             "dns": 1,
             "tcp": 1,
             "tls": 454,
             "request": 0,
             "firstByte": 277,
             "download": 0,
             "total": 734
           }
         },
         "message": "Response code 404 (Not Found)",
         "stack": "HTTPError: Response code 404 (Not Found)\n    at Request.<anonymous> (/usr/src/app/node_modules/got/dist/source/as-promise/index.js:118:42)\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)",
         "options": {
           "headers": {
             "user-agent": "RenovateBot/37.105.1 (https://github.com/renovatebot/renovate)",
             "authorization": "***********",
             "accept": "application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.docker.distribution.manifest.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json",
             "accept-encoding": "gzip, deflate, br"
           },
           "url": "https://registry.ms.io/v2/company/openjdk-17/manifests/4.7.102",
           "hostType": "docker",
           "username": "",
           "password": "",
           "method": "HEAD",
           "http2": false
         },
         "response": {
           "statusCode": 404,
           "statusMessage": "Not Found",
           "body": "",
           "headers": {
             "content-length": "114",
             "content-type": "application/json; charset=utf-8",
             "date": "Fri, 09 Feb 2024 18:28:16 GMT",
             "server": "nginx",
             "set-cookie": ["sid=53cdc4189fb79a4091b3f745675d0d06; Path=/; HttpOnly"],
             "x-request-id": "a55db2a4-7760-4dad-8135-9e6a1a192f0c",
             "connection": "Close"
           },
           "httpVersion": "1.1",
           "retryCount": 0
         }
       },
       "registryHost": "https://registry.ms.io",
       "dockerRepository": "company/openjdk-17",
       "tag": "4.7.102"
DEBUG: Could not determine new digest for update. (repository=seba/testRenovate)
       "packageName": "registry.ms.io/company/node-16",
       "currentValue": "4.7.100",
       "datasource": "docker",
       "newValue": "4.7.102",
       "bucket": "non-major"

It seems that Renovate is replacing only the docker image name in the PR (1)

[viceice]

Working on it today. Reproduction forked to https://github.com/renovate-reproductions/27728-docker-replacement-digest