bug(git-lfs): Can't check signature: No public key

When building our devcontainer on our Gitlab-CI build pipeline (to use it in ci-jobs) with the following command

devcontainer build . --image-name $CI_REGISTRY_IMAGE/devcontainer:$CI_COMMIT_SHA --workspace-folder . --cache-from="type=local,src=.buildx/cache" --cache-to="type=local,dest=.buildx/cache" --cache-to="type=registry,ref=reg.goibykus.de/stp/devcontainer:cache,mode=max,image-manifest=true"

the git-lfs feature fails installation with the following log:

#18 0.145 ===========================================================================
#18 0.145 Feature       : Git Large File Support (LFS)
#18 0.145 Description   : Installs Git Large File Support (Git LFS) along with needed dependencies. Useful for base Dockerfiles that often are missing required install dependencies like git and curl.
#18 0.145 Id            : ghcr.io/devcontainers/features/git-lfs
[2024-08-01T07:30:38.548Z] 
#18 0.145 Version       : 1.2.2
#18 0.145 Documentation : https://github.com/devcontainers/features/tree/main/src/git-lfs
#18 0.145 Options       :
#18 0.145     VERSION="latest"
#18 0.145     AUTOPULL="true"
#18 0.145     INSTALLDIRECTLYFROMGITHUBRELEASE="false"
#18 0.145 ===========================================================================
[2024-08-01T07:30:38.741Z] #18 0.184 Installing Git LFS...
#18 0.188 (*) No apt package for bookworm amd64. Installing manually.
[2024-08-01T07:30:38.961Z] #18 0.558 GIT_LFS_VERSION=3.5.1
#18 0.558 Looking for release artfact: git-lfs-linux-amd64-v3.5.1.tar.gz
[2024-08-01T07:30:45.146Z] #18 6.743 (*) Keyserver hkp://keyserver.ubuntu.com is not reachable.
[2024-08-01T07:30:50.164Z] #18 11.76 (*) Keyserver hkp://keyserver.pgp.com is not reachable.
[2024-08-01T07:30:51.203Z] #18 12.80 (*) Downloading GPG key...
[2024-08-01T07:30:51.364Z] #18 12.81 gpg: keybox '/tmp/tmp-gnupg/pubring.kbx' created
[2024-08-01T07:30:52.248Z] #18 13.85 gpg: key F1BA225C0223B187: new key but contains no user ID - skipped
#18 13.85 gpg: Total number processed: 1
#18 13.85 gpg:           w/o user IDs: 1
[2024-08-01T07:30:52.505Z] #18 14.10 gpg: /tmp/tmp-gnupg/trustdb.gpg: trustdb created
#18 14.10 gpg: key F54FE648088335A9: public key "Chris Darroch (CODE SIGNING KEY) <chrisd@apache.org>" imported
[2024-08-01T07:30:52.668Z] #18 14.11 gpg: Total number processed: 1
#18 14.11 gpg:               imported: 1
[2024-08-01T07:30:52.781Z] #18 14.38 gpg: key ABA67BE5A5795889: new key but contains no user ID - skipped
[2024-08-01T07:30:52.872Z] #18 14.38 gpg: Total number processed: 1
#18 14.38 gpg:           w/o user IDs: 1
#18 14.39 gpg: Signature made Thu Mar  7 21:01:49 2024 UTC
#18 14.39 gpg:                using RSA key 4DB92D1D8CEE7E54F06713452D0C9BC12F82B3A1
#18 14.39 gpg:                issuer "bk2204@github.com"
#18 14.39 gpg: Can't check signature: No public key
#18 14.39 ERROR: Feature "Git Large File Support (LFS)" (ghcr.io/devcontainers/features/git-lfs) failed to install! Look at the documentation at https://github.com/devcontainers/features/tree/main/src/git-lfs for help troubleshooting this error.

I could not reproduce the issue locally when starting the devcontainer through vscode. Setting the git-lfs feature to Version 1.2.1 fixed the issue in the pipeline. The pipeline job is run in a container based on mcr.microsoft.com/vscode/devcontainers/javascript-node:0-18 with docker, buildx and devcontainer-cli installed on top.

Here is my devcontainer.json and dockerfile:

{
  "name": "devcontainer",
  "build": {
    "dockerfile": "Dockerfile",
    "cacheFrom": [
      "type=registry,ref=reg.goibykus.de/stp/devcontainer:cache"
    ]
  },
  "containerEnv": {
    "PIPX_HOME": "/usr/local/py-utils",
    "PIPX_BIN_DIR": "/usr/local/py-utils/bin"
  },
  "features": {
    "ghcr.io/devcontainers/features/git-lfs:1.2.1": {},
    "ghcr.io/rio/features/skaffold:2": {
      "version": "v2.10.1"
    },
    "ghcr.io/devcontainers/features/docker-outside-of-docker:1": {},
    "ghcr.io/devcontainers/features/kubectl-helm-minikube:1": {
      "minikube": "none",
      "helm": "3.14.4"
    },
    "ghcr.io/rio/features/k3d:1": {
      "version": "v5.6.3"
    },
    "ghcr.io/devcontainers-contrib/features/istioctl:1": {},
    "ghcr.io/stuartleeks/dev-container-features/shell-history:0": {},
    "ghcr.io/nucleuscloud/devcontainer-features/helmfile:0": {
      "version": "v0.160.0"
    },
    "ghcr.io/dhoeric/features/k9s:1": {
      "version": "0.32.4"
    }
  },
  "mounts": [
    "source=${localEnv:HOME}${localEnv:USERPROFILE}/.kube,target=/home/node/.kube,type=bind,consistency=cached"
  ],
  "customizations": {
    "vscode": {
      "extensions": [
        "EditorConfig.EditorConfig",
        "kennylong.kubernetes-yaml-formatter",
        "anweber.vscode-httpyac",
        "yokawasa.jwt-debugger",
        "ms-kubernetes-tools.vscode-kubernetes-tools",
        "mhutchie.git-graph",
        "eamodio.gitlens",
        "gitlab.gitlab-workflow",
        "valentjn.vscode-ltex"
      ],
      "settings": {
        "git.allowForcePush": true,
        "git.rebaseWhenSync": true,
        "gitlens.plusFeatures.enabled": false,
        "httpyac.environmentSelectedOnStart": [
          "devcontainer",
          "$shared"
        ],
        "yaml.customTags": [
          "!reference sequence"
        ],
        "files.associations": {
          "environments.yaml": "helm",
          "helmfile.yaml": "helm",
          "*.helmfile.yaml": "helm",
          "*.yaml.gotmpl": "helm"
        },
        "dev.containers.dockerCredentialHelper": false
      }
    }
  }
}

FROM reg.goibykus.de/microsoft-mcr/devcontainers/typescript-node:20-bookworm

@prathameshzarkar9 / @gauravsaini04 Can either of you help debug this? thanks!

@CaffeineDaemon Can you provide us with a sample repro? also, can you try with an image other than reg.goibykus.de/microsoft-mcr/devcontainers/typescript-node:20-bookworm? (maybe [mcr.microsoft.com/devcontainers/typescript-node:22](https://github.com/devcontainers/images/tree/main/src/typescript-node)

Decoding the error message you got:

The script first attempts to reach two keyservers (hkp://keyserver.ubuntu.com and hkp://keyserver.pgp.com), but both were not reachable as shown here :
- (*) Keyserver hkp://keyserver.ubuntu.com is not reachable.
- (*) Keyserver hkp://keyserver.pgp.com is not reachable.
- The script then attempts to get the GPG keys and seems like it partially succeeded, as shown by these messages:
  - gpg: key F1BA225C0223B187: new key but contains no user ID - skipped
  - gpg: key ABA67BE5A5795889: new key but contains no user ID - skipped
This above shows while the keys had been downloaded, they were incomplete or invalid for use
- One key (F54FE648088335A9) was successfully imported later on :
  - gpg: key F54FE648088335A9: public key "Chris Darroch (CODE SIGNING KEY) chrisd@apache.org" imported
- this indicates receive_gpg_keys() was able to import at least one valid key
Now, despite retrieving and importing some keys, GPG could not verify the signature of the Git LFS tarball, can be seen through this:
- gpg: Can't check signature: No public key
What we think: I think the keyservers might be down, or your network might be blocking access to them. Can you please check your network configuration for any firewall or proxies etc. before we inspect any further.

@samruddhikhandale Sorry for the late reply, i was busy and forgot to reply to you. Here is a minimal reproduction of the issue withe the image suggested by you:

{
  "name": "test",
  "image": "mcr.microsoft.com/devcontainers/typescript-node:22",
  "features": {
    "ghcr.io/devcontainers/features/git-lfs:1.2.2": {}
  }
}

It yields the same error.

But it seems @gauravsaini04 is right, i cannot reach the keyservers from the pipeline runner at all when using the standard port (11371). It seems it is blocked for outgoing connections in our network.

I will try to convince our network guys to open it, but they can be a bit stubborn on this topic.

https://datatracker.ietf.org/doc/html/draft-shaw-openpgp-hkp-00 Section 2 suggests using port 80 instead:

It has been suggested by some that for reasons of maximum compatibility with firewalls and filtering HTTP proxies, it is better to use the standard HTTP port (TCP port 80)

@gauravsaini04 what do you think about adding a fallback to port 80 when the keyservers cannot be reached? This could save me and other users of the feature some trouble.

Edit: Just appending :80 to the keyserver URL seems to work with gpg to use port 80 instead of 11371, i tested it with gpg --keyserver hkp://keyserver.ubuntu.com:80 --search-key 'your@mail.com'

Thanks @CaffeineDaemon for getting back to us, appreciate it!

Re-reading this, looks like the git-lfs Feature might be running into similar issues as reported in https://github.com/devcontainers/features/issues/1055 (especially because of https://github.com/devcontainers/features/issues/1072#issuecomment-2351997273 and the fact that keys.openpgp.org sometimes strips user IDs from keys)

@gauravsaini04 Can we make similar changes to the git-lfs Feature as of https://github.com/devcontainers/features/pull/1056 ? Thanks!

Edit: Just appending :80 to the keyserver URL seems to work with gpg to use port 80 instead of 11371, i tested it with gpg --keyserver hkp://keyserver.ubuntu.com:80 --search-key 'your@mail.com'

If that does't fix the issues faced by @CaffeineDaemon, then we can definitely look into ^ request.

The pipeline job is run in a container based on mcr.microsoft.com/vscode/devcontainers/javascript-node:0-18 with docker, buildx and devcontainer-cli installed on top.

@CaffeineDaemon On a side note, I'd recommend pinning the image to major version 1 instead of 0 in order to receive security patches.

Have added solution in pr #1124

devcontainers / features

bug(git-lfs): Can't check signature: No public key #1072