x509: certificate signed by unknown authority (cannot provide ca.crt)

[ymettier]

Component

agent

Describe the bug

The bug

I want to build and push an OCI image (aka docker image) on my self-hosted registry. I'm running Woodpecker with Kubernetes backend.

I have this error : x509: certificate signed by unknown authority.

Explanation

The docker build is not aware of my ca.crt file.

How to fix the bug

1. Find a way to provide my ca.crt to the agent (easy to do, need a little hack of the helm chart)
2. Find a way for the agent to provide that ca.crt to plugins like plugin-docker-buildx (like WOODPECKER_BACKEND_DOCKER_VOLUMES but that does not work with Kubernetes backend)

What I did, what I tried...

I have read a lot of documentation on how to build+push my image. I have also read some parts of source code of Woodpecker Kubernetes backend. I have hacked the helm chart in order to put my ca.crt in a ConfigMap and mount it (with success) in either the agent pod or the dind pod (I will probably do a PR for this in repo woodpecker-ci/helm).

I have tried with WOODPECKER_BACKEND_DOCKER_VOLUMES variable but this is restricted to Docker backend. I found nothing similar for Kubernetes.

At last, it does not work whatever I do.

My favorite try (with and without dind enabled) :

  publish-woodpecker-branch:
    image: woodpeckerci/plugin-docker-buildx
    settings:
      registry: myregistry:5001
      logins:
        - registry: https://myregistry:5001/v2
      repo: myregistry:5001/awesome/tool
      platforms: linux/arm64
      tags:
        - ${CI_COMMIT_TAG=build}-${CI_COMMIT_SHA:0:8}-arm64

This should work but I fail at mounting the ConfigMap in the plugin-docker-buildx pod.

Another try (too complex I guess):

  publish-woodpecker-branch:
    image: docker:20.10.12-dind
    secrets:
      - registry_ca_crt
    commands:
      - mkdir -p "/etc/docker/myregistry:5001"
      - echo "$${REGISTRY_CA_CRT}" > "/etc/docker/myregistry:5001/ca.crt"
      - docker build -t "myregistry:5001/awesome/tool:$${CI_COMMIT_TAG=build}-$${CI_COMMIT_SHA:0:8}-arm64" .
      - docker push "myregistry:5001/awesome/tool:$${CI_COMMIT_TAG=build}-$${CI_COMMIT_SHA:0:8}-arm64"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

I get this error : Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

I have also tried with Podman but I'm getting problems to run in privileged mode.

System Info

### Version

{"source":"https://github.com/woodpecker-ci/woodpecker","version":"next-3033abc3"}

Helm values

Notice the hack with extraVolumes and extraVolumeMounts. registry-ca-crt is a ConfigMap that contains a key named ca.crt and the value is the CA certificate.

---
server:
  image:
    tag: next-3033abc3b4
  env:
    WOODPECKER_ADMIN: "<redacted>"
    WOODPECKER_HOST: "http://woodpecker.<redacted>"
    WOODPECKER_GITEA_URL: "http://gitea.<redacted>"
    WOODPECKER_GITEA: "true"
    WOODPECKER_GITEA_SKIP_VERIFY: "true"
    WOODPECKER_OPEN: "true"
    WOODPECKER_ORGS: "<redacted>"

  extraSecretNamesForEnvFrom:
    - woodpecker-gitea-client
    - woodpecker-gitea-secret
    - woodpecker-secret

  fullnameOverride: "woodpecker"

  persistentVolume:
    enabled: true
    size: 10Gi
    storageClass: "<redacted>"

agent:
  image:
    tag: next-3033abc3b4
  env:
    WOODPECKER_SERVER: "woodpecker.woodpecker.svc.cluster.local:9000"
    WOODPECKER_BACKEND: "kubernetes"
    WOODPECKER_BACKEND_K8S_STORAGE_CLASS: "<redacted (same as above)>"
#    WOODPECKER_BACKEND_DOCKER_VOLUMES: "/etc/docker/certs.d:/etc/docker/certs.d:ro"

  extraSecretNamesForEnvFrom:
    - woodpecker-secret

  extraVolumes:
    - name: registry-ca-crt
      configMap:
        name: registry-ca-crt

#  dind:
#    enabled: true
#    extraVolumeMounts:
#      - name: registry-ca-crt
#        mountPath: "/etc/docker/certs.d/myregistry:5001"

  fullnameOverride: "woodpecker-agent"

### Additional context

_No response_

### Validations

- [X] Read the [Contributing Guidelines](https://github.com/woodpecker-ci/woodpecker/blob/master/CONTRIBUTING.md).
- [X] Read the [docs](https://woodpecker-ci.org/docs/intro).
- [X] Check that there isn't [already an issue](https://github.com/woodpecker-ci/woodpecker/issues) that reports the same bug to avoid creating a duplicate.
- [X] Checked that the bug isn't fixed in the `next` version already [https://woodpecker-ci.org/faq#which-version-of-woodpecker-should-i-use]
- [X] Check that this is a concrete bug. For Q&A join our [Discord Chat Server](https://discord.gg/fcMQqSMXJy) or the [Matrix room](https://matrix.to/#/#woodpecker:matrix.org).

[ymettier]

The PR mentionned in the bug description (extraVolumes and extraVolumeMounts) is https://github.com/woodpecker-ci/helm/pull/47.

[zc-devs]

Sorry, but I don't see how it's related to Woodpecker/put your own CI. Especially as a bug.

Find a way to provide my ca.crt to the agent

For why? In context of Kubernetes backend. Your build/publish step executes in different pod.

Find a way for the agent to provide that ca.crt to plugins like plugin-docker-buildx

It's responsibility of used tool/container/plugin to expose to you some configuration parameters.

You can at least

1. Just store certificate in a repository
2. Download from external storage
3. Store in WP secrets as PEM and in step put it in a file (yep, like you did in 'another try').
4. Almost the same as 3, storing PEM in Global env var, if you want that certificate in each pipeline.

I get this error

Because in Kunernetes there is no docker socket. I got the same when tried docker build.

For Kubernetes backend I suggest to use kaniko. And this may help in your case.

[ymettier]

Hello,

I consider this is a bug because https://woodpecker-ci.org/plugins/Docker%20Buildx says Use custom registries and includes a note on how to provide the CA certificate on runner server host. This seems to be possible with docker backend but not with kubernetes backend. Missing feature or bug as a regression ?

As far as I understand, this is not a bug of woodpecker-ci but a bug in woodpecker-ci plugin "docker-buildx" ? Sadly I cannot report it on https://github.com/woodpecker-ci/plugin-docker-buildx (no issue tracking system).

I just found https://codeberg.org/woodpecker-plugins/docker-buildx/issues : should I report it there ?

Thanks for the hint with Kaniko (I did not know about it). I will give a try. If it works for me with Kaniko, I may report it somewhere in https://github.com/woodpecker-ci/woodpecker so you can add it to the documentation ?

[zc-devs]

Helpful links, that I used: https://docs.docker.com/get-started/overview/#docker-architecture https://medium.com/geekculture/create-docker-images-without-docker-daemon-kaniko-847a688155a6 https://codefresh.io/blog/docker-images-without-docker-practical-guide/ https://earthly.dev/blog/docker-vs-buildah-vs-kaniko/ https://stackoverflow.com/questions/67495607/what-is-the-difference-between-kaniko-and-buildkit-buildx

And this is an example using WP. Without custom certificate, but working solution in general.

[ymettier]

Hello,

OK, it works with Kaniko.

Here is my .woodpecker.yml (extracts) :

workspace:
    base: /src
    path: /repo

steps:
  build:
    ...

  publish:
    image: gcr.io/kaniko-project/executor:debug
    secrets:
      - registry_ca_crt
    commands:
      - echo "$${REGISTRY_CA_CRT}" > "/kaniko/ssl/certs/additional-ca-cert-bundle.crt"
      - /kaniko/executor --destination "myregistry:5001/awesome/tool:$${CI_COMMIT_TAG=build}-$${CI_COMMIT_SHA:0:8}-arm64" --context "/src/repo" --dockerfile "/src/repo/Dockerfile"

It seems that --context is mandatory. I have tested with --dockerfile. No idea if it works without it.

Thanks for the help.

I have a solution for my problem, so I will not try to make it work with plugin/buildx. I still consider there is a bug somewhere (as I explained in my previous post) but this is no more a problem for me. I guess we can close this issue ?

Last point : it would be a nice feature if the contents of my secret registry_ca_crt above could be defined in a Kubernetes secret. Then I could use it either with volume or as an environment variable defined from my secret. It would be a nice feature because I would not have to define it as a Woodpecker secret in all my repos. Should I open an issue (tagged "feature") for that ?

[zc-devs]

Last point : it would be a nice feature if the contents of my secret registry_ca_crt above could be defined in a Kubernetes secret. Then I could use it either with volume or as an environment variable defined from my secret. It would be a nice feature because I would not have to define it as a Woodpecker secret in all my repos.

1. I looked at closed issues/PRs yesterday... Seems you can mount volume :)
2. You can try Global variables also.

I guess we can close this issue ?

You decide :) If you want to improve the docs, you can rename issue and use it as a base for following PR.

should I report it there?

Seems that kaniko, buildx and plugin-docker-buildx can use custom certs. So there is no bug or missing functionality. You can just open PR with improved docs. Otherwise you should open the issue in respective repos (buildx or plugin-docker-buildx).

[zc-devs]

the contents of my secret registry_ca_crt above could be defined in a Kubernetes secret. Then I could use it as an environment variable defined from my secret.

If you want exactly that behavior, then it's not implemented as I know. Then it will be feature request (this issue or new one), I think.

[6543]

looks more like an issue for our bildx plugin ?

can we close this here?

[ymettier]

Hello,

I'll try to finish this issue with as much as infos as I can.

@zc-devs

I looked at closed issues/PRs yesterday... Seems you can mount volume :)

True, but with Kubernetes backend, it does not work with configMaps or secrets (yet ?). I have not checked for an issue "wanted feature" for that but maybe it should be created ? Again : for me it now works with kaniko so, with Agile methods in mind, I will not create an issue for what I don't need :)

You can try Global variables also.

Yes! Thanks for that tip. This is what I use now. I like the "Fill it once, use it everywhere" idea and it works.

@6543

looks more like an issue for our bildx plugin ?

Maybe. So I wrote it : https://codeberg.org/woodpecker-plugins/docker-buildx/issues/69

I have no reason to keep that issue open. I close it. Feel free to re-open it if I was too fast at closing ! And thanks for your help and tips !