Feature: Support gitea as internal registry

[cmoulliard]

Have you searched for this feature request?

• [X] I searched but did not find similar requests

Problem Statement

Description

To build/push an image against the gitea registry, the way to go is to use the ingress-nginx address resolved against the localhost 127.0.0.1. While this approach is fine when users will build/push an image outside of the k8s cluster, that will fail using the internal k8s service of gitea as you can see hereafter the error reported doing in a pod a buildah build of a dockerfile

[buildah-image : build-and-push] ## Buildah version
[buildah-image : build-and-push] buildah version 1.37.3 (image-spec 1.1.0, runtime-spec 1.2.0)

[buildah-image : build-and-push] ## Build the project ...
[buildah-image : build-and-push] STEP 1/11: FROM registry.access.redhat.com/ubi8/openjdk-21:1.20
[buildah-image : build-and-push] Trying to pull registry.access.redhat.com/ubi8/openjdk-21:1.20...
[buildah-image : build-and-push] Getting image source signatures
[buildah-image : build-and-push] Checking if image destination supports signatures
[buildah-image : build-and-push] Copying blob sha256:8dc97931d0a29118b7e8dd695ac355f7b569223a972f3eb87f2ff07fc9fc190a
[buildah-image : build-and-push] Copying blob sha256:b46e4e7892d6177335aee5445f59105231c351f2fb68a24f25ee7b2656e29674
[buildah-image : build-and-push] Copying config sha256:b8d81704f56858c6859ce949133635bb716162ddd3c8d012ec77572449403153
[buildah-image : build-and-push] Writing manifest to image destination
[buildah-image : build-and-push] Storing signatures
[buildah-image : build-and-push] STEP 2/11: ENV LANGUAGE='en_US:en'
[buildah-image : build-and-push] STEP 3/11: COPY --chown=185 target/quarkus-app/lib/ /deployments/lib/
[buildah-image : build-and-push] STEP 4/11: COPY --chown=185 target/quarkus-app/*.jar /deployments/
[buildah-image : build-and-push] STEP 5/11: COPY --chown=185 target/quarkus-app/app/ /deployments/app/
[buildah-image : build-and-push] STEP 6/11: COPY --chown=185 target/quarkus-app/quarkus/ /deployments/quarkus/
[buildah-image : build-and-push] STEP 7/11: EXPOSE 8080
[buildah-image : build-and-push] STEP 8/11: USER 185
[buildah-image : build-and-push] STEP 9/11: ENV JAVA_OPTS_APPEND="-Dquarkus.http.host=0.0.0.0 -Djava.util.logging.manager=org.jboss.logmanager.LogManager"
[buildah-image : build-and-push] STEP 10/11: ENV JAVA_APP_JAR="/deployments/quarkus-run.jar"
[buildah-image : build-and-push] STEP 11/11: ENTRYPOINT [ "/opt/jboss/container/java/run/run-java.sh" ]
[buildah-image : build-and-push] COMMIT my-gitea-http.gitea:3000/giteaadmin/my-quarkus-app
[buildah-image : build-and-push] Getting image source signatures
[buildah-image : build-and-push] Copying blob sha256:dd5e77a90e609b328f2e49aa60e50bd8837e505c157060c337725413ccf449f1
[buildah-image : build-and-push] Copying blob sha256:8b30b41a0b038bf660c4538ae04bb77b7cdbfcfcb0f5a378129ddf82b91542e7
[buildah-image : build-and-push] Copying blob sha256:3d118423613f432336691bc30be7ef697cabccac4f7c386b134418b876c55428
[buildah-image : build-and-push] Copying config sha256:7ebf93612de328321595acc2f8343ece1adb6def11e963614331b4e70a20f257
[buildah-image : build-and-push] Writing manifest to image destination
[buildah-image : build-and-push] --> 7ebf93612de3

[buildah-image : build-and-push] Successfully tagged my-gitea-http.gitea:3000/giteaadmin/my-quarkus-app:latest
[buildah-image : build-and-push] 7ebf93612de328321595acc2f8343ece1adb6def11e963614331b4e70a20f257

[buildah-image : build-and-push] + buildah --storage-driver=overlay push --tls-verify=false --digestfile /tmp/image-digest my-gitea-http.gitea:3000/giteaadmin/my-quarkus-app docker://my-gitea-http.gitea:3000/giteaadmin/my-quarkus-app
[buildah-image : build-and-push] Getting image source signatures
[buildah-image : build-and-push] Copying blob sha256:3d118423613f432336691bc30be7ef697cabccac4f7c386b134418b876c55428
[buildah-image : build-and-push] Copying blob sha256:dd5e77a90e609b328f2e49aa60e50bd8837e505c157060c337725413ccf449f1
[buildah-image : build-and-push] Copying blob sha256:8b30b41a0b038bf660c4538ae04bb77b7cdbfcfcb0f5a378129ddf82b91542e7
[buildah-image : build-and-push] Error: pushing image "my-gitea-http.gitea:3000/giteaadmin/my-quarkus-app" to "docker://my-gitea-http.gitea:3000/giteaadmin/my-quarkus-app": trying to reuse blob sha256:dd5e77a90e609b328f2e49aa60e50bd8837e505c157060c337725413ccf449f1 at destination: Get "https://gitea.cnoe.localtest.me:8443/v2/token?scope=repository%3Agiteaadmin%2Fmy-quarkus-app%3Apull%2Cpush&service=container_registry": dial tcp: lookup gitea.cnoe.localtest.me: no such host

Possible Solution

I suggest that we support such an option. That will require that we do different things such as:

• Adapt the kind template file https://github.com/cnoe-io/idpbuilder/blob/main/pkg/kind/resources/kind.yaml.tmpl to include the internal address of the registry

  
  containerdConfigPatches:

• |- [plugins."io.containerd.grpc.v1.cri".registry.mirrors."gitea.{{ .Host }}:{{ .Port }}"] endpoint = ["https://gitea.{{ .Host }}"] [plugins."io.containerd.grpc.v1.cri".registry.configs."gitea.{{ .Host }}".tls] insecure_skip_verify = true

• In a 2nd step, we should also mount the Certificate to the HTTP(S) endpoint the gitea service as secret or configMap to support HTTP & HTTPS calls as some clients could refuse to access the oci registry using plain HTTP

Alternatives Considered

No response

[cmoulliard]

Questions:

Are we sure that changes documented here https://cnoe.io/docs/reference-implementation/installations/idpbuilder/how-it-works#in-cluster-dns-configuration and coded using such a template https://github.com/cnoe-io/idpbuilder/blob/main/pkg/build/templates/coredns/cm-coredns-default.yaml.tmpl#L6-L13 work ?

Could it be possible that when a DNS request is issued by a pod running inside the cluster that it gets first the loopback address instead of the internal ingress-nginx-controller.ingress-nginx.svc.cluster.local ? Remark: I'm asking such a question as this is typically what I'm observing here: https://github.com/containers/buildah/issues/5767#issuecomment-2393969891

[buildah-image : build-and-push] Error: pushing image "gitea.cnoe.localtest.me:8443/giteaadmin/my-quarkus-app" to "docker://gitea.cnoe.localtest.me:8443/giteaadmin/my-quarkus-app": trying to reuse blob sha256:dd5e77a90e609b328f2e49aa60e50bd8837e505c157060c337725413ccf449f1 at destination: pinging container registry gitea.cnoe.localtest.me:8443: Get "http://gitea.cnoe.localtest.me:8443/v2/": dial tcp: lookup gitea.cnoe.localtest.me: no such host

If I log the requests issued against coreDNS, this is what I got when buildah push the image (and uses coreDNS to resolve the hostname)

[INFO] 10.244.0.12:37511 - 56726 "A IN gitea.cnoe.localtest.me.argocd.svc.cluster.local. udp 77 false 1232" NOERROR qr,aa,rd 138 0.000053458s                                                                
[INFO] 10.244.0.12:37409 - 39399 "AAAA IN gitea.cnoe.localtest.me.argocd.svc.cluster.local. udp 77 false 1232" NOERROR qr,aa,rd 159 0.000028125s
[INFO] 10.244.0.12:52955 - 51195 "A IN gitea.cnoe.localtest.me.argocd.svc.cluster.local. udp 77 false 1232" NOERROR qr,aa,rd 138 0.000025792s                                                                
[INFO] 10.244.0.44:50201 - 39583 "AAAA IN gitea.cnoe.localtest.me.demo.svc.cluster.local. udp 64 false 512" NOERROR qr,aa,rd 157 0.0000765s                                                                  
[INFO] 10.244.0.44:50201 - 5272 "A IN gitea.cnoe.localtest.me.demo.svc.cluster.local. udp 64 false 512" NOERROR qr,aa,rd 136 0.000096417s
[INFO] 10.244.0.44:35337 - 40588 "AAAA IN gitea.cnoe.localtest.me.demo.svc.cluster.local. udp 64 false 512" NOERROR qr,aa,rd 157 0.000028667s                                                                
[INFO] 10.244.0.44:35337 - 27279 "A IN gitea.cnoe.localtest.me.demo.svc.cluster.local. udp 64 false 512" NOERROR qr,aa,rd 136 0.00005675s

If I deploy the pod within the same namespace as gitea I also get a Host not found from buildah cmd executed and CoreDNS reports

[INFO] 10.244.0.67:55276 - 61423 "A IN gitea.cnoe.localtest.me.gitea.svc.cluster.local. udp 65 false 512" NOERROR qr,aa,rd 137 0.000061792s                                         coredns-84b9ff5f77-7nmw2
[INFO] 10.244.0.67:41023 - 24297 "A IN gitea.cnoe.localtest.me.gitea.svc.cluster.local. udp 65 false 512" NOERROR qr,aa,rd 137 0.000022959s                                         coredns-84b9ff5f77-7nmw2
[INFO] 10.244.0.67:41023 - 28392 "AAAA IN gitea.cnoe.localtest.me.gitea.svc.cluster.local. udp 65 false 512" NOERROR qr,aa,rd 158 0.000035s
coredns-84b9ff5f77-9pjn7
[INFO] 10.244.0.67:49237 - 28479 "AAAA IN gitea.cnoe.localtest.me.gitea.svc.cluster.local. udp 65 false 512" NOERROR qr,aa,rd 158 0.0001005s                                        coredns-84b9ff5f77-9pjn7
[INFO] 10.244.0.67:49237 - 63294 "A IN gitea.cnoe.localtest.me.gitea.svc.cluster.local. udp 65 false 512" NOERROR qr,aa,rd 137 0.000047667s

Looking to the log of the CoreDNS pod don't help as requests are issued but we don' t know which IP is finally or not returned to the issuer :-(

@nabuskey

[cmoulliard]

I'm becoming mad as all the tests that I'm doing work using: ping, curl, dig executed part of a pod created using dnsutils tool : kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml

1. Test using DIG (OK)

CoreDNS log

[INFO] 10.244.0.34:48924 - 3815 "A IN gitea.cnoe.localtest.me. udp 64 false 4096" NOERROR qr,aa,rd 113 0.000078125s

DIG

kubectl exec -i -t dnsutils -- dig gitea.cnoe.localtest.me
E1007 13:59:17.052919   16272 websocket.go:296] Unknown stream id 1, discarding message

; <<>> DiG 9.16.27 <<>> gitea.cnoe.localtest.me
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3815
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 7b2fa01f5e43ef58 (echoed)
;; QUESTION SECTION:
;gitea.cnoe.localtest.me. IN  A

;; ANSWER SECTION:
ingress-nginx-controller.ingress-nginx.svc.cluster.local. 6 IN A 10.96.14.241

;; Query time: 1 msec
;; SERVER: 10.96.0.10#53(10.96.0.10)
;; WHEN: Mon Oct 07 11:59:17 UTC 2024
;; MSG SIZE  rcvd: 136

2. Test using curl (OK)

CoreDNS

[INFO] 10.244.0.34:52236 - 58946 "AAAA IN gitea.cnoe.localtest.me.default.svc.cluster.local. udp 67 false 512" NOERROR qr,aa,rd 160 0.000073625s                                                          │
[INFO] 10.244.0.34:52236 - 58738 "A IN gitea.cnoe.localtest.me.default.svc.cluster.local. udp 67 false 512" NOERROR qr,aa,rd 139 0.000102542s                                                             │

Curl

kubectl exec -i -t dnsutils -- curl -s -k  https://gitea.cnoe.localtest.me:8443
*   Trying 10.96.14.241:8443...
* Connected to gitea.cnoe.localtest.me (10.96.14.241) port 8443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: O=cnoe.io
*  start date: Oct  7 10:18:58 2024 GMT
*  expire date: Oct  7 16:18:58 2025 GMT
*  issuer: O=cnoe.io
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0xaaaaeb64fca0)
> GET / HTTP/2
> Host: gitea.cnoe.localtest.me:8443
...
<!DOCTYPE html>
<html lang="en-US" data-theme="gitea-auto">
<head>
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <title>Gitea: Git with a cup of tea</title>
  <link rel="manifest" href="data:application/json;base64,eyJuYW1lIjoiR2l0ZWE6IEdpdCB3aXRoIGEgY3VwIG9mIHRlYSIsInNob3J0X25hbWUiOiJHaXRlYTogR2l0IHdpdGggYSBjdXAgb2YgdGVhIiwic3RhcnRfdXJsIjoiaHR0cHM6Ly9naXRlYS5jbm9lLmxvY2FsdGVzdC5tZTo4NDQzLyIsImljb25zIjpbeyJzcmMiOiJodHRwczovL2dpdGVhLmNub2UubG9jYWx0ZXN0Lm1lOjg0NDMvYXNzZXRzL2ltZy9sb2dvLnBuZyIsInR5cGUiOiJpbWFnZS9wbmciLCJzaXplcyI6IjUxMng1MTIifSx7InNyYyI6Imh0dHBzOi8vZ2l0ZWEuY25vZS5sb2NhbHRlc3QubWU6ODQ0My9hc3NldHMvaW1nL2xvZ28uc3ZnIiwidHlwZSI6ImltYWdlL3N2Zyt4bWwiLCJzaXplcyI6IjUxMng1MTIifV19">
  <meta name="author" content="Gitea - Git with a cup of tea">

3. Test using ping

CoreDNS

[INFO] 10.244.0.34:37510 - 43524 "AAAA IN gitea.cnoe.localtest.me.default.svc.cluster.local. udp 67 false 512" NOERROR qr,aa,rd 160 0.000067583s                                                          │
[INFO] 10.244.0.34:37510 - 43358 "A IN gitea.cnoe.localtest.me.default.svc.cluster.local. udp 67 false 512" NOERROR qr,aa,rd 139 0.000082042s

Ping (OK)

kubectl exec -i -t dnsutils -- ping -c 4 gitea.cnoe.localtest.me
E1007 14:03:58.078530   20398 websocket.go:296] Unknown stream id 1, discarding message
PING gitea.cnoe.localtest.me (10.96.14.241): 56 data bytes
64 bytes from 10.96.14.241: seq=0 ttl=62 time=0.199 ms
64 bytes from 10.96.14.241: seq=1 ttl=62 time=0.568 ms
64 bytes from 10.96.14.241: seq=2 ttl=62 time=0.812 ms
64 bytes from 10.96.14.241: seq=3 ttl=62 time=0.449 ms

[cmoulliard]

To fix my issue, this is I what I hard coded as coreDNS rewrite rules.

rewrite name regex gitea.cnoe.localtest.me.gitea.svc.cluster.local ingress-nginx-controller.ingress-nginx.svc.cluster.local
rewrite name exact cnoe.localtest.me ingress-nginx-controller.ingress-nginx.svc.cluster.local

@nabuskey

[nabuskey]

To fix my issue, this is I what I hard coded as coreDNS rewrite rules.

rewrite name regex gitea.cnoe.localtest.me.gitea.svc.cluster.local ingress-nginx-controller.ingress-nginx.svc.cluster.local
rewrite name exact cnoe.localtest.me ingress-nginx-controller.ingress-nginx.svc.cluster.local

@nabuskey

Hmm this shouldn't be necessary because anything under cnoe.localtest.me gets re-written to the ingress address: https://github.com/cnoe-io/idpbuilder/blob/main/pkg/build/templates/coredns/cm-coredns-default.yaml.tmpl

We are also building an example of using the gitea registry and it does seem to work. Your dig, curl, and ping tests all show things are working. It's possible buildah is using its own way of resolving names.

[cmoulliard]

We are also building an example of using the gitea registry and it does seem to work. Your dig, curl, and ping tests all show things are working. It's possible buildah is using its own way of resolving names.

I don't think so. As buildah client don't offer a verbose mode, the only way to figure out what it does is to look into the code :-(

Such a problem will certainly occur with other container image building tool.

Ideally we should try to better match the incoming DNS requests as here that fails with buildah.

Unfortunately the regex syntax is not well documented, horrible to be used. I spend hours this afternoon without success using different examples

name regex .*\.cnoe\.localtest\.me\.demo\.cluster\.local ingress-nginx-controller.ingress-nginx.svc.cluster.local

name regex (.*)\.cnoe\.localtest\.me\.demo\.svc\.cluster\.local ingress-nginx-controller.ingress-nginx.svc.cluster.local
name regex (.*)\.cnoe\.localtest\.me\.demo\.cluster\.local ingress-nginx-controller.ingress-nginx.svc.cluster.local

name regex (.*)\.cnoe\.localtest\.me\.(.*)\.svc\.cluster\.local ingress-nginx-controller.ingress-nginx.svc.cluster.local
name regex (.*)\.cnoe\.localtest\.me\.(.*)\.cluster\.local ingress-nginx-controller.ingress-nginx.svc.cluster.local

[nabuskey]

gitea.cnoe.localtest.me.gitea.svc.cluster.local

Is this the address that's being queried? That does not look right to me. Queries should be made for gitea.cnoe.localtest.me. Do you have domain search configured in your pod that's adding the cluster.local stuff?

[cmoulliard]

Is this the address that's being queried?

No .The address which is requested by buildah is gitea.cnoe.localtest.me as the image is referenced as gitea.cnoe.localtest.me:8443/giteaadmin/my-quarkus-app

Here is the buildah log

[buildah-image : build-and-push] Error: pushing image "gitea.cnoe.localtest.me:8443/giteaadmin/my-quarkus-app" to "docker://gitea.cnoe.localtest.me:8443/giteaadmin/my-quarkus-app": trying to reuse blob sha256:dd5e77a90e609b328f2e49aa60e50bd8837e505c157060c337725413ccf449f1 at destination: pinging container registry gitea.cnoe.localtest.me:8443: Get "http://gitea.cnoe.localtest.me:8443/v2/": dial tcp: lookup gitea.cnoe.localtest.me: no such host

The address queried gitea.cnoe.localtest.me.gitea.svc.cluster.local is reported within the log of coreDNS.

[cmoulliard]

Do you have domain search configured in your pod that's adding the cluster.local stuff?

No

[nabuskey]

This didn't work for me to get to the failure point you mentioned: https://github.com/ch007m/idpbuilder-gitea-issue/tree/main

It hangs at maven step and couldn't proceed. No logs or anything provided.

I just tested with buildah in k8s cluster and it does work for me:

# cat Containerfile
FROM registry.access.redhat.com/ubi8/ubi
ADD myecho /usr/local/bin
ENTRYPOINT "/usr/local/bin/myecho"
# cat myecho
echo "This container works!"

buildah build --tls-verify=0 -t myecho .
STEP 1/3: FROM registry.access.redhat.com/ubi8/ubi
Trying to pull registry.access.redhat.com/ubi8/ubi:latest...
Getting image source signatures
Copying blob 148a3ed2f70e done
Copying config 4f03f39cd4 done
Writing manifest to image destination
Storing signatures
STEP 2/3: ADD myecho /usr/local/bin
STEP 3/3: ENTRYPOINT "/usr/local/bin/myecho"
COMMIT myecho
Getting image source signatures
Copying blob 4493ee5cf8cf skipped: already exists
Copying blob c9248985de20 done
Copying config 7f6ef92039 done
Writing manifest to image destination
Storing signatures
--> 7f6ef920398
Successfully tagged localhost/myecho:latest
7f6ef920398832cee24ee1d4fec76bc65e19772320ab5e6e8f7cf1073126d263

# buildah  tag 7f6ef9203988 gitea.cnoe.localtest.me:8443/giteaadmin/myecho:latest

# buildah push --tls-verify=0 gitea.cnoe.localtest.me:8443/giteaadmin/myecho:latest
Getting image source signatures
Copying blob c9248985de20 done
Copying blob 4493ee5cf8cf done
Copying config 7f6ef92039 done
Writing manifest to image destination
Storing signatures

# buildah rmi -f 7f6ef9203988
untagged: localhost/myecho:latest
untagged: gitea.cnoe.localtest.me:8443/giteaadmin/myecho:latest
7f6ef920398832cee24ee1d4fec76bc65e19772320ab5e6e8f7cf1073126d263

# buildah pull --tls-verify=0 gitea.cnoe.localtest.me:8443/giteaadmin/myecho:latest
Trying to pull gitea.cnoe.localtest.me:8443/giteaadmin/myecho:latest...
Getting image source signatures
Copying blob e9438c8fb003 skipped: already exists
Copying blob 280bf46c51d6 done
Copying config 7f6ef92039 done
Writing manifest to image destination
Storing signatures
7f6ef920398832cee24ee1d4fec76bc65e19772320ab5e6e8f7cf1073126d263

outside cluster:

docker pull gitea.cnoe.localtest.me:8443/giteaadmin/myecho:latest
latest: Pulling from giteaadmin/myecho
e9438c8fb003: Pull complete
280bf46c51d6: Pull complete
Digest: sha256:f3f9766d210600fd4c636f9a583c3f1059c9c4a02f4ea700a10f63b8f7d01aaf
Status: Downloaded newer image for gitea.cnoe.localtest.me:8443/giteaadmin/myecho:latest
gitea.cnoe.localtest.me:8443/giteaadmin/myecho:latest

[cmoulliard]

I just tested with buildah in k8s cluster and it does work for me:

Did you created a pod within the idpbuilder cluster to test the commands you mentioned ?

[cmoulliard]

I will create a new pipeline using simply your test case to reproduce the error.

[cmoulliard]

I can reproduce the error using the default configMap of CoreDNS.

The pod (created by a Tekton task) can curl the gitea API Swagger

## Host and DNS nameservers cfg ...
# Kubernetes-managed hosts file.
127.0.0.1   localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
fe00::0 ip6-mcastprefix
fe00::1 ip6-allnodes
fe00::2 ip6-allrouters
10.244.0.23 build-push-image-curl-gitea-pod
search demo.svc.cluster.local svc.cluster.local cluster.local dns.podman
nameserver 10.96.0.10
options ndots:5
## Using as image: [quay.io/curl/curl](http://quay.io/curl/curl)
## Curl to the registry host: https://gitea.cnoe.localtest.me:8443/api/swagger ...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Host [gitea.cnoe.localtest.me:8443](http://gitea.cnoe.localtest.me:8443/) was resolved.
* IPv6: (none)
* IPv4: 10.96.14.241
*   Trying 10.96.14.241:8443...
* ALPN: curl offers h2,http/1.1
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [447 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [80 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / x25519 / id-ecPublicKey
* ALPN: server accepted h2
* Server certificate:
*  subject: [O=cnoe.io](http://o=cnoe.io/)
*  start date: Oct  7 10:18:58 2024 GMT
*  expire date: Oct  7 16:18:58 2025 GMT
*  issuer: [O=cnoe.io](http://o=cnoe.io/)
*  SSL certificate verify result: self-signed certificate (18), continuing anyway.
*   Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA256
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [57 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [57 bytes data]
* Connected to [gitea.cnoe.localtest.me](http://gitea.cnoe.localtest.me/) (10.96.14.241) port 8443
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://gitea.cnoe.localtest.me:8443/api/swagger
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: [gitea.cnoe.localtest.me:8443](http://gitea.cnoe.localtest.me:8443/)]
* [HTTP/2] [1] [:path: /api/swagger]
* [HTTP/2] [1] [user-agent: curl/8.10.1]
* [HTTP/2] [1] [accept: */*]
} [5 bytes data]
> GET /api/swagger HTTP/2
> Host: [gitea.cnoe.localtest.me:8443](http://gitea.cnoe.localtest.me:8443/)
> User-Agent: curl/8.10.1
> Accept: */*
> 
* Request completely sent off
{ [5 bytes data]
< HTTP/2 200 
< date: Tue, 08 Oct 2024 12:57:41 GMT
< content-type: text/html; charset=utf-8
< content-length: 701
< cache-control: max-age=0, private, must-revalidate, no-transform
< set-cookie: i_like_gitea=0094044c3a9cf9af; Path=/; HttpOnly; Secure; SameSite=Lax
< set-cookie: _csrf=9Xp8AFgYupbIHdyoC0vjp558G8U6MTcyODM5MjI2MTU3NjQyMzEwMg; Path=/; Max-Age=86400; HttpOnly; Secure; SameSite=Lax
< x-frame-options: SAMEORIGIN
<!DOCTYPE html>
< strict-transport-security: max-age=31536000; includeSubDomains
< 
<html lang="en">
    <head>
        <title>Gitea API</title>
        <link href="/assets/css/swagger.css?v=1.22.0" rel="stylesheet">
    </head>
    <body>
        <a class="swagger-back-link" href="/"><svg viewBox="0 0 16 16" class="svg octicon-reply" aria-hidden="true" width="16" height="16"><path d="M6.78 1.97a.75.75 0 0 1 0 1.06L3.81 6h6.44A4.75 4.75 0 0 1 15 10.75v2.5a.75.75 0 0 1-1.5 0v-2.5a3.25 3.25 0 0 0-3.25-3.25H3.81l2.97 2.97a.749.749 0 0 1-.326 1.275.75.75 0 0 1-.734-.215L1.47 7.28a.75.75 0 0 1 0-1.06l4.25-4.25a.75.75 0 0 1 1.06 0"/></svg>Return to Gitea</a>
{ [701 bytes data]

100   701  100   701    0     0  67690      0 --:--:-- --:--:-- --:--:-- 70100
* Connection #0 to host [gitea.cnoe.localtest.me](http://gitea.cnoe.localtest.me/) left intact
        <div id="swagger-ui" data-source="/swagger.v1.json"></div>
        <script src="/assets/js/swagger.js?v=1.22.0"></script>
    </body>
</html>

and coreDNS logs such information

[INFO] 10.244.0.9:34821 - 19433 "A IN gitea.cnoe.localtest.me.argocd.svc.cluster.local. udp 77 false 1232" NXDOMAIN qr,aa,rd 159 0.000122458s
[INFO] 10.244.0.9:60991 - 65075 "AAAA IN gitea.cnoe.localtest.me.argocd.svc.cluster.local. udp 77 false 1232" NXDOMAIN qr,aa,rd 159 0.000105833s
[INFO] 10.244.0.9:47992 - 35361 "A IN gitea.cnoe.localtest.me.svc.cluster.local. udp 70 false 1232" NXDOMAIN qr,aa,rd 152 0.000052625s
[INFO] 10.244.0.9:46282 - 64908 "AAAA IN gitea.cnoe.localtest.me.svc.cluster.local. udp 70 false 1232" NXDOMAIN qr,aa,rd 152 0.000067584s
[INFO] 10.244.0.9:45610 - 13287 "AAAA IN gitea.cnoe.localtest.me.cluster.local. udp 66 false 1232" NOERROR qr,aa,rd 148 0.000096375s
[INFO] 10.244.0.9:46353 - 30971 "A IN gitea.cnoe.localtest.me.cluster.local. udp 66 false 1232" NOERROR qr,aa,rd 127 0.000074416s

The Buildah pod created also by a tekton task fails

[maven-package-test : mvn-goals] [INFO] ------------------------------------------------------------------------
[maven-package-test : mvn-goals] [INFO] BUILD SUCCESS
[maven-package-test : mvn-goals] [INFO] ------------------------------------------------------------------------
[maven-package-test : mvn-goals] [INFO] Total time:  30.867 s
[maven-package-test : mvn-goals] [INFO] Finished at: 2024-10-08T14:33:29Z
[maven-package-test : mvn-goals] [INFO] ------------------------------------------------------------------------

[buildah-image : build-and-push] ## Buildah version
[buildah-image : build-and-push] buildah version 1.37.3 (image-spec 1.1.0, runtime-spec 1.2.0)
[buildah-image : build-and-push] ## Build the project ...
[buildah-image : build-and-push] STEP 1/11: FROM registry.access.redhat.com/ubi8/openjdk-21:1.20
[buildah-image : build-and-push] Trying to pull registry.access.redhat.com/ubi8/openjdk-21:1.20...
[buildah-image : build-and-push] Getting image source signatures
[buildah-image : build-and-push] Checking if image destination supports signatures
[buildah-image : build-and-push] Copying blob sha256:b46e4e7892d6177335aee5445f59105231c351f2fb68a24f25ee7b2656e29674
[buildah-image : build-and-push] Copying blob sha256:8dc97931d0a29118b7e8dd695ac355f7b569223a972f3eb87f2ff07fc9fc190a
[buildah-image : build-and-push] Copying config sha256:b8d81704f56858c6859ce949133635bb716162ddd3c8d012ec77572449403153
[buildah-image : build-and-push] Writing manifest to image destination
[buildah-image : build-and-push] Storing signatures
[buildah-image : build-and-push] STEP 2/11: ENV LANGUAGE='en_US:en'
[buildah-image : build-and-push] STEP 3/11: COPY --chown=185 target/quarkus-app/lib/ /deployments/lib/
[buildah-image : build-and-push] STEP 4/11: COPY --chown=185 target/quarkus-app/*.jar /deployments/
[buildah-image : build-and-push] STEP 5/11: COPY --chown=185 target/quarkus-app/app/ /deployments/app/
[buildah-image : build-and-push] STEP 6/11: COPY --chown=185 target/quarkus-app/quarkus/ /deployments/quarkus/
[buildah-image : build-and-push] STEP 7/11: EXPOSE 8080
[buildah-image : build-and-push] STEP 8/11: USER 185
[buildah-image : build-and-push] STEP 9/11: ENV JAVA_OPTS_APPEND="-Dquarkus.http.host=0.0.0.0 -Djava.util.logging.manager=org.jboss.logmanager.LogManager"
[buildah-image : build-and-push] STEP 10/11: ENV JAVA_APP_JAR="/deployments/quarkus-run.jar"
[buildah-image : build-and-push] STEP 11/11: ENTRYPOINT [ "/opt/jboss/container/java/run/run-java.sh" ]
[buildah-image : build-and-push] COMMIT gitea.cnoe.localtest.me:8443/giteaadmin/my-quarkus-app
[buildah-image : build-and-push] Getting image source signatures
[buildah-image : build-and-push] Copying blob sha256:dd5e77a90e609b328f2e49aa60e50bd8837e505c157060c337725413ccf449f1
[buildah-image : build-and-push] Copying blob sha256:8b30b41a0b038bf660c4538ae04bb77b7cdbfcfcb0f5a378129ddf82b91542e7
[buildah-image : build-and-push] Copying blob sha256:1afbc0eb5ccf4bc27865798faecac3391e265df9bfaeb5e133588be2df4f8c1e
[buildah-image : build-and-push] Copying config sha256:e14b39998e596b26b458e30899950342808ec56855cc4efe2701d752b449b4c9
[buildah-image : build-and-push] Writing manifest to image destination
[buildah-image : build-and-push] --> e14b39998e59
[buildah-image : build-and-push] Successfully tagged gitea.cnoe.localtest.me:8443/giteaadmin/my-quarkus-app:latest
[buildah-image : build-and-push] e14b39998e596b26b458e30899950342808ec56855cc4efe2701d752b449b4c9
[buildah-image : build-and-push] + buildah --storage-driver=overlay push --tls-verify=false --digestfile /tmp/image-digest gitea.cnoe.localtest.me:8443/giteaadmin/my-quarkus-app docker://gitea.cnoe.localtest.me:8443/giteaadmin/my-quarkus-app
[buildah-image : build-and-push] Getting image source signatures
[buildah-image : build-and-push] Copying blob sha256:dd5e77a90e609b328f2e49aa60e50bd8837e505c157060c337725413ccf449f1
[buildah-image : build-and-push] Copying blob sha256:8b30b41a0b038bf660c4538ae04bb77b7cdbfcfcb0f5a378129ddf82b91542e7
[buildah-image : build-and-push] Copying blob sha256:1afbc0eb5ccf4bc27865798faecac3391e265df9bfaeb5e133588be2df4f8c1e
[buildah-image : build-and-push] Error: pushing image "gitea.cnoe.localtest.me:8443/giteaadmin/my-quarkus-app" to "docker://gitea.cnoe.localtest.me:8443/giteaadmin/my-quarkus-app": trying to reuse blob sha256:dd5e77a90e609b328f2e49aa60e50bd8837e505c157060c337725413ccf449f1 at destination: pinging container registry gitea.cnoe.localtest.me:8443: Get "http://gitea.cnoe.localtest.me:8443/v2/": dial tcp: lookup gitea.cnoe.localtest.me: no such host

and coredns log reports

[INFO] 10.244.0.35:35516 - 11795 "AAAA IN gitea.cnoe.localtest.me.svc.cluster.local. udp 59 false 512" NXDOMAIN qr,aa,rd 152 0.000026083s
[INFO] 10.244.0.35:35516 - 62994 "A IN gitea.cnoe.localtest.me.svc.cluster.local. udp 59 false 512" NXDOMAIN qr,aa,rd 152 0.000030917s
[INFO] 10.244.0.35:38213 - 20558 "AAAA IN gitea.cnoe.localtest.me.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 148 0.000018417s
[INFO] 10.244.0.35:38213 - 53833 "A IN gitea.cnoe.localtest.me.cluster.local. udp 55 false 512" NOERROR qr,aa,rd 127 0.000017834s
[INFO] 10.244.0.35:52151 - 29077 "A IN gitea.cnoe.localtest.me.demo.svc.cluster.local. udp 64 false 512" NXDOMAIN qr,aa,rd 157 0.000017459s
[INFO] 10.244.0.35:52151 - 33172 "AAAA IN gitea.cnoe.localtest.me.demo.svc.cluster.local. udp 64 false 512" NXDOMAIN qr,aa,rd 157 0.000035791s
[INFO] 10.244.0.35:51807 - 6290 "AAAA IN gitea.cnoe.localtest.me.svc.cluster.local. udp 59 false 512" NXDOMAIN qr,aa,rd 152 0.000041708s
[INFO] 10.244.0.35:51807 - 26514 "A IN gitea.cnoe.localtest.me.svc.cluster.local. udp 59 false 512" NXDOMAIN qr,aa,rd 152 0.000103916s
...

[nabuskey]

@cmoulliard Can you test this for me? I tested with fedora image. I think there's an additional security feature that's causing this issue.

1. Run a pod with quay.io/buildah/stable:v1
2. Reproduce and observe dns response.

# curl https://argocd.cnoe.localtest.me
curl: (6) Could not resolve host: argocd.cnoe.localtest.me
# # curl https://cnoe.localtest.me
curl: (60) SSL certificate problem: self-signed certificate
More details here: https://curl.se/docs/sslcerts.html

So http request against cnoe.localtest.me works, but argoc.cnoe.localtest.me does not. Because...

# dig argocd.cnoe.localtest.me
;; QUESTION SECTION:
;argocd.cnoe.localtest.me.  IN  A

;; ANSWER SECTION:
ingress-nginx-controller.ingress-nginx.svc.cluster.local. 30 IN A 10.96.148.124

# dig cnoe.localtest.me
;; QUESTION SECTION:
;cnoe.localtest.me.     IN  A

;; ANSWER SECTION:
cnoe.localtest.me.  30  IN  A   10.96.148.124

The difference is the answer response. Apparently exact match in coredns returns the answer section as requested while regex match is up to users. See: https://coredns.io/plugins/rewrite/#response-rewrites

Some distributions like Fedora are apparently very sensitive to this. To resolve this, try updating the coredns config like so:

rewrite stop {
        name regex (.*).cnoe.localtest.me ingress-nginx-controller.ingress-nginx.svc.cluster.local answer auto
    }

The answer auto section is what you need.

Once updated, wait for it to propagate. About 30 seconds.

# curl https://argocd.cnoe.localtest.me
curl: (60) SSL certificate problem: self-signed certificate
More details here: https://curl.se/docs/sslcerts.html

[cmoulliard]

The difference is the answer response. Apparently exact match in coredns returns the answer section as requested while regex match is up to users. See: https://coredns.io/plugins/rewrite/#response-rewrites

I was exactly thinking about that as I read somewhere that some CLI check also the answer to be sure that there is no man in the middle attack. I will make a test tomorrow. Many thanks for your feedback

[cmoulliard]

I did a new test successfully this morning

[buildah-image : build-and-push] Successfully tagged gitea.cnoe.localtest.me:8443/giteaadmin/my-quarkus-app:latest
[buildah-image : build-and-push] 74674d0d1472ecfb8bb2e42cf9a0a96b7d76d4820172aee9d903dae87dc3fd90
[buildah-image : build-and-push] + buildah --storage-driver=overlay push --tls-verify=false --digestfile /tmp/image-digest gitea.cnoe.localtest.me:8443/giteaadmin/my-quarkus-app docker://gitea.cnoe.localtest.me:8443/giteaadmin/my-quarkus-app
[buildah-image : build-and-push] Getting image source signatures
[buildah-image : build-and-push] Copying blob sha256:20389ed4aeda74d541449deffb02c642dac8a466318e5659eac7444bfe4343cc
[buildah-image : build-and-push] Copying blob sha256:dd5e77a90e609b328f2e49aa60e50bd8837e505c157060c337725413ccf449f1
[buildah-image : build-and-push] Copying blob sha256:8b30b41a0b038bf660c4538ae04bb77b7cdbfcfcb0f5a378129ddf82b91542e7
[buildah-image : build-and-push] Copying config sha256:74674d0d1472ecfb8bb2e42cf9a0a96b7d76d4820172aee9d903dae87dc3fd90
[buildah-image : build-and-push] Writing manifest to image destination
[buildah-image : build-and-push] + set +x
[buildah-image : build-and-push] sha256:4e236a2bce1f0102f505d35a2661d2e9c214f5a84373ca2942a388b3b566768bgitea.cnoe.localtest.me:8443/giteaadmin/my-quarkus-app

using as rewrite rules

rewrite stop {
  name regex (.*).cnoe.localtest.me ingress-nginx-controller.ingress-nginx.svc.cluster.local answer auto
}
rewrite name exact cnoe.localtest.me ingress-nginx-controller.ingress-nginx.svc.cluster.local

Will you or should I submit a PR ? @nabuskey

[nabuskey]

PR by you would be great :)