can't access sso from inside pod due to "Could not resolve host"

DanielFroehlich commented 2 months ago

I am trying to integrated quay with sso. Fails, because inside the quay pod, access to sso fails with "Could not resolve host".

Quite easy to reproduce by debugging into a node:

% oc debug node/ucs57
Temporary namespace openshift-debug-jzkt9 is created for debugging node...
Starting pod/ucs57-debug-kchnr ...
To use host binaries, run `chroot /host`
Pod IP: 10.32.96.57
If you don't see a command prompt, try pressing enter.
sh-4.4# curl https://sso.coe.muc.redhat.com/realms/coe-sso/.well-known/openid-configuration
curl: (6) Could not resolve host: sso.coe.muc.redhat.com

now just by switching from the pod context to the host context, it suddenly works:

sh-4.4# chroot /host
sh-5.1# curl https://sso.coe.muc.redhat.com/realms/coe-sso/.well-known/openid-configuration
{"issuer":"https://sso.coe.muc.redhat.com/realms/coe-sso","authorization_end
...
...

So, something is wrong with DNS. Could be the DNS entry of SSO, which looks a bit strange to me:

sh-5.1# dig sso.coe.muc.redhat.com
; <<>> DiG 9.16.23-RH <<>> sso.coe.muc.redhat.com
...
;; ANSWER SECTION:
sso.coe.muc.redhat.com. 30  IN  CNAME   *.apps.isar.coe.muc.redhat.com.
*.apps.isar.coe.muc.redhat.com. 30 IN   A   10.32.98.2

The CNAME with the star in the beginning (*.apps...) irritates me a bit. Is that actually allowed????

Because: for quay.coe.muc.redhat.com, I used the canonical router name:

sh-5.1# dig quay.coe.muc.redhat.com
...
;; ANSWER SECTION:
quay.coe.muc.redhat.com. 10 IN  CNAME   router-default.apps.isar.coe.muc.redhat.com.
router-default.apps.isar.coe.muc.redhat.com. 10 IN A 10.32.98.2

and that actually works also from inside the pod context:

sh-5.1# exit
exit
sh-4.4# curl -v quay.coe.muc.redhat.com
* Rebuilt URL to: quay.coe.muc.redhat.com/
*   Trying 10.32.98.2...
* TCP_NODELAY set
* Connected to quay.coe.muc.redhat.com (10.32.98.2) port 80 (#0)

Please investigate and advise! I dont dare to change the sso DNS entry....

DanielFroehlich commented 2 months ago

I also just realised that the curl versions between the debug pod and the node are really different: the pod uses curl from Release-Date: 2018-09-05 the node uses curl from Release-Date: 2021-04-14 That makes me suspicious that in some important base image, we use use really old libraries? Hope this is not due to FIPS...

rbo commented 2 months ago

fixed

dig sso.coe.muc.redhat.com

; <<>> DiG 9.18.24 <<>> sso.coe.muc.redhat.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49582
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;sso.coe.muc.redhat.com.        IN  A

;; ANSWER SECTION:
sso.coe.muc.redhat.com. 12  IN  CNAME   router-default.apps.isar.coe.muc.redhat.com.
router-default.apps.isar.coe.muc.redhat.com. 86400 IN A 10.32.98.2

;; Query time: 53 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Wed Apr 24 13:56:26 CEST 2024
;; MSG SIZE  rcvd: 106

I agree *.apps.isar in cname looks not that good.

stormshift / support

can't access sso from inside pod due to "Could not resolve host" #165