search

truenas / charts

TrueNAS SCALE Apps Catalogs & Charts
BSD 3-Clause "New" or "Revised" License
296 stars 287 forks source link

Certificate error when pulling image at app start #2442

Closed jagdtigger closed 4 months ago

jagdtigger commented 4 months ago

Hello all!

I get this error for Collabora and metube when i try to start them:

Failed to pull image "alexta69/metube:2024-05-02": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/alexta69/metube:2024-05-02": failed to copy: httpReadSeeker: failed open: failed to do request: Get "https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/4b/4b6fb6a591967d6632d3641aa3b5066a89950d05d11730b5831ac45fd628f6cc/data?verify=1715000495-3kn2B01XBTM3UXMc0kacf5mE2dc%3D": x509: certificate is valid for *.?.me, ?.me, not production.cloudflare.docker.com (used a "?" to redact my domains)

Started after upgrading to dragonfish but rebooting into the previous boot env (23.10.2) did not help.........

sync-by-unito[bot] commented 4 months ago

➤ Stavros Kois commented:

Hello, looks like you have some DNS Overrides on your network, pointing cloudflare to your domain.

jagdtigger commented 4 months ago

My first idea exactly, but when i do an nslookup my router and quad9 the resulting IPs are the same: `root@zenifer[~/tmp]# nslookup production.cloudflare.docker.com Server: 10.125.210.1 Address: 10.125.210.1#53

Non-authoritative answer: Name: production.cloudflare.docker.com Address: 104.16.101.215 Name: production.cloudflare.docker.com Address: 104.16.98.215 Name: production.cloudflare.docker.com Address: 104.16.99.215 Name: production.cloudflare.docker.com Address: 104.16.100.215 Name: production.cloudflare.docker.com Address: 104.16.97.215 Name: production.cloudflare.docker.com Address: 2606:4700::6810:61d7 Name: production.cloudflare.docker.com Address: 2606:4700::6810:62d7 Name: production.cloudflare.docker.com Address: 2606:4700::6810:63d7 Name: production.cloudflare.docker.com Address: 2606:4700::6810:64d7 Name: production.cloudflare.docker.com Address: 2606:4700::6810:65d7

root@zenifer[~/tmp]# nslookup production.cloudflare.docker.com 9.9.9.9 Server: 9.9.9.9 Address: 9.9.9.9#53

Non-authoritative answer: Name: production.cloudflare.docker.com Address: 104.16.99.215 Name: production.cloudflare.docker.com Address: 104.16.97.215 Name: production.cloudflare.docker.com Address: 104.16.100.215 Name: production.cloudflare.docker.com Address: 104.16.101.215 Name: production.cloudflare.docker.com Address: 104.16.98.215 Name: production.cloudflare.docker.com Address: 2606:4700::6810:63d7 Name: production.cloudflare.docker.com Address: 2606:4700::6810:64d7 Name: production.cloudflare.docker.com Address: 2606:4700::6810:65d7 Name: production.cloudflare.docker.com Address: 2606:4700::6810:61d7 Name: production.cloudflare.docker.com Address: 2606:4700::6810:62d7 `

Opening the URL from a browser on my desktop (uses the router DNS as the NAS) results in a file called "data". Further complicating the issue the only affected packages are the 2 i mentioned. The rest starts and updates without issues.

stavros-k commented 4 months ago

Assuming this lookup is from your TrueNAS server, it looks okay to me.

Is there any chance your firewall modifies outbound requests? But then again does not make sense to only happen on some containers and not all.

jagdtigger commented 4 months ago

Yes, its from the NAS.

Dont think its the router, otherwise i would not be able to open the link on my desktop that uses router DNS as well. Plus pfblockerng only applies block rules to network traffic. The only alteration its doing is via unbound but i would spot it right away from the returned ip......

stavros-k commented 4 months ago

Can you try excluding the server from pfblocker or disabling it for a second? I've tried several servers in different geo locations and I couldnt reproduce this issue.

jagdtigger commented 4 months ago

Not too sure, ill look into that tomorrow (just woke up and have to go to work in 2 hours).

I poked around in TN again and noticed that while nslookup gets the proper ip both curl and wget resolves to the routers WAN ip....

jagdtigger commented 4 months ago

Disabled the NDSBL python module in unbound and the issue went away....... Im still puzzled though about how curl and wget got a different ip than what nslookup showed.

stavros-k commented 4 months ago

I suppose nslookup fetched the correct IP from DNS

curl is working on HTTP, So after it got the correct IP, it did the request and got redirected to some other hostname (which probably blocked by NDSLBL)

Closing this now I guess. Thanks for getting back!