mholt
commented
5 hours ago Thanks for the report, but yeah we'll need the stack trace. Otherwise it could be anything, anywhere.
Closed pimlie closed 1 hour ago
mholt
commented
5 hours ago Thanks for the report, but yeah we'll need the stack trace. Otherwise it could be anything, anywhere.
pimlie
commented
5 hours ago I just added the line the error occured on in case that helps. Trying to get Caddy to recreate the certificate now but just removing the certificate folder in the caddy/certificates/acme-v02.api.letsencrypt.org-directory/ folder doesn't seem to trigger a DNS-01 re-validation :(
pimlie
commented
4 hours ago Creating a new cert for a test domain, so here ya go :)
INF ts=1730489131.3124416 msg=panic: certificate worker: runtime error: invalid memory address or nil pointer dereference
goroutine 96 [running]:
github.com/caddyserver/certmagic.(*jobManager).worker.func1()
github.com/caddyserver/certmagic@v0.21.4/async.go:58 +0x65
panic({...})
runtime/panic.go:770 +0x132
github.com/libdns/leaseweb.(*Provider).createRecordSet(...)
github.com/libdns/leaseweb@v0.3.1/client.go:84 +0x35f
github.com/libdns/leaseweb.(*Provider).AppendRecords(...)
github.com/libdns/leaseweb@v0.3.1/provider.go:56 +0x235
github.com/caddyserver/certmagic.(*DNSManager).createRecord(...)
github.com/caddyserver/certmagic@v0.21.4/solvers.go:401 +0x815
github.com/caddyserver/certmagic.(*DNS01Solver).Present(...)
github.com/caddyserver/certmagic@v0.21.4/solvers.go:265 +0x19e
github.com/caddyserver/certmagic.solverWrapper.Present(...)
github.com/caddyserver/certmagic@v0.21.4/solvers.go:827 +0x22e
github.com/mholt/acmez/v2.(*Client).presentForNextChallenge(...)
github.com/mholt/acmez/v2@v2.0.3/client.go:410 +0x7ab
github.com/mholt/acmez/v2.(*Client).solveChallenges(...)
github.com/mholt/acmez/v2@v2.0.3/client.go:361 +0x26d
github.com/mholt/acmez/v2.(*Client).ObtainCertificate(...)
github.com/mholt/acmez/v2@v2.0.3/client.go:136 +0x598
github.com/caddyserver/certmagic.(*ACMEIssuer).doIssue(...)
github.com/caddyserver/certmagic@v0.21.4/acmeissuer.go:477 +0x668
github.com/caddyserver/certmagic.(*ACMEIssuer).Issue(...)
github.com/caddyserver/certmagic@v0.21.4/acmeissuer.go:371 +0xa7
github.com/caddyserver/caddy/v2/modules/caddytls.(*ACMEIssuer).Issue(...)
github.com/caddyserver/caddy/v2@v2.8.4/modules/caddytls/acmeissuer.go:248 +0x25
github.com/caddyserver/certmagic.(*Config).obtainCert.func2(...)
github.com/caddyserver/certmagic@v0.21.4/config.go:626 +0xcb4
github.com/caddyserver/certmagic.doWithRetry(...)
github.com/caddyserver/certmagic@v0.21.4/async.go:104 +0x233
github.com/caddyserver/certmagic.(*Config).obtainCert(...)
github.com/caddyserver/certmagic@v0.21.4/config.go:700 +0x729
github.com/caddyserver/certmagic.(*Config).ObtainCertAsync(...)
github.com/caddyserver/certmagic@v0.21.4/config.go:505
github.com/caddyserver/certmagic.(*Config).manageOne.func1()
github.com/caddyserver/certmagic@v0.21.4/config.go:415 +0x73
github.com/caddyserver/certmagic.(*jobManager).worker(0x2ab12c0)
github.com/caddyserver/certmagic@v0.21.4/async.go:73 +0x11b
created by github.com/caddyserver/certmagic.(*jobManager).Submit in goroutine 1
github.com/caddyserver/certmagic@v0.21.4/async.go:50 +0x279
francislavoie
commented
4 hours ago How did you build Caddy exactly? Show caddy build-info
Make sure you're using master to ensure you're using the latest version of Caddy.
mholt
commented
4 hours ago That actually looks like a bug in the leaseweb plugin.
pimlie
commented
3 hours ago Make sure you're using master to ensure you're using the latest version of Caddy.
I'm using the caddy:builder container, see Dockerfile above.
That actually looks like a bug in the leaseweb plugin.
To be honest, I just tried to make the existing plugin work. If there is missing error handling in the plugin then I believe you.
But what happened was that it was already working on server X, then I tried the same branch on server Y on which I got the panic. After debugging I noticed I couldn't contact the api either using wget, so I fixed a routing issue and after that the panic was resolved too.
pimlie
commented
3 hours ago You might be able to repro this with any dns provider if you add a 0.0.0.1 api.provider.com entry to your /etc/hosts file. It seems at least I can repro the panic with a valid route but pointing the api domain to an invalid/unreachable ip
francislavoie
commented
2 hours ago xcaddy build \
--with github.com/caddyserver/certmagic@v0.21.4That's the wrong way to build. You can't just upgrade certmagic without upgrading Caddy as well. Do xcaddy build master, and omit certmagic (latest Caddy on master uses latest certmagic).
pimlie
commented
1 hour ago Ah ok, sorry had no idea. Maybe I'm just too spoiled by other dev env's that I expected some internal dependency check if two closely developed packages shouldn't be used without each other's corresponding versions ;)
As I dont have the time to retest this again I'm gonna assume that that was the issue until it bites me again. Thanks.
This is not the same error as #6609 as I built caddy manually with
certmagic@0.21.4.I tracked the panic down to a network issue, it occurs when the api of the dns provider isn't reachable. So it seems that network issues might not be nicely handled yet? To be clear, the api hostname could be resolved through DNS, but actually connecting to the api failed due to a network routing issue.
Unfortunately I don't have a stack trace anymore as docker only kept 100 lines of the log, but let me know if you really need one as I can break the network routing again if needed. The panic occured on this line though (Im also using that dev branch as my PR hasnt been merged yet) :)