search

mholt / caddy-dynamicdns

Caddy app that keeps your DNS records (A/AAAA) pointed at itself.
Apache License 2.0
251 stars 25 forks source link

[Porkbun] DNS Entries are not overwritten, they are appended #49

Closed mietzen closed 1 year ago

mietzen commented 1 year ago

I'm using dynamic_dns with provider porkbun the A records are created, but instead of overwriting the old IP with with the new one, the old entrie is left untouched and a new one is created.

DNS-Entries image

# ./caddy run --envfile /opt/CaddyV2/.env
2023/09/02 05:31:42.858 INFO    using adjacent Caddyfile
2023/09/02 05:31:42.997 INFO    admin   admin endpoint started  {"address": "localhost:2019", "enforce_origin": false, "origins": ["//127.0.0.1:2019", "//localhost:2019", "//[::1]:2019"]}
2023/09/02 05:31:43.001 INFO    tls.cache.maintenance   started background certificate maintenance  {"cache": "0x870094600"}
2023/09/02 05:31:43.002 INFO    http.auto_https server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS {"server_name": "srv0", "https_port": 443}
2023/09/02 05:31:43.006 INFO    http.auto_https enabling automatic HTTP->HTTPS redirects    {"server_name": "srv0"}
2023/09/02 05:31:43.043 INFO    tls cleaning storage unit   {"description": "FileStorage:/opt/CaddyV2/data"}
2023/09/02 05:31:43.043 INFO    http    enabling HTTP/3 listener    {"addr": ":443"}
2023/09/02 05:31:43.047 INFO    http.log    server running  {"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2023/09/02 05:31:43.049 INFO    http.log    server running  {"name": "remaining_auto_https_redirects", "protocols": ["h1", "h2", "h3"]}
2023/09/02 05:31:43.049 INFO    http    enabling automatic TLS certificate management   {"domains": ["home.mietzen.xyz", "*.home.mietzen.xyz"]}
2023/09/02 05:31:43.060 INFO    tls finished cleaning storage units
2023/09/02 05:31:48.787 INFO    autosaved config (load with --resume flag)  {"file": "/root/.config/caddy/autosave.json"}
2023/09/02 05:31:48.791 INFO    serving initial configuration
2023/09/02 05:31:55.162 INFO    dynamic_dns domain not found in DNS {"domain": "wireguard.home.mietzen.xyz"}
2023/09/02 05:31:55.163 INFO    dynamic_dns domain not found in DNS {"domain": "wireguard.home.mietzen.xyz"}
2023/09/02 05:31:55.163 INFO    dynamic_dns domain not found in DNS {"domain": "home-assistant.home.mietzen.xyz"}
2023/09/02 05:31:55.163 INFO    dynamic_dns domain not found in DNS {"domain": "home-assistant.home.mietzen.xyz"}
2023/09/02 05:31:55.163 INFO    dynamic_dns domain not found in DNS {"domain": "vaultwarden.home.mietzen.xyz"}
2023/09/02 05:31:55.163 INFO    dynamic_dns domain not found in DNS {"domain": "vaultwarden.home.mietzen.xyz"}
2023/09/02 05:31:55.227 INFO    dynamic_dns updating DNS record {"zone": "mietzen.xyz", "type": "A", "name": "wireguard.home", "value": "xxx.xxx.xxxx.172", "ttl": 3600}
2023/09/02 05:31:55.227 INFO    dynamic_dns updating DNS record {"zone": "mietzen.xyz", "type": "A", "name": "home-assistant.home", "value": "xxx.xxx.xxxx.172", "ttl": 3600}
2023/09/02 05:31:55.228 INFO    dynamic_dns updating DNS record {"zone": "mietzen.xyz", "type": "A", "name": "vaultwarden.home", "value": "xxx.xxx.xxxx.172", "ttl": 3600}
2023/09/02 05:31:57.973 INFO    dynamic_dns finished updating DNS   {"current_ips": ["xxx.xxx.xxxx.172"]}
{
    storage file_system {
        root /opt/CaddyV2/data
    }
    log caddy {
        output file /opt/CaddyV2/data/logs/caddy.log {
            roll_size 10MiB
            roll_local_time
            roll_keep 5
            roll_keep_for 336h
        }
        format console {
            time_local
            time_format wall
        }
        level INFO
    }
    email contact@mietzen.xyz
    dynamic_dns {
        provider porkbun {
            api_key {env.PORKBUN_API_KEY}
            api_secret_key {env.PORKBUN_API_SECRET_KEY}
        }
        domains {
            mietzen.xyz wireguard.home
            mietzen.xyz home-assistant.home
            mietzen.xyz vaultwarden.home
        }
        versions ipv4
        ip_source command /opt/CaddyV2/fritzbox_ext_ip 192.168.178.1
        check_interval 5m
        ttl 1h
    }
}
# ./caddy version
v2.7.4 h1:J8nisjdOxnYHXlorUKXY75Gr6iBfudfoGhrJ8t7/flI=
# /opt/CaddyV2/caddy list-modules
admin.api.load
admin.api.metrics
admin.api.pki
admin.api.reverse_proxy
caddy.adapters.caddyfile
caddy.config_loaders.http
caddy.listeners.http_redirect
caddy.listeners.proxy_protocol
caddy.listeners.tls
caddy.logging.encoders.console
caddy.logging.encoders.filter
caddy.logging.encoders.filter.cookie
caddy.logging.encoders.filter.delete
caddy.logging.encoders.filter.hash
caddy.logging.encoders.filter.ip_mask
caddy.logging.encoders.filter.query
caddy.logging.encoders.filter.regexp
caddy.logging.encoders.filter.rename
caddy.logging.encoders.filter.replace
caddy.logging.encoders.json
caddy.logging.writers.discard
caddy.logging.writers.file
caddy.logging.writers.net
caddy.logging.writers.stderr
caddy.logging.writers.stdout
caddy.storage.file_system
events
http
http.authentication.hashes.bcrypt
http.authentication.hashes.scrypt
http.authentication.providers.http_basic
http.encoders.gzip
http.encoders.zstd
http.handlers.acme_server
http.handlers.authentication
http.handlers.copy_response
http.handlers.copy_response_headers
http.handlers.encode
http.handlers.error
http.handlers.file_server
http.handlers.headers
http.handlers.invoke
http.handlers.map
http.handlers.metrics
http.handlers.push
http.handlers.request_body
http.handlers.reverse_proxy
http.handlers.rewrite
http.handlers.static_response
http.handlers.subroute
http.handlers.templates
http.handlers.tracing
http.handlers.vars
http.ip_sources.static
http.matchers.client_ip
http.matchers.expression
http.matchers.file
http.matchers.header
http.matchers.header_regexp
http.matchers.host
http.matchers.method
http.matchers.not
http.matchers.path
http.matchers.path_regexp
http.matchers.protocol
http.matchers.query
http.matchers.remote_ip
http.matchers.vars
http.matchers.vars_regexp
http.precompressed.br
http.precompressed.gzip
http.precompressed.zstd
http.reverse_proxy.selection_policies.client_ip_hash
http.reverse_proxy.selection_policies.cookie
http.reverse_proxy.selection_policies.first
http.reverse_proxy.selection_policies.header
http.reverse_proxy.selection_policies.ip_hash
http.reverse_proxy.selection_policies.least_conn
http.reverse_proxy.selection_policies.query
http.reverse_proxy.selection_policies.random
http.reverse_proxy.selection_policies.random_choose
http.reverse_proxy.selection_policies.round_robin
http.reverse_proxy.selection_policies.uri_hash
http.reverse_proxy.selection_policies.weighted_round_robin
http.reverse_proxy.transport.fastcgi
http.reverse_proxy.transport.http
http.reverse_proxy.upstreams.a
http.reverse_proxy.upstreams.multi
http.reverse_proxy.upstreams.srv
pki
tls
tls.certificates.automate
tls.certificates.load_files
tls.certificates.load_folders
tls.certificates.load_pem
tls.certificates.load_storage
tls.client_auth.leaf
tls.get_certificate.http
tls.get_certificate.tailscale
tls.handshake_match.remote_ip
tls.handshake_match.sni
tls.issuance.acme
tls.issuance.internal
tls.issuance.zerossl
tls.stek.distributed
tls.stek.standard

  Standard modules: 106

dns.providers.cloudflare
dns.providers.porkbun
dynamic_dns
dynamic_dns.ip_sources.command
dynamic_dns.ip_sources.interface
dynamic_dns.ip_sources.simple_http
dynamic_dns.ip_sources.upnp

  Non-standard modules: 7

  Unknown modules: 0

OS: OPNsense 23.7.3-amd64

mietzen commented 1 year ago

I ran it again with debug logging:

2023/09/02 09:24:58 DEBUG   dynamic_dns found DNS record    {"type": "A", "name": "wireguard.home.mietzen.xyz", "zone": "mietzen.xyz", "value": "xxx.xxx.xxx.123"}
2023/09/02 09:24:58 DEBUG   dynamic_dns found DNS record    {"type": "A", "name": "home-assistant.home.mietzen.xyz", "zone": "mietzen.xyz", "value": "xxx.xxx.xxx.123"}
2023/09/02 09:24:58 DEBUG   dynamic_dns found DNS record    {"type": "A", "name": "vaultwarden.home.mietzen.xyz", "zone": "mietzen.xyz", "value": "xxx.xxx.xxx.123"}
2023/09/02 09:24:58 DEBUG   dynamic_dns found DNS record    {"type": "A", "name": "wireguard.home.mietzen.xyz", "zone": "mietzen.xyz", "value": "xxx.xxx.xxx.172"}
2023/09/02 09:24:58 DEBUG   dynamic_dns found DNS record    {"type": "A", "name": "home-assistant.home.mietzen.xyz", "zone": "mietzen.xyz", "value": "xxx.xxx.xxx.172"}
2023/09/02 09:24:58 DEBUG   dynamic_dns found DNS record    {"type": "A", "name": "vaultwarden.home.mietzen.xyz", "zone": "mietzen.xyz", "value": "xxx.xxx.xxx.172"}
2023/09/02 09:24:58 INFO    dynamic_dns domain not found in DNS {"domain": "wireguard.home.mietzen.xyz"}
2023/09/02 09:24:58 INFO    dynamic_dns domain not found in DNS {"domain": "wireguard.home.mietzen.xyz"}
2023/09/02 09:24:58 INFO    dynamic_dns domain not found in DNS {"domain": "home-assistant.home.mietzen.xyz"}
2023/09/02 09:24:58 INFO    dynamic_dns domain not found in DNS {"domain": "home-assistant.home.mietzen.xyz"}
2023/09/02 09:24:58 INFO    dynamic_dns domain not found in DNS {"domain": "vaultwarden.home.mietzen.xyz"}
2023/09/02 09:24:58 INFO    dynamic_dns domain not found in DNS {"domain": "vaultwarden.home.mietzen.xyz"}
2023/09/02 09:24:58 DEBUG   dynamic_dns looked up current IPs from DNS  {"lastIPs": {"home-assistant.home.mietzen.xyz":{"A":[""],"AAAA":[""]},"vaultwarden.home.mietzen.xyz":{"A":[""],"AAAA":[""]},"wireguard.home.mietzen.xyz":{"A":[""],"AAAA":[""]}}}
2023/09/02 09:24:58 DEBUG   dynamic_dns.ip_sources.command  running command {"command": "/opt/CaddyV2/fritzbox_ext_ip", "args": ["192.168.178.1"], "dir": "", "timeout": 30000000000}
2023/09/02 09:24:59 DEBUG   dynamic_dns.ip_sources.command  parsed ip succesfull    {"command": "/opt/CaddyV2/fritzbox_ext_ip", "args": ["192.168.178.1"], "stdout": "xxx.xxx.xxx.172\n", "ip": "xxx.xxx.xxx.172"}
2023/09/02 09:24:59 INFO    dynamic_dns updating DNS record {"zone": "mietzen.xyz", "type": "A", "name": "wireguard.home", "value": "xxx.xxx.xxx.172", "ttl": 3600}
2023/09/02 09:24:59 INFO    dynamic_dns updating DNS record {"zone": "mietzen.xyz", "type": "A", "name": "home-assistant.home", "value": "xxx.xxx.xxx.172", "ttl": 3600}
2023/09/02 09:24:59 INFO    dynamic_dns updating DNS record {"zone": "mietzen.xyz", "type": "A", "name": "vaultwarden.home", "value": "xxx.xxx.xxx.172", "ttl": 3600}
2023/09/02 09:25:00 INFO    dynamic_dns finished updating DNS   {"current_ips": ["xxx.xxx.xxx.172"]}

dynamic_dns seems to find the records, but doesn't associate them with the ones it should update.

2023/09/02 09:24:58 DEBUG   dynamic_dns found DNS record    {"type": "A", "name": "wireguard.home.mietzen.xyz", "zone": "mietzen.xyz", "value": "xxx.xxx.xxx.123"}
...
2023/09/02 09:24:58 INFO    dynamic_dns domain not found in DNS {"domain": "wireguard.home.mietzen.xyz"}
...
2023/09/02 09:24:58 DEBUG   dynamic_dns looked up current IPs from DNS  {"lastIPs": {"home-assistant.home.mietzen.xyz":{"A":[""],"AAAA":[""]},"vaultwarden.home.mietzen.xyz":{"A":[""],"AAAA":[""]},"wireguard.home.mietzen.xyz":{"A":[""],"AAAA":[""]}}}
...
2023/09/02 09:24:59 INFO    dynamic_dns updating DNS record {"zone": "mietzen.xyz", "type": "A", "name": "wireguard.home", "value": "xxx.xxx.xxx.172", "ttl": 3600}

My best guess is that zoneand name are not properly handled by: https://github.com/libdns/porkbun or https://github.com/caddy-dns/porkbun

Since it includes the zone in the name and dynamic_dns expects the name to not include the zone, is this right?

francislavoie commented 1 year ago

Yep, this must be a problem with https://github.com/libdns/porkbun because this plugin uses SetRecords which implies that existing records should be replaced instead of appended.

https://github.com/mholt/caddy-dynamicdns/blob/3cdd858980a495596e96ece0d666cd1ddba61fb0/dynamicdns.go#L247

whirlthesquirrel commented 1 year ago

I just ran into this with the Namecheap provider, too.

@francislavoie Isn't there an issue in this repo though ("lookupCurrentIPsFromDNS") that causes the "domain not found in DNS" in the first place? Even with a "SetRecords" fix in the provider, this will continue to happen.

In fact, in this case, it will update the "current" records map with a null IP, even when one exists. Just a hunch, but this might influence DNS providers in the "SetRecords" call to believe there isn't an outstanding record (sure, you could argue it could independently verify) for the affected zone(s) which might be the cause of this whole thing.

mholt commented 1 year ago

@whirlthesquirrel To make sure I understand, are you suggesting that the records passed into SetRecords() have different values (a different name and type)? As far as I can tell, that would be the only way to append instead of replace, at least with a correct implementation of SetRecords.

whirlthesquirrel commented 1 year ago

@mholt I think I was reading too much into the "domain not found in DNS" logging and didn't see the post-processing that takes place before the call to "SetRecords".

Here are my log entries in debug:

caddy  | {"level":"debug","ts":1694021296.4225838,"logger":"dynamic_dns","msg":"found DNS record","type":"A","name":"*","zone":"example.com","value":"old_ip"}
caddy  | {"level":"debug","ts":1694021296.4226384,"logger":"dynamic_dns","msg":"found DNS record","type":"A","name":"@","zone":"example.com","value":"unrelated_ip"}
caddy  | {"level":"debug","ts":1694021296.4226441,"logger":"dynamic_dns","msg":"found DNS record","type":"A","name":"www","zone":"example.com","value":"unrelated_ip"}
caddy  | {"level":"info","ts":1694021296.4226513,"logger":"dynamic_dns","msg":"domain not found in DNS","domain":"*.example.com"}
caddy  | {"level":"info","ts":1694021296.4226577,"logger":"dynamic_dns","msg":"domain not found in DNS","domain":"*.example.com"}
caddy  | {"level":"debug","ts":1694021296.4226687,"logger":"dynamic_dns","msg":"looked up current IPs from DNS","lastIPs":{"*.example.com":{"A":["old_ip"],"AAAA":[""]}}}
caddy  | {"level":"debug","ts":1694021296.6638966,"logger":"dynamic_dns.ip_sources.simple_http","msg":"lookup","type":"IPv4","endpoint":"https://api64.ipify.org","ip":"new_ip"}
caddy  | {"level":"info","ts":1694021296.6639485,"logger":"dynamic_dns","msg":"updating DNS record","zone":"example.com","type":"A","name":"*","value":"new_ip","ttl":0}
caddy  | {"level":"info","ts":1694021296.6639616,"logger":"dynamic_dns","msg":"updating DNS record","zone":"example.com","type":"A","name":"*","value":"new_ip","ttl":0}
caddy  | {"level":"info","ts":1694021297.377286,"logger":"dynamic_dns","msg":"finished updating DNS","current_ips":["new_ip"]}

I'm not sure I understand why there is two updates on the same record (or if it's still just one call to "SetRecords"), but I suppose that's not really related to this issue?

francislavoie commented 1 year ago
whirlthesquirrel commented 1 year ago

I updated the logging to log the record type as you mentioned (I was thinking the same thing too), along with the current IPs.

From when there was no previously existing A record:

caddy  | {"level":"debug","ts":1694096804.235109,"logger":"dynamic_dns","msg":"found DNS record","type":"A","name":"@","zone":"example.com","value":"unrelated_ip"}
caddy  | {"level":"debug","ts":1694096804.2352726,"logger":"dynamic_dns","msg":"found DNS record","type":"A","name":"www","zone":"example.com","value":"unrelated_ip"}
caddy  | {"level":"info","ts":1694096804.2352831,"logger":"dynamic_dns","msg":"domain not found in DNS","domain":"*.example.com","type":"A"}
caddy  | {"level":"info","ts":1694096804.2352898,"logger":"dynamic_dns","msg":"domain not found in DNS","domain":"*.example.com","type":"AAAA"}
caddy  | {"level":"debug","ts":1694096804.2353253,"logger":"dynamic_dns","msg":"looked up current IPs from DNS","lastIPs":{"*.example.com":{"A":[""],"AAAA":[""]}}}
caddy  | {"level":"debug","ts":1694096804.417885,"logger":"dynamic_dns.ip_sources.simple_http","msg":"lookup","type":"IPv4","endpoint":"https://api64.ipify.org","ip":"new_ip"}
caddy  | {"level":"debug","ts":1694096804.4179177,"logger":"dynamic_dns","msg":"found current IP","value":"new_ip","type":"A"}
caddy  | {"level":"info","ts":1694096804.4179363,"logger":"dynamic_dns","msg":"updating DNS record","zone":"example.com","type":"A","name":"*","value":"new_ip","ttl":0}
caddy  | {"level":"info","ts":1694096805.7142298,"logger":"dynamic_dns","msg":"finished updating DNS","current_ips":["new_ip"]}

From when there was a single, outdated A record:

caddy  | {"level":"debug","ts":1694096926.982758,"logger":"dynamic_dns","msg":"found DNS record","type":"A","name":"*","zone":"example.com","value":"old_ip"}
caddy  | {"level":"debug","ts":1694096926.9828343,"logger":"dynamic_dns","msg":"found DNS record","type":"A","name":"@","zone":"example.com","value":"unrelated_ip"}
caddy  | {"level":"debug","ts":1694096926.9828475,"logger":"dynamic_dns","msg":"found DNS record","type":"A","name":"www","zone":"example.com","value":"unrelated_ip"}
caddy  | {"level":"info","ts":1694096926.9828606,"logger":"dynamic_dns","msg":"domain not found in DNS","domain":"*.example.com","type":"AAAA"}
caddy  | {"level":"debug","ts":1694096926.9828837,"logger":"dynamic_dns","msg":"looked up current IPs from DNS","lastIPs":{"*.example.com":{"A":["old_ip"],"AAAA":[""]}}}
caddy  | {"level":"debug","ts":1694096927.2460074,"logger":"dynamic_dns.ip_sources.simple_http","msg":"lookup","type":"IPv4","endpoint":"https://api64.ipify.org","ip":"new_ip"}
caddy  | {"level":"debug","ts":1694096927.24605,"logger":"dynamic_dns","msg":"found current IP","value":"new_ip","type":"A"}
caddy  | {"level":"info","ts":1694096927.2460687,"logger":"dynamic_dns","msg":"updating DNS record","zone":"example.com","type":"A","name":"*","value":"new_ip","ttl":0}
caddy  | {"level":"info","ts":1694096928.0516918,"logger":"dynamic_dns","msg":"finished updating DNS","current_ips":["new_ip"]}

From when the new record has already been appended with the existing, outdated A record:

caddy  | {"level":"debug","ts":1694096065.9506721,"logger":"dynamic_dns","msg":"found DNS record","type":"A","name":"*","zone":"example.com","value":"old_ip"}
caddy  | {"level":"debug","ts":1694096065.9507217,"logger":"dynamic_dns","msg":"found DNS record","type":"A","name":"*","zone":"example.com","value":"new_ip"}
caddy  | {"level":"debug","ts":1694096065.950729,"logger":"dynamic_dns","msg":"found DNS record","type":"A","name":"@","zone":"example.com","value":"unrelated_ip"}
caddy  | {"level":"debug","ts":1694096065.9507356,"logger":"dynamic_dns","msg":"found DNS record","type":"A","name":"www","zone":"example.com","value":"unrelated_ip"}
caddy  | {"level":"info","ts":1694096065.950744,"logger":"dynamic_dns","msg":"domain not found in DNS","domain":"*.example.com","type":"AAAA"}
caddy  | {"level":"debug","ts":1694096065.9507565,"logger":"dynamic_dns","msg":"looked up current IPs from DNS","lastIPs":{"*.example.com":{"A":["new_ip"],"AAAA":[""]}}}
caddy  | {"level":"debug","ts":1694096066.222695,"logger":"dynamic_dns.ip_sources.simple_http","msg":"lookup","type":"IPv4","endpoint":"https://api64.ipify.org","ip":"new_ip"}
caddy  | {"level":"debug","ts":1694096066.222732,"logger":"dynamic_dns","msg":"found current IP","value":"new_ip","type":"A"}
caddy  | {"level":"debug","ts":1694096066.2227442,"logger":"dynamic_dns","msg":"no IP address change; no update needed"}

That's strange, because from the log I posted before, it looks like the middle case would be the one to reproduce the same behavior (2 "domain not found in DNS" entries, assuming "A" and "AAAA"?), which didn't this time. Not sure what's going on there and how it's different from before.

Should "domain not found in DNS" be emitted on "AAAA" records when only "versions ipv4" is specified?

Also, Namecheap has the concept of a "Dynamic DNS password", which is separate from an API key. To use the API, you have to have a whitelisted client IP. I believe Caddy was able to update the records without whitelisting, but not fetch the existing ones without the client IP being whitelisted. If fetching existing records is a requirement for proper functionality here, and since whitelisting a new IP would have to be done manually, it seems counter-intuitive to use an API key for dynamic DNS. Something to think about if the current DNS provider interface is not sufficient to solve this, as it would unnecessarily update every time.

francislavoie commented 1 year ago

Not sure what's going on there and how it's different from before.

:man_shrugging: I guess you're saying you can't replicate the problem after adjusting the logging?

Should "domain not found in DNS" be emitted on "AAAA" records when only "versions ipv4" is specified?

Probably not. An oversight when I implemented that I guess. PR is welcome if you want to adjust it :+1:

I believe Caddy was able to update the records without whitelisting, but not fetch the existing ones without the client IP being whitelisted.

:scream: that seems backwards... write operations should have "higher security" than read operations.

Either way I'd just argue "Namecheap's API is dumb and bad, and there's not much we can do about that".

whirlthesquirrel commented 1 year ago

🤷‍♂️ I guess you're saying you can't replicate the problem after adjusting the logging?

Unfortunately, yeah. 😞

😱 that seems backwards... write operations should have "higher security" than read operations.

Either way I'd just argue "Namecheap's API is dumb and bad, and there's not much we can do about that".

Honestly, fair enough! 😆 Thanks for your help.