Closed defkev closed 3 years ago
I'm only PC user on Windows haven't experienced this issue yet. After reading your long term observation, I'm wondering:
cert_refresh_delay
to 24hours, i do recall reading a blog post (or whatever it was) about it some time ago that certs shouldn't be older than that. dnscrypt itself doesn't report anything out the ordinary (no errors, no crashes)The system is nothing fancy just a headless router doing NAT (ipv4) and router advertising (ipv6) along iptables, DNS and some VPN (wireguard) for a small network of ~50 clients dnscrypt is frontfaced with unbound doing caching/transparent-proxying and blacklisting (mainly to block MS metrics collection from clients on the network)
I do recall having intermediate DNSSEC validation errors from the main instance using random resolvers (some do this better than others) which is why i added the backup instance of dnscrypt-proxy with a static resolver to use as a fallback, which has (mostly) resolved the validation problems.
So that we don't try to chase an issue that doesn't exist any more, could you try 2.0.45, if possible official builds (so that it doesn't use an old Go version either) ?
Does that graph show virtual memory or active memory?
I'll keep an eye on epel for when 2.0.45 drops.
2.0.44 was build against golang 1.14.12 according to the build log on Koji Current in epel is 1.15.5 so i suspect 2.0.45 will be build against that or newer
Graph shows real physical memory as reported by hrSWRunPerfMem
taken from /proc
Who is the bug affecting?
Me
What is affected by this bug?
Memory util
When does this occur?
Linear
Where does it happen?
On the server running dnscrypt
How do we replicate the issue?
Run dnscrypt-proxy using the (mostly) default configuration for +20 days
Non-default config settings:
Expected behavior (i.e. solution)
Not leaking memory
Other Comments
dnscrypt-proxy version 2.0.44 (from epel) on CentOS 7 core as a systemd service
I've seen this for quite some time now since upgrading to dnscrypt-proxy2
EDIT: I used to run dnscrypt-proxy2 from epel til the problem started, then moved over to the release page in hopes to fix it with a more recent version but since moved back to epel as the behavior was the same.
Memory util will grow linear:![mem](https://user-images.githubusercontent.com/5265345/108645932-a7052680-74b4-11eb-8c1a-ee91465819ee.png)
till the system eventually runs out of (available) memory causing dnscrypt-proxy to stall the CPU causing queries to become sluggish and unresponsive:![cpu](https://user-images.githubusercontent.com/5265345/108645892-918ffc80-74b4-11eb-804e-cb35cc514fd5.png)
Charts plot the last 30 days
Effectively what #1352 and #1580 have reported.
This the main instance running (as of now) 104 public-resolvers (ipv4, ipv6, doh, dnscrypt, dnssec, no logging, no filter) with the default rotate of 4 hours and the default lb strategy. I can somewhat alleviate the problem (see the last ~5 days) by a) disabling
doh_servers
(which limits the list to ~61 resolvers) and b) increase thecert_refresh_delay
to 24 hours, tho i haven't tested this long term yet but suspect that (at least the latter) only delays the issue.The problem can always be solved by restarting dnscrypt-proxy, as seen in the charts.
Oddly enough, i run a backup instance of dnscrypt-proxy on the same server using a static list of resolvers (cloudflare ipv4 and ipv6) with the exact same runtime which doesn't have the problem.
Which leaves me to assume that the memory used by the resolvers doesn't get properly recycled by go. I haven't looked at the code yet so this is but a wild guess.
I'd appreciate any advice next to having crontab restart the service every xx days
Cheers