Closed Tristan971 closed 1 year ago
where did you get OpenSSL 3.0.5+quic-mangadex-50d2515
? just curious
where did you get
OpenSSL 3.0.5+quic-mangadex-50d2515
? just curious
I build those here https://gitlab.com/mangadex-pub/haproxy/-/jobs/2759190154
Something that may or may not be relevant to this is that we run on an anycasted network, and some users picking up QUIC advertisement from the test LBs (ie running 2.7-dev) were leaking to "stable" ones running 2.6.2 (and thus without the fix for #1759) with QUIC bound but not enabled. This means they would occasionally cause worker crashes on other LBs as per that issue. (in large part due to some consumer ISPs having routing policies one could only qualify as "rather exotic")
The rate of such occurences (leaking across edge locations + triggering the bug of that other issue) does looks way too small to explain this issue on its own however.
Alright so now that I have the issue live, here's a few more pieces of internal haproxy state; hope it helps
Hi Tristan,
while not sure, I suspect that the bug fixed by commit c1640f79f ("BUG/MEDIUM: fd/threads: fix incorrect thread selection in wakeup broadcast") could cause stalls on idle backend connections when takeover happens at high rates. That wouldn't explain why the stream's timeout wouldn't clear them up though.
In your case we're seeing that th 3 connections you picked had data to be read and were lacking storage room, so I'm not 100% convinced this can be related though.
I recently found a bug which prevents session to be released. This is caused when transferring H3 DATA frames from the client (typically POST requests), but it requires specific memory condition to be triggered. I will soon provide a fix, please tell me if this solves at least partially your observations.
I'll have a try with 72a78e8290a1f34c08225f63e7233d211195387f though the backend affected is purely OPTIONS/GET traffic (it's the one associated with our image CDN, so not POSTs there) π
It's quite interesting that it'd be this specific backend (uploaded covers) and not the other CDN backend (uploaded chapter pages), as the only difference between them is that:
Then again, it could be so many things that it's hard to say... Feel free to let me know if some combination of flags or dumps could help.
@a-denoyelle It does look much better with your patch.
There is still an increase in open sessions compared to H2, but that could be just due to #1808 currently forcing me to run with O0 so it could just be being slower causing that.
Once that one is fixed I should be able to get better long-term data
From the latest developments in #1808 I've now gotten a QUIC-enabled instance to be up for a little bit longer. Unfortunately that showed that the issue is fixed (or mostly mitigated) on the frontend side, but not on the backend side (ie FE conns definitely don't grow incontrolably anymore, but BE conns still do).
Are there metrics/commands/etc that I could provide to help? (that said we could also just wait for 1808 to be fully dealt with)
Something that could be related is that the backends for which the connections accumulate are specifically those serving images, and they are afaik the only ones that currently use HTTP 206 in our stack (mainline nginx and prerelease ATS).
Here's a little bit more data after the recent other QUIC-related fixes:
We can see 2 interesting things here. First, the frontend still leaks, but very slowly; let's ignore that for now.
More importantly, the backend leaks but not at all at a consistent rate. If you look closely, instead it "jumps" every now and then rather than continuously grow. Zooming in shows it more clearly:
I'm not familiar with HAProxy's backend session management but that looks a lot like what happens to a server thread pool that is never allowed to shrink. ie any temporary slowdown triggers new threads to be allocated, but they're never reclaimed.
Now that should eventually peak however (when no slowdown is big enough to trigger a scale-up), which doesn't look to be the case here, so presumably those sessions are also not reused later on either.
Now the unfortunate thing with this theory is that... there's no obvious correlation with backend response times over the same (zoomed-in) timerange...
Either way, maybe/hopefully this gives some idea of where to start looking
Note that I have pushed a bug fix about a memleak in the stream part of our QUIC implementation here which could explain the one described here on the frontend: https://github.com/haproxytech/quic-dev/commit/ea4a5cbbdfa71cd287d453dffbdf643846754bba
Note that I have pushed a bug fix about a memleak in the stream part of our QUIC implementation here which could explain the one described here on the frontend: haproxytech/quic-dev@ea4a5cb
Nice, I'll give it a go
Unfortunately it doesn't seem like that was the main contributor this time around π
Ok, thank you for your response. I have found another memleak on TX part. Here is a fix: https://github.com/haproxytech/quic-dev/commit/da9c441886dbfa02840a93e367f65fd6d312c835
Now merged into master, thanks Fred!
I shall give it a spin π
Just mentioning some improvement to this that I observed in the last deployment
Before is eba9088a7c8ea4e1b0de6a9db38c6a39d5d4b204 and after is e35463c767d693d24fc26b6575749e8594e56204. Blue annotations (on the right-side panel) are 12 hours into the deployment.
It's not so obvious on the image, but this is 1.2k sessions vs 800 sessions open against that same backend after 12 hours, so an actual 50% improvement.
Now... the diff between those 2 version is only e35463c767d693d24fc26b6575749e8594e56204 (on the quic side anyway), which makes no sense to me. Maybe it's sheer luck, maybe it can tell you something, I don't know but thought it might be interesting nonetheless.
Thanks Tristan. I continue to think that this is unrelated to QUIC. However what might be happening is that some QUIC slowdowns towards the client could indeed increase concurrency and affect the calculated number of average concurrent connections to the server, which is used as a target to increase/shrink the pool size. It should shrink, since that average reduceds following an exponential curve, and that's the strangest part here. I'm pretty sure that it indicates an issue somewhere else that might be triggered by an unexpected behavior in QUIC, but I have no idea what.
The next time the amount of connections is very high, it could be useful to send "show servers conn" and "show fd" in case we can spot a pattern there.
I continue to think that this is unrelated to QUIC. [β¦] I'm pretty sure that it indicates an issue somewhere else that might be triggered by an unexpected behavior in QUIC
That would make sense indeed
it could be useful to send "show servers conn" and "show fd" in case we can spot a pattern there
Noted, Iβll try and get that from time to time then
I'm wondering to what extent this could be related to this one: #1842 If some streams are not properly reported as failures, maybe some connections do not finish on the other side.
Itβs interesting that you mention it; from https://github.com/haproxy/haproxy/issues/1842#issuecomment-1229416839 maybe itβs relevant for me to say that this is a 206-heavy (ish) backend that is disproportionately affected
Iβll pull the data you suggested shortly
Alright, here's some data
Thanks for the dump. All of these FDs are connections to servers that have a response buffer full. This means that we got blocked processing the response, very likely transfering it to the client. What I don't understand is why these don't time out. Maybe we're seeing a stupid condition such as (for example) a retransmit that refreshes a timeout so that the timeout never expires or something in that vein. A "show sess all" filtering on them would help. Be careful there are many possibly private info in such a dump (IP etc), it may require some post-processing. If it's annoying, it could be useful to try to spot in such a dump some entries with "age=" followed by 3-4 digits (if these exist at all) and only provide 3-4 of them so that we can try to see a pattern emerge.
Got it, that sounds like a probable cause indeed
dump some entries with "age=" followed by 3-4 digits (if these exist at all)
Well I have some sessions in there with 20+ hours of age :-)
Since 1 backend is overwhelmingly affected (uploads_covers) I've filtered for that one only; and further for show sess all
I've kept only its sessions that were at least 10 hours old. Haven't made much stats on my side yet but here are the files, though the sessions have almost everything in common so pattern yes but I couldn't say what part of it is relevant
(For reference, the instance I pulled this from currently has a total of 11'129 sessions, of which 10'729 are this specific backend)
Thanks Tristan, once again thanks to your captures we have a clue now. It seems there's a disagreement between the QUIC layer and the stream layer about the status of an aborted or timed out request. We're still investigating the exact impacts and how to best fix it (it may require quite a bit more thinking) but there's definitely something not clean there. Stay tuned!
I saw some commits related to this issue, so I've just deployed ecb40b2 and will monitor the results (takes at least 24h to really see the impact for sure)
It should not be completely fixed, as we spent the day analyzing all this part and saw a few issues that we don't know how best to address without risking to break more than we fix, so we'll have to wait for Amaury's return in two weeks for this. But some parts could be slightly better though. I the mean time we're improving the traces and dumps so that it's easier to debug, because the QUIC part cannot be inspected via "show fd" and that doesn't help.
Sounds about right for the preliminary data I see on my end (test LBs are in Europe, so traffic is indeed going down as it's the middle of the night here, but the magnitudes are too big to be just traffic patterns related)
Just checked and there are however some very real improvements already, so that's nice π
We shall wait for Amaury's return either way for now π
Excellent! Probably that with improved traces he'll have everything needed to figure how what to do on return ;-)
Hi Tristan,
We found several reasons which may explain your behavior but it's not easy to determine what is the root cause. Can you try the attached patch and give us your feedback please ? (.txt as extension to be able to link the file here). 0001-BUG-MEDIUM-mux-quic-reset-htx-buffer-in-snd_buf-on-t.txt
I will give it a try and report back.
Will take at least a day to see the impact for sure however
Unfortunately the patch causes a very quick segfault:
Detailed Description of the Problem
A look into TCP stack metrics of QUIC-serving nodes suggests something is preventing a subset of HTTP sessions from being released, causing overtime increase memleak-style of allocated TCP sockets.
Expected Behavior
Expecting allocated TCP sockets to roughly match (ie be on the same order of magnitude) in-use TCP sockets.
Steps to Reproduce the Behavior
Not entirely clear, but it does require serving QUIC requests at least.
Do you have any idea what may have caused this?
No response
Do you have an idea how to solve the issue?
No response
What is your configuration?
Output of
haproxy -vv
Last Outputs and Backtraces
No response
Additional Information
Noticed some oddities in metrics when serving QUIC traffic before (see https://github.com/haproxy/haproxy/issues/1759#issuecomment-1168786465), but this time since #1759 was fixed I let it run full-blast for a bit on part of our edge.
Then looking at metrics I noticed the very high amount of sessions and growing![2022-07-24 10_53_17](https://user-images.githubusercontent.com/875533/180641812-7ee77494-1888-4d70-aa4c-67996229bf94.png)
An interesting aspect is that both frontend and backend sessions keep growing. Also they grow per-backend nearly perfectly uniformly, despite there being 2 very different backends involved (one local nginx and two remote Apache Traffic Server instances):![2022-07-24 10_53_25](https://user-images.githubusercontent.com/875533/180641817-ec17427f-d9c1-4164-affc-52f8e794d26c.png)
For comparison, this is what it looks like when QUIC isn't advertised (same edge node and backends, same traffic)
![2022-07-24 10_55_59](https://user-images.githubusercontent.com/875533/180641863-821a1a73-2f8d-4299-bb39-888ec98fa5dd.png)
Just in case it was the metrics being wrong (I've been suspecting there is something up with them and QUIC for a bit), I checked node-exporter's metrics for those nodes and there is indeed a TCP socket leak going on over the same period (this screenshot is for 1/2 test edge nodes):![2022-07-24 10_33_42](https://user-images.githubusercontent.com/875533/180641109-bd2fad4d-8dbb-4269-a838-1c1812d53d59.png)
And again, how it looks like when QUIC isn't advertised, all other things being equal:![2022-07-24 10_34_52](https://user-images.githubusercontent.com/875533/180641141-ba6b4b2a-0561-49f5-a0e5-04cd6426d1ad.png)
I tried running a
show sess
command on the admin socket, and didn't get a matching amount of sessions back, so the leaking ones seem unaccounted. I didn't note down specific numbers at the time however, and will try again and note the precise values.Overall I don't have much more insight on how to start looking into this, here are global packet statistics on the node, I can probably provide a bunch of other system metrics if they would help.