aukevanleeuwen
commented
2 days ago @joanlopez / @mstoykov , since I happened to see you active on the other issue... I realise this issue is quite involved, but maybe you can at least answer the following quick question: I'm trying to force HTTP/1.1 because of this issue. On the CLI locally I can do that with:
k6 run src/k6/load-test.js --env scenario=post_it --env GODEBUG=http2client=0because it will hit this code: https://github.com/grafana/k6/blob/011a6b40de0109f77954faf898cf1643eab5d07e/js/runner.go#L255-L271
But it doesn't seem to work if I run this code with k6 cloud .... Can I force HTTP/1.1 when running tests on K6 Cloud?
Thanks in advance. Obviously I hope you are intrigued by this issue as well 😀
mstoykov
Brief summary
K6 'hangs' on specific sequence of requests. I cannot reproduce this behaviour with (for example)
curlor the native golang http client.k6 version
v0.52.0, but tested as far back as v0.40.0
OS
macOS 14.5
Docker version and image (if applicable)
No response
Steps to reproduce the problem
I have a strange issue with K6. In one of my load test scenarios,
k6was hanging on aPOSTrequest. Running the test with different configurations of VUs I've narrowed it down to a specific sequence of requests.GET https://example.com/some/pathGET https://example.com/some/pathPOST https://example.com/some/pathwith request body of 16K random stringGET https://example.com/some/pathGET https://example.com/some/pathPOST https://example.com/some/pathwith request body of 16K random string (this request hangs)This is quite hard for me to debug (and describe in this issue), so bear with me. I've tried to go through a bunch of things trying to pin it down to something specific.
These requests fail on an API that is behind Akamai (with HTTP/2 behaviour configured). Some observations:
GETrequests, or just thePOSTrequests, it all runs fine. So it's only the alternation ofGETs andPOSTs that triggers this issue. Weirder still. If I would half the request body size of thePOST, the amount of iterations (2GETs and 1POST) doubles.curlrequests using--nextto make sure that we have connection reuse and trying to get as close as possible to whatever K6 is doing. It just works.In all of this testing, looking at logs from
GODEBUG=http2debug=2and breaking inside the k6 and http client source code I've narrowed it down to this:While sending the POST data it enters this loop: https://github.com/grafana/k6/blob/011a6b40de0109f77954faf898cf1643eab5d07e/vendor/golang.org/x/net/http2/transport.go#L1841-L1860
The first time around this is going fine because, the allowed length to send in this frame is
16384, which is the entire buffer, remaining is[]and it will send the frame withflags=END_STREAM. All good. The secondPOSTrequest however will enter that same loop, but for some reason that is beyond my understanding,cs.awaitFlowControl(len(remain))will return16383, so one byte less. This frame is created (without theEND_STREAMflag obviously) and sent, it will enter the next iteration of the loop because there is still one byte to send and hitcs.awaitFlowControl(len(remain))again. However inside that method on this line: https://github.com/grafana/k6/blob/011a6b40de0109f77954faf898cf1643eab5d07e/vendor/golang.org/x/net/http2/transport.go#L1934cs.flow.available()will return0and therefore will block on thecc.cond.Wait()mutex (https://github.com/grafana/k6/blob/011a6b40de0109f77954faf898cf1643eab5d07e/vendor/golang.org/x/net/http2/transport.go#L1946). It will not get out of this state, unless the timeout on the server is hit and some error is thrown then.I wish I could explain this simpler to be honest, but I don't really. Following are the logs from both
k6and my own small CLI tool ran withGODEBUG=http2debug=2. Maybe it will help.This is the K6 output that is hanging.
So the strange behaviour is here (snippet from above):
The content length and the length of the data frame don't match up. The fact that this is in fact 16383 rather than 16384 is already strange, but stranger still is the fact that it doesn't send yet another data frame with the last byte afterwards.
Notice that the failed request is roughly halfway, afterwards it will issue yet another 2 (successful)
GETrequests, but fail on thePOSTagain 🤷 .Expected behaviour
For
k6not to hang on these requests.Actual behaviour
k6hangs.