No way to send raw data in post requests in vegeta

Version and Runtime

Paste the output of `vegeta -version` here.
If you are not running the latest version of Vegeta, please try upgrading because your issue may have already been fixed.

12.8.0

Expected Behaviour

Sending raw data to the server in post requests through vegeta, and the server can receive request data, whenever curl can work out with the same effect.

When we execute the following curl command, the server can successfully receive the raw data that is specified with option -d. The raw data is a plain string, instead of normal key=value string.

curl -H 'eq:Y6IxBROYbOAA5RxSJSCbUw==' -X POST https://testURL -d '6+7JxfoPsvEsjADT8WWYFx1PJRmeVle1UXZJGBniea5J3DuB8OV67JAH1Uty296rTy0ReTBU4Sf1WcAPmN7WPGadp76DWypQKoKOxA4TYUtfJnrfSWndtxvm+VNU4h7u84Ao2t4kd3eFtccIu95C784bZx4F9lSICS6R0eVVHNj5zSn/PkwfiMS7rczSPL1Q0Ixn2rCgmG6zbrqvzw+PrOzSb/spclDG9BM2Xgjpe/ydXpQXUV50Ncg6CYLOKEU7qDSe4JQiKycHGnKFLljv6uzhJWvlIXsvyGF7B6BT6jM0BHF+z45W1j0rNDIH9Zxx6jyEel7FDHBA44K4KnF2PsyecRgDuY1UV579pAZjCji0UFb3Lsu2jJiLT6gcjG3j5pJgWY85H2C+fLjfU48BY9X4iWtJpeoISHwbqJ/4kzE1tdVUGN4fwWLYx5afX4bcbZbzMelAnXcK+huaUA/GLOqR2neNIfifBrg1MLcUW+BNorAJwfRJIf0/Lvc0t+q/'

What we expect to see in vegeta is that the sever can also receive the raw data string when we execute the following vegeta commands:

echo "6+7JxfoPsvEsjADT8WWYFx1PJRmeVle1UXZJGBniea5J3DuB8OV67JAH1Uty296rTy0ReTBU4Sf1WcAPmN7WPGadp76DWypQKoKOxA4TYUtfJnrfSWndtxvm+VNU4h7u84Ao2t4kd3eFtccIu95C784bZx4F9lSICS6R0eVVHNj5zSn/PkwfiMS7rczSPL1Q0Ixn2rCgmG6zbrqvzw+PrOzSb/spclDG9BM2Xgjpe/ydXpQXUV50Ncg6CYLOKEU7qDSe4JQiKycHGnKFLljv6uzhJWvlIXsvyGF7B6BT6jM0BHF+z45W1j0rNDIH9Zxx6jyEel7FDHBA44K4KnF2PsyecRgDuY1UV579pAZjCji0UFb3Lsu2jJiLT6gcjG3j5pJgWY85H2C+fLjfU48BY9X4iWtJpeoISHwbqJ/4kzE1tdVUGN4fwWLYx5afX4bcbZbzMelAnXcK+huaUA/GLOqR2neNIfifBrg1MLcUW+BNorAJwfRJIf0/Lvc0t+q/" > body.txt

echo "POST http://testURL" | vegeta attack -rate 50/s -duration 1m -body body.txt -header "eq: Y6IxBROYbOAA5RxSJSCbUw==" -header "Content-Type: text/plain" > results.bin

vegeta report results.bin

Actual Behaviour

No data has been received in the server side system.

Steps to Reproduce

echo "6+7JxfoPsvEsjADT8WWYFx1" > body.txt
echo "POST http://testURL" | vegeta attack -rate 1/s -duration 10s -body body.txt -header "eq: Y6IxBROYbOAA5RxSJSCbUw==" -header "Content-Type: text/plain" > results.bin &
Check the server system and its logs to verify whether it has received the requests data: "6+7JxfoPsvEsjADT8WWYFx1"

Additional Context

@fengnex: You can easily check whether the problem is with Vegeta or your server by running that attack against this test server in the Vegeta repo (it requires you have Go installed, let me know if you want a static binary instead):

git clone https://github.com/tsenart/vegeta
cd vegeta
go run ./internal/cmd/echosrv/main.go :6060 -dump

Then in another tab or terminal window, run the attack against localhost:6060 and see if your request body is printed to stdout. If it is, then you have a problem with your server, because vegeta is actually sending the request body. Otherwise, I'll look into it.

@tsenart Thank you! I'm afraid that you may want to look into it, please.

This is because when attacking internal/cmd/echosrv/main.go with the following command:

echo "6+7JxfoPsvEsjADT8WWYFx1PJRmeVle1UXZJGBniea5J3DuB8OV67JAH1Uty296rTy0ReTBU4Sf1WcAPmN7WPGadp76DWypQKoKOxA4TYUtfJnrfSWndtxvm+VNU4h7u84Ao2t4kd3eFtccIu95C784bZx4F9lSICS6R0eVVHNj5zSn/PkwfiMS7rczSPL1Q0Ixn2rCgmG6zbrqvzw+PrOzSb/spclDG9BM2Xgjpe/ydXpQXUV50Ncg6CYLOKEU7qDSe4JQiKycHGnKFLljv6uzhJWvlIXsvyGF7B6BT6jM0BHF+z45W1j0rNDIH9Zxx6jyEel7FDHBA44K4KnF2PsyecRgDuY1UV579pAZjCji0UFb3Lsu2jJiLT6gcjG3j5pJgWY85H2C+fLjfU48BY9X4iWtJpeoISHwbqJ/4kzE1tdVUGN4fwWLYx5afX4bcbZbzMelAnXcK+huaUA/GLOqR2neNIfifBrg1MLcUW+BNorAJwfRJIf0/Lvc0t+q/" > body.txt
echo "POST http://localhost:6060" | vegeta attack -rate 1/s -duration 5s -body body.txt > results.bin

the output we got in the client side looks like the following; you see, the content's format is json format and its content is also totally different from the output of the curl command which use the same request data :

{"attack":"","seq":0,"code":200,"timestamp":"2020-07-10T14:09:00.908243982+08:00","latency":5415603,"bytes_out":513,"bytes_in":647,"error":"","body":"UE9TVCAvIEhUVFAvMS4xDQpIb3N0OiBsb2NhbGhvc3Q6NjA2MA0KQWNjZXB0LUVuY29kaW5nOiBnemlwDQpDb250ZW50LUxlbmd0aDogNTEzDQpVc2VyLUFnZW50OiBHby1odHRwLWNsaWVudC8xLjENClgtVmVnZXRhLVNlcTogMA0KDQo2KzdKeGZvUHN2RXNqQURUOFdXWUZ4MVBKUm1lVmxlMVVYWkpHQm5pZWE1SjNEdUI4T1Y2N0pBSDFVdHkyOTZyVHkwUmVUQlU0U2YxV2NBUG1ON1dQR2FkcDc2RFd5cFFLb0tPeEE0VFlVdGZKbnJmU1duZHR4dm0rVk5VNGg3dTg0QW8ydDRrZDNlRnRjY0l1OTVDNzg0Ylp4NEY5bFNJQ1M2UjBlVlZITmo1elNuL1Brd2ZpTVM3cmN6U1BMMVEwSXhuMnJDZ21HNnpicnF2encrUHJPelNiL3NwY2xERzlCTTJYZ2pwZS95ZFhwUVhVVjUwTmNnNkNZTE9LRVU3cURTZTRKUWlLeWNIR25LRkxsanY2dXpoSld2bElYc3Z5R0Y3QjZCVDZqTTBCSEYrejQ1VzFqMHJORElIOVp4eDZqeUVlbDdGREhCQTQ0SzRLbkYyUHN5ZWNSZ0R1WTFVVjU3OXBBWmpDamkwVUZiM0xzdTJqSmlMVDZnY2pHM2o1cEpnV1k4NUgyQytmTGpmVTQ4Qlk5WDRpV3RKcGVvSVNId2JxSi80a3pFMXRkVlVHTjRmd1dMWXg1YWZYNGJjYlpiek1lbEFuWGNLK2h1YVVBL0dMT3FSMm5lTklmaWZCcmcxTUxjVVcrQk5vckFKd2ZSSklmMC9MdmMwdCtxLwo=","method":"POST","url":"http://localhost:6060","headers":[{"key":"Content-Length","value":"647"}{"key":"Content-Type","value":"text/plain; charset=utf-8"}{"key":"Date","value":"Fri, 10 Jul 2020 06:09:00 GMT"}]}

In contrast, if we access the program in internal/cmd/echosrv/main.go with curl by executing the command:

curl -X POST http://localhost:6060 -d '6+7JxfoPsvEsjADT8WWYFx1PJRmeVle1UXZJGBniea5J3DuB8OV67JAH1Uty296rTy0ReTBU4Sf1WcAPmN7WPGadp76DWypQKoKOxA4TYUtfJnrfSWndtxvm+VNU4h7u84Ao2t4kd3eFtccIu95C784bZx4F9lSICS6R0eVVHNj5zSn/PkwfiMS7rczSPL1Q0Ixn2rCgmG6zbrqvzw+PrOzSb/spclDG9BM2Xgjpe/ydXpQXUV50Ncg6CYLOKEU7qDSe4JQiKycHGnKFLljv6uzhJWvlIXsvyGF7B6BT6jM0BHF+z45W1j0rNDIH9Zxx6jyEel7FDHBA44K4KnF2PsyecRgDuY1UV579pAZjCji0UFb3Lsu2jJiLT6gcjG3j5pJgWY85H2C+fLjfU48BY9X4iWtJpeoISHwbqJ/4kzE1tdVUGN4fwWLYx5afX4bcbZbzMelAnXcK+huaUA/GLOqR2neNIfifBrg1MLcUW+BNorAJwfRJIf0/Lvc0t+q/'

The output of the previous operation in the same console looks like:

POST / HTTP/1.1
Host: localhost:6060
Accept: */*
Content-Length: 512
Content-Type: application/x-www-form-urlencoded
User-Agent: curl/7.29.0

6+7JxfoPsvEsjADT8WWYFx1PJRmeVle1UXZJGBniea5J3DuB8OV67JAH1Uty296rTy0ReTBU4Sf1WcAPmN7WPGadp76DWypQKoKOxA4TYUtfJnrfSWndtxvm+VNU4h7u84Ao2t4kd3eFtccIu95C784bZx4F9lSICS6R0eVVHNj5zSn/PkwfiMS7rczSPL1Q0Ixn2rCgmG6zbrqvzw+PrOzSb/spclDG9BM2Xgjpe/ydXpQXUV50Ncg6CYLOKEU7qDSe4JQiKycHGnKFLljv6uzhJWvlIXsvyGF7B6BT6jM0BHF+z45W1j0rNDIH9Zxx6jyEel7FDHBA44K4KnF2PsyecRgDuY1UV579pAZjCji0UFb3Lsu2jJiLT6gcjG3j5pJgWY85H2C+fLjfU48BY9X4iWtJpeoISHwbqJ/4kzE1tdVUGN4fwWLYx5afX4bcbZbzMelAnXcK+huaUA/GLOqR2neNIfifBrg1MLcUW+BNorAJwfRJIf0/Lvc0t+q/

Sorry, the instructions I sent you to run the test server were wrong, we need to pass the -dump flag before the listen address. I don't know how you got that output for curl.

Run this instead go run ./internal/cmd/echosrv/main.go -dump :6060 and try again.

@tsenart You are right. The output of curl requests and vegeta's attacking are highly similar to each other, but there are still some differences lie in Accept, Accept-Encoding and Content-Type:

Accept: Not Specified (vegeta) vs. / (curl)
Accept-Encoding: gzip (vegeta) vs. Not Specified (curl)
Content-Type: Not Specified (vegeta) vs. application/x-www-form-urlencoded (curl)

Considering the curl's request can be successfully received and processed by the server, maybe the differences, such as the gzip Accept-Encoding, cause the problem. We still need to verify the cause and solve it.

The output of vegeta and it keeps the same when adding -format=http to vegeta attack:

POST / HTTP/1.1
Host: 192.168.88.104:6060
Accept-Encoding: gzip
Content-Length: 513
User-Agent: Go-http-client/1.1
X-Vegeta-Seq: 0

6+7JxfoPsvEsjADT8WWYFx1PJRmeVle1UXZJGBniea5J3DuB8OV67JAH1Uty296rTy0ReTBU4Sf1WcAPmN7WPGadp76DWypQKoKOxA4TYUtfJnrfSWndtxvm+VNU4h7u84Ao2t4kd3eFtccIu95C784bZx4F9lSICS6R0eVVHNj5zSn/PkwfiMS7rczSPL1Q0Ixn2rCgmG6zbrqvzw+PrOzSb/spclDG9BM2Xgjpe/ydXpQXUV50Ncg6CYLOKEU7qDSe4JQiKycHGnKFLljv6uzhJWvlIXsvyGF7B6BT6jM0BHF+z45W1j0rNDIH9Zxx6jyEel7FDHBA44K4KnF2PsyecRgDuY1UV579pAZjCji0UFb3Lsu2jJiLT6gcjG3j5pJgWY85H2C+fLjfU48BY9X4iWtJpeoISHwbqJ/4kzE1tdVUGN4fwWLYx5afX4bcbZbzMelAnXcK+huaUA/GLOqR2neNIfifBrg1MLcUW+BNorAJwfRJIf0/Lvc0t+q/
2020/07/10 23:10:49 Rate: 1.001/s

Similarly, the output of the curl request in previous posts:

POST / HTTP/1.1
Host: 192.168.88.104:6060
Accept: */*
Content-Length: 512
Content-Type: application/x-www-form-urlencoded
User-Agent: curl/7.29.0

6+7JxfoPsvEsjADT8WWYFx1PJRmeVle1UXZJGBniea5J3DuB8OV67JAH1Uty296rTy0ReTBU4Sf1WcAPmN7WPGadp76DWypQKoKOxA4TYUtfJnrfSWndtxvm+VNU4h7u84Ao2t4kd3eFtccIu95C784bZx4F9lSICS6R0eVVHNj5zSn/PkwfiMS7rczSPL1Q0Ixn2rCgmG6zbrqvzw+PrOzSb/spclDG9BM2Xgjpe/ydXpQXUV50Ncg6CYLOKEU7qDSe4JQiKycHGnKFLljv6uzhJWvlIXsvyGF7B6BT6jM0BHF+z45W1j0rNDIH9Zxx6jyEel7FDHBA44K4KnF2PsyecRgDuY1UV579pAZjCji0UFb3Lsu2jJiLT6gcjG3j5pJgWY85H2C+fLjfU48BY9X4iWtJpeoISHwbqJ/4kzE1tdVUGN4fwWLYx5afX4bcbZbzMelAnXcK+huaUA/GLOqR2neNIfifBrg1MLcUW+BNorAJwfRJIf0/Lvc0t+q/2020/07/11 18:31:30 Rate: 1.001/s

Vegeta is ok in the scenario we have discussed, and it has no relationship with the Accept-Encoding and other http header options. In this case, there is a character \x0A (new line) automatically added at the end of the parameter in the context of vegeta, which is not the case when it comes to curl and causes the server cannot process the string successfully.

Could you please check where the \x0A has been added and remove it whenever there is no such a character at the end of the body.txt? Then, vegeta will be consistent with curl in this respect.@tsenart

tsenart / vegeta