HTTPclient POST redirect bug - "your client has issued a malformed or illegal request" error.

sairan22 commented 1 year ago

Board

any ESP32

Device Description

any device

Hardware Configuration

No.

Version

latest master (checkout manually)

IDE Name

PlatformIO

Operating System

https://script.google.com

Flash frequency

40Mhz

PSRAM enabled

yes

Upload speed

115200

Description

HTTPclient incorrectly set the "Content-Length" when following redirect (dropped to GET/HEAD).

To fix it, replace line 605 - 607 with the following (same as that in ESP8266HTTPclient):

    addHeader(F("Content-Length"), String(payload && size > 0 ? size : 0));

Sketch

1. Create Google sheet.
2. On the Google sheet, create Apps Script to respond (any text) to POST request.
3. Deploy the Apps Script as Web App and get the URL.
4. Write ESP32 code to send POST request to the URL and log the debug output.

Debug Message

[ 38277][D][HTTPClient_200fix.cpp:1172] connect():  connected to script.googleusercontent.com:443
[ 38286][D][HTTPClient_200fix.cpp:1234] sendHeader(): Sending header:
-----
GET /macros/echo?user_content_key=_google_generated_key__ HTTP/1.1
Host: script.googleusercontent.com
User-Agent: ESP32HTTPClient
Connection: keep-alive
Accept-Encoding: identity;q=1,chunked;q=0.1,*;q=0
Content-Length: 32

-----

[ 38335][V][ssl_client.cpp:374] send_ssl_data(): Writing HTTP request with 491 bytes...
[ 38464][V][HTTPClient_200fix.cpp:1269] handleHeaderResponse(): RX: 'HTTP/1.0 400 Bad Request'
[ 38465][V][HTTPClient_200fix.cpp:1269] handleHeaderResponse(): RX: 'Content-Type: text/html; charset=UTF-8'
[ 38472][V][HTTPClient_200fix.cpp:1269] handleHeaderResponse(): RX: 'Referrer-Policy: no-referrer'
[ 38480][V][HTTPClient_200fix.cpp:1269] handleHeaderResponse(): RX: 'Content-Length: 1555'
[ 38488][V][HTTPClient_200fix.cpp:1269] handleHeaderResponse(): RX: 'Date: Fri, 09 Jun 2023 02:58:05 GMT'
[ 38497][V][HTTPClient_200fix.cpp:1269] handleHeaderResponse(): RX: ''
[ 38504][D][HTTPClient_200fix.cpp:1326] handleHeaderResponse(): code: 400
[ 38510][D][HTTPClient_200fix.cpp:1329] handleHeaderResponse(): size: 1555
[ 38517][D][HTTPClient_200fix.cpp:644] sendRequest(): sendRequest code=400

[ 38532][E][ssl_client.cpp:37] _handle_error(): [data_to_read():366]: (-76) UNKNOWN ERROR CODE (004C)
[ 38536][V][ssl_client.cpp:326] stop_ssl_socket(): Cleaning SSL connection.
[ 38540][D][HTTPClient_200fix.cpp:1465] writeToStreamDataBlock(): connection closed or file end (written: 1555).
[ 38549][D][HTTPClient_200fix.cpp:408] disconnect(): tcp is closed

[ 38555][D][HTTPClient_200fix.cpp:408] disconnect(): tcp is closed

[ 38561][V][ssl_client.cpp:326] stop_ssl_socket(): Cleaning SSL connection.
done, reply: [HTTP] code: 400 payload: <!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 400 (Bad Request)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>400.</b> <ins>Thatâ€™s an error.</ins>
  <p>Your client has issued a malformed or illegal request.  <ins>Thatâ€™s all we know.</ins>

Other Steps to Reproduce

No response

I have checked existing issues, online documentation and the Troubleshooting Guide

[X] I confirm I have checked existing issues, online documentation and Troubleshooting guide.

SuGlider commented 1 year ago

@sairan22 - is it a GET or POST? GET has no Content-Length POST usually must have a non zero Content-Length

sairan22 commented 1 year ago

GET is okay since Content-Length is always zero and the bug doesn't get triggered.

The bug is triggered this way:

client sends POST with non-zero payload.
server replies with code HTTP_CODE_FOUND or HTTP_CODE_SEE_OTHER.
client sends GET (to follow the redirect) with wrongly retained non-zero Content-Length.

I should've mentioned that the early portion of the debug log (where POST were sent) had been truncated by my terminal logger since it was too long.

mrengineer7777 commented 1 year ago

It seems you have also modified the library HTTPClient_200fix.cpp which then crashes.

sairan22 commented 1 year ago

Nope. The debug log is on the original codes. (I copied original HTTPclient to HTTPclient_200fix, and do the debug log). I then modified it with the fix copied from ESP8266HTTPclient.cpp which fixed the bug.

Carl-UB commented 10 months ago

I've encountered this issue as well under the same circumstances. Any idea when it's likely to be fixed? I've implemented the change as described above by sairan22 which worked, but would be good to get it fixed properly!

espressif / arduino-esp32