ArduinoHTTPClient times out (HTTP_ERROR_TIMED_OUT = -3) even though server sends 200 HTTP Response

[x] I have read the Troubleshooting section of the ReadMe

What type of issues is this?

[x] Question or request for help

What are you working with?

Modem: SIM7000G Main processor board: ESP32 TinyGSM version: vshymanskyy/TinyGSM@^0.11.7

Code:

bool MQTTEndpoint::getMQTTEndpoint(Client &client)
{
    char final_url[256]; 
    sprintf(final_url, "http://%s/api/firmware/%s/", domain, apiKey);
    bool endpoint_found = false;
    while (!endpoint_found) 
    {
        HttpClient http(client, domain, 80);
        http.setTimeout(10000);
        int err = http.get(final_url);
        if (err != 0)
        {
            Logger.logMessage(LOG_LEVEL_ERROR, "HTTP GET not successful");
        }
        err = http.responseStatusCode();
        Logger.logMessage(LOG_LEVEL_DEBUG, "The HTTP Response code is: %d", err);
        if (err >= 300 || err < 200)
        {
            Logger.logMessage(LOG_LEVEL_ERROR, "Failed to get MQTT Endpoint due to bad HTTP response code");
            delay(30000);
            continue;
        }
        String payload = http.readString();

Scenario, steps to reproduce

TCP connection to server appears sucessful due to the dumped AT commands and checking the server lock i can see a 200 response from the module's IP. However the ArduinoHTTPClient is returning -3 error code for timeout. I can see the 509 byte response from the request but ArduinoHTTPClient isn't getting the data for some reason.

Server response log:

<module ip> - - [07/Jan/2024:21:55:02 +0000] "GET /api/firmware/<redacted>/ HTTP/1.1" 200 149 "-" "Arduino/2.2.0" "<module ip>" response-time=0.023

Expected result

the HTTP response is passed to HTTPClient with http.responseStatusCode()=200

Actual result

http.responseStatusCode(); = -3 indicating that the response has timed out even though server returned 200

Debug and AT command log

AT+CIPCLOSE=0

+CME ERROR: operation not allowed
AT+CIPSTART=0,"TCP","<domain>",80

OK

0, CONNECT OK
AT+CIPSEND=0,3

>GET 
DATA ACCEPT:0,3
AT+CIPSEND=0,1

>  
DATA ACCEPT:0,1
AT+CIPSEND=0,87

>http://<full url>/ 
DATA ACCEPT:0,87
AT+CIPSEND=0,9

> HTTP/1.1 
DATA ACCEPT:0,9
AT+CIPSEND=0,2

>

DATA ACCEPT:0,2
AT+CIPSEND=0,6

>Host:  
DATA ACCEPT:0,6
AT+CIPSEND=0,29

><domain> 
DATA ACCEPT:0,29
AT+CIPSEND=0,2

>

DATA ACCEPT:0,2
AT+CIPSEND=0,10

>User-Agent 
DATA ACCEPT:0,10
AT+CIPSEND=0,2

>:  
DATA ACCEPT:0,2
AT+CIPSEND=0,13

>Arduino/2.2.0 
DATA ACCEPT:0,13
AT+CIPSEND=0,2

>

DATA ACCEPT:0,2
AT+CIPSEND=0,10

>Connection 
DATA ACCEPT:0,10
AT+CIPSEND=0,2

>:  
DATA ACCEPT:0,2
AT+CIPSEND=0,5

>close 
DATA ACCEPT:0,5
AT+CIPSEND=0,2

>

DATA ACCEPT:0,2
AT+CIPSEND=0,2

>

DATA ACCEPT:0,2

+CIPRXGET: 1,0
AT+CIPRXGET=4,0

+CIPRXGET: 4,0,509

OK

0, CLOSED
The HTTP Response code is: -3
Failed to get MQTT Endpoint due to bad HTTP response code

The problem was that http client instance was not being passed a valid reference to the gsmClient instance

What didn't work:

I think this is because gsmClient i was using not initialized with a modem properly.

// main.cpp
TinyGsm modem = cellFunctions.modem;
TinyGsmClient gsmClient = cellFunctions.gsmClient;

// cellFunctions.h
class CellFunctions
{
public:
        CellFunctions(HardwareSerial &serialGsmRef, HardwareSerial &SerialATRef, StreamDebugger &debugger);
.
.
.

        TinyGsm modem;           // instance of TinyGsm modem class
        TinyGsmClient gsmClient; // instance of the TinyGsm client class to use with mqtt
};

// getMQTTEndpoint.cpp
bool MQTTEndpoint::getMQTTEndpoint(Client &client)
{
    char final_url[256]; 
    sprintf(final_url, "http://%s/api/firmware/%s/", domain, apiKey);
    bool endpoint_found = false;
    while (!endpoint_found) 
    {
        HttpClient http(client, domain, 80);
        http.setTimeout(10000);
        int err = http.get(final_url);
        if (err != 0)
        {
            Logger.logMessage(LOG_LEVEL_ERROR, "HTTP GET not successful");
        }
        err = http.responseStatusCode();
        Logger.logMessage(LOG_LEVEL_DEBUG, "The HTTP Response code is: %d", err);
        if (err >= 300 || err < 200)
        {
            Logger.logMessage(LOG_LEVEL_ERROR, "Failed to get MQTT Endpoint due to bad HTTP response code");
            delay(30000);
            continue;
        }
        String payload = http.readString();

What did work

//main.cpp
TinyGsm modem = cellFunctions.modem;
TinyGsmClient gsmClient(modem);  // here i actually pass the modem to the gsmClient before using the client to call getMQTTEndpoint

vshymanskyy / TinyGSM