vshymanskyy / TinyGSM

A small Arduino library for GSM modules, that just works
GNU Lesser General Public License v3.0
1.91k stars 713 forks source link

TinyGSM File Download. #570

Closed naturalmechanics closed 2 years ago

naturalmechanics commented 2 years ago

[ x] I have read the Troubleshooting section of the ReadMe

What type of issues is this?

[ ] Request to support a new module

[ ] Bug or problem compiling the library [ x] Bug or issue with library functionality (ie, sending data over TCP/IP) [ ] Question or request for help

What are you working with?

Modem: u-blox Model: SARA-R412M-02B Revision: M0.10.00 [Mar 28 2019 17:13:41] SVN: 05 IMEI: 354679093770504 Main processor board: SODAQ SARA R412 TinyGSM version: 0.11.3 Code:

const char server_ota[] = "ota.webaro.de";//"116.202.190.107";
        const int  port_ota     = 80;
        const char host_ota[]   = "ota.webaro.de";

        if (!client.connect(server_ota, port_ota)) {
            SerialUSB.println(" fail to connect to server .... ");
            delay(1000);
            return false;
        }

        const char resource[]    = "/UPDATE.BIN";

        // Make a HTTP GET request:
        client.print(String("GET ") + resource + " HTTP/1.0\r\n");
        client.print(String("Host: ") + host_ota + "\r\n");
        client.print("Connection: close\r\n\r\n");

        const uint32_t clientReadTimeout   = 5000;
        uint32_t       clientReadStartTime = millis();
        String         headerBuffer;
        bool           finishedHeader = false;
        uint32_t       contentLength  = 0;

        while (!finishedHeader) {                                                           // SerialUSB.println("waiting for resp ..!");
            int nlPos;

            if (client.available()) {
            clientReadStartTime = millis();
                while (client.available()) {
                    char c = client.read();
                    headerBuffer += c;

                    //Uncomment the lines below to see the data coming into the buffer
                    if (c < 16)
                      SerialUSB.print('0');
                    //SerialUSB.print(c, HEX);
                    SerialUSB.print(' ');
                    if (isprint(c))
                      SerialUSB.print(reinterpret_cast<char> (c));
                    else {
                      SerialUSB.print('*');
                      SerialUSB.print(' ');
                    }

                    // Let's exit and process if we find a new line
                    //if (headerBuffer.indexOf(F("\r\n")) >= 0) break;
                }
            }
            // See if we have a new line.
            nlPos = headerBuffer.indexOf(F("\r\n"));

            if (nlPos > 0) {
            headerBuffer.toLowerCase();
            // Check if line contains content-length
            if (headerBuffer.startsWith(F("content-length:"))) {
                contentLength =
                    headerBuffer.substring(headerBuffer.indexOf(':') + 1).toInt();
                SerialUSB.print(F("Got Content Length: "));  // uncomment for
            SerialUSB.println(contentLength);            // confirmation
            }

            headerBuffer.remove(0, nlPos + 2);  // remove the line
            } else if (nlPos == 0) {
            // if the new line is empty (i.e. "\r\n" is at the beginning of the line),
            // we are done with the header.
            finishedHeader = true;
            }
        }

Scenario, steps to reproduce

I want to download the file UPDATE.BIN. But the response is only printing "*" s.

Replace server_ota with 116.202.190.107 and hostname with tracking.webaro.de. You do get a response this time.

However, the response is :

H T T P / 1 . 1 2 0 0 O K 0 * 0 * S e r v e r : n g i n x / 1 . 1 8 . 0 ( U b u n t u ) 0 * 0 * D a t e : S u n , 2 5 J u l 2 0 2 1 2 0 : 2 5 : 2 2 G M T 0 * 0 * C o n t e n t - T y p e : t e x t / h t m l ; c h a r s e t = U T F - 8 0 * 0 * C o n t e n t - L e n g t h : 2 2 9 0 * 0 * C o n n e c t i o n : c l o s e 0 * 0 * c a c h e - c o n t r o l : m a x - a g e = 3 6 0 0 0 * 0 * l a s t - m o d i f i e d : M o n , 2 6 A p r 2 0 2 1 0 9 : 3 9 : 1 8 G M T 0 * 0 * e t a g : W / " 1 2 0 5 9 4 4 - 2 2 9 - 2 0 2 1 - 0 4 - 2 6 T 0 9 : 3 9 : 1 8 . 0 0 0 Z " 0 * 0 * 0 * 0 *

Then it downloads an HTML FILE :

<html><header> <title>Machine Tracking</title> <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto:300,400,500"> <script src="/src.008db41b.js"></script> </header><body> <div id="app"></div> </body></html><html><header> <title>Machine Tracking</title> <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto:300,400,500"> <script src="/src.008db41b.js"></script> </header><body> <div id="app"></div> </body></html><html><header> <title>Machine Tracking</title> <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto:300,400,500"> <script src="/src.008db41b.js"></script> </header><body> <div id="app"></div> </body></html>

The URLs i have given you are the correct URLs. You can check the files there

Expected result

H T T P / 1 . 1 2 0 0 O K 0 * 0 * S e r v e r : n g i n x / 1 . 1 8 . 0 ( U b u n t u ) 0 * 0 * D a t e : S u n , 2 5 J u l 2 0 2 1 2 0 : 2 5 : 2 2 G M T 0 * 0 * C o n t e n t - T y p e : a p p l i c a t i o n / o c t a t e - s t r e a m ; c h a r s e t = U T F - 8 0 * 0 * C o n t e n t - L e n g t h : 1 3 3 9 4 4 0 * 0 * C o n n e c t i o n : c l o s e 0 * 0 * c a c h e - c o n t r o l : m a x - a g e = 3 6 0 0 0 * 0 * l a s t - m o d i f i e d : M o n , 2 6 A p r 2 0 2 1 0 9 : 3 9 : 1 8 G M T 0 * 0 * e t a g : W / " 1 2 0 5 9 4 4 - 2 2 9 - 2 0 2 1 - 0 4 - 2 6 T 0 9 : 3 9 : 1 8 . 0 0 0 Z " 0 * 0 * 0 * 0 *

Actual result

Either only * are printed, or the wrong HTTP response.

Debug and AT command log

For the first case, where only '+' are printed :


+CEREG: 0,5

OK
 OK
AT+CEREG?

+CEREG: 0,5

OK
Network connected
Connecting to APN: 

AT+CGATT=1

OK
AT+CGDCONT=1,"IP",""

OK
AT+CGACT=1,1

OK
OK, attempting to send authentication .. 
opening socket
AT+USOCR=6

+USOCR: 0
success

OK
continue ...
AT+USOCO=0,"ota.webaro.de",80
connecting to IP 2

OK
AT+USOWR=0,26

@GET /UPDATE.BIN HTTP/1.0

+USOWR: 0,26

OK
AT+USOWR=0,21

@Host: ota.webaro.de

+USOWR: 0,21

OK
AT+USOWR=0,21

@Connection: close

+USOWR: 0,21

OK
AT+USORD=0,0

+USORD: 0,0
OK
AT+USOCTL=0,10

+USOCTL: 0,10,4

OK

+UUSORD: 0,1360
AT+USORD=0,0

+USORD: 0,1360
OK
AT+USORD=0,1360

+CME ERROR: Operation not allowed

 * AT+USORD=0,0

+USORD: 0,16320
OK
AT+USORD=0,2047

+CME ERROR: Operation not allowed

 * AT+USORD=0,0

+USORD: 0,16320
OK
AT+USORD=0,2047

+CME ERROR: Operation not allowed

 * AT+USORD=0,0

+USORD: 0,16384
OK
AT+USORD=0,2047

+CME ERROR: Operation not allowed

 * AT+USORD=0,0

+USORD: 0,16384
OK
AT+USORD=0,2047

+CME ERROR: Operation not allowed

 * AT+USORD=0,0

+USORD: 0,16384
OK
AT+USORD=0,2047

+CME ERROR: Operation not allowed

 * AT+USORD=0,0

+USORD: 0,16384
OK
AT+USORD=0,2047

+CME ERROR: Operation not allowed

 * AT+USORD=0,0

+USORD: 0,16384
OK
AT+USORD=0,2047

+CME ERROR: Operation not allowed

 * AT+USORD=0,0

+USORD: 0,16384
OK
AT+USORD=0,2047

+CME ERROR: Operation not allowed

 * AT+USORD=0,0

+USORD: 0,16384
OK
AT+USORD=0,2047

+CME ERROR: Operation not allowed

 * AT+USORD=0,0

+USORD: 0,16384
OK
AT+USORD=0,2047

+CME ERROR: Operation not allowed

 * AT+USORD=0,0

+USORD: 0,16384
OK
AT+USORD=0,2047

+CME ERROR: Operation not allowed

 * AT+USORD=0,0

+USORD: 0,16384
OK
AT+USORD=0,2047

+CME ERROR: Operation not allowed

 * AT+USORD=0,0

The modem has been initialized properly. It can send POST requests without problems. The GET is performing strangely.

I undertstand, that this is possibly not a tinyGSM issue. Perhaps the servers are configured wrong. But I hope that you will point me in the right direction. Thank you

adrianca88 commented 2 years ago

Example on ESP32 and SIM808:

/** 
     * Download file using http Connection
     * WARNING: buffer must be pre-allocated
     */
    uint32_t DownloadFile(const char* file, uint8_t* buffer, uint32_t size)
    {
        if (!file || !buffer || size <= 0) {
            return 0;
        }

        // Connect to network
        if (!mSIM->isGprsConnected()) {
          connectGSM();
        }

        // Connect to server
        Serial.print(F("Connecting to server..."));
        if (!mSIM_Client->connect(mServer, HTTP_PORT, HTTP_TIMEOUT_SEC)) {
            Serial.println(F(" FAIL"));
            return 0;
        }
        Serial.println(F(" OK!"));

        mSIM_Client->print(String("GET ") + file + " HTTP/1.0\r\n");
        mSIM_Client->print(String("Host: ") + mServer + ":" + HTTP_PORT + "\r\n");
        mSIM_Client->print("Connection: close\r\n\r\n");

        // Parse header
        Serial.println(F("Waiting for response header"));

        uint32_t clientReadStartTime = millis();
        String headerBuffer;
        bool finishedHeader = false;
        uint32_t contentLength = 0;

        while (!finishedHeader) {
            int nlPos;

            if (mSIM_Client->available()) {
                clientReadStartTime = millis();
                while (mSIM_Client->available()) {
                  char c = mSIM_Client->read();
                  headerBuffer += c;

                  /*// Uncomment the lines below to see the data coming into the buffer
                  if (isprint(c))
                    Serial.print(reinterpret_cast<char> (c));
                  else
                    Serial.print('*');*/

                  if (headerBuffer.indexOf(F("\r\n")) >= 0) {
                    break;
                  }
                }
            } else if (millis() - clientReadStartTime > DOWNLOAD_FILE_TIMEOUT) {
                Serial.println(F(">>> Client Timeout !"));
                break;
            }

            // See if we have a new line.
            nlPos = headerBuffer.indexOf(F("\r\n"));

            if (nlPos > 0) {
                headerBuffer.toLowerCase();
                // Check if line contains content-length
                if (headerBuffer.startsWith(F("content-length:"))) {
                    contentLength = headerBuffer.substring(headerBuffer.indexOf(':') + 1).toInt();
                }
                headerBuffer.remove(0, nlPos + 2);  // remove the line
            } else if (nlPos == 0) {
                // if the new line is empty (i.e. "\r\n" is at the beginning of the line), we are done with the header.
                finishedHeader = true;
            }
        }

        if (contentLength != size || !finishedHeader) {
            Serial.println(F("Error parsing header content-length size or different sizes\n"));
            mSIM_Client->stop();
            return 0;
        }

        // Read content
        Serial.println(F("Reading response data"));
        uint32_t readLength = 0;
        clientReadStartTime = millis();
        while (readLength < contentLength && mSIM_Client->connected() && millis() - clientReadStartTime < DOWNLOAD_FILE_TIMEOUT) {
            while (mSIM_Client->available()) {
                buffer[readLength++] = mSIM_Client->read();
                clientReadStartTime = millis();
            }
        }        

        mSIM_Client->stop();

        return readLength;
    }
shivaniumredkar1102 commented 2 years ago

Example on ESP32 and SIM808:

/** 
     * Download file using http Connection
     * WARNING: buffer must be pre-allocated
     */
    uint32_t DownloadFile(const char* file, uint8_t* buffer, uint32_t size)
    {
        if (!file || !buffer || size <= 0) {
            return 0;
        }

        // Connect to network
        if (!mSIM->isGprsConnected()) {
          connectGSM();
        }

        // Connect to server
        Serial.print(F("Connecting to server..."));
        if (!mSIM_Client->connect(mServer, HTTP_PORT, HTTP_TIMEOUT_SEC)) {
            Serial.println(F(" FAIL"));
            return 0;
        }
        Serial.println(F(" OK!"));

        mSIM_Client->print(String("GET ") + file + " HTTP/1.0\r\n");
        mSIM_Client->print(String("Host: ") + mServer + ":" + HTTP_PORT + "\r\n");
        mSIM_Client->print("Connection: close\r\n\r\n");

        // Parse header
        Serial.println(F("Waiting for response header"));

        uint32_t clientReadStartTime = millis();
        String headerBuffer;
        bool finishedHeader = false;
        uint32_t contentLength = 0;

        while (!finishedHeader) {
            int nlPos;

            if (mSIM_Client->available()) {
                clientReadStartTime = millis();
                while (mSIM_Client->available()) {
                  char c = mSIM_Client->read();
                  headerBuffer += c;

                  /*// Uncomment the lines below to see the data coming into the buffer
                  if (isprint(c))
                    Serial.print(reinterpret_cast<char> (c));
                  else
                    Serial.print('*');*/

                  if (headerBuffer.indexOf(F("\r\n")) >= 0) {
                    break;
                  }
                }
            } else if (millis() - clientReadStartTime > DOWNLOAD_FILE_TIMEOUT) {
                Serial.println(F(">>> Client Timeout !"));
                break;
            }

            // See if we have a new line.
            nlPos = headerBuffer.indexOf(F("\r\n"));

            if (nlPos > 0) {
                headerBuffer.toLowerCase();
                // Check if line contains content-length
                if (headerBuffer.startsWith(F("content-length:"))) {
                    contentLength = headerBuffer.substring(headerBuffer.indexOf(':') + 1).toInt();
                }
                headerBuffer.remove(0, nlPos + 2);  // remove the line
            } else if (nlPos == 0) {
                // if the new line is empty (i.e. "\r\n" is at the beginning of the line), we are done with the header.
                finishedHeader = true;
            }
        }

        if (contentLength != size || !finishedHeader) {
            Serial.println(F("Error parsing header content-length size or different sizes\n"));
            mSIM_Client->stop();
            return 0;
        }

        // Read content
        Serial.println(F("Reading response data"));
        uint32_t readLength = 0;
        clientReadStartTime = millis();
        while (readLength < contentLength && mSIM_Client->connected() && millis() - clientReadStartTime < DOWNLOAD_FILE_TIMEOUT) {
            while (mSIM_Client->available()) {
                buffer[readLength++] = mSIM_Client->read();
                clientReadStartTime = millis();
            }
        }        

        mSIM_Client->stop();

        return readLength;
    }

Is over the air update is working using above code. I am facing timeout issue in file downloading using MC60.

SRGDamia1 commented 2 years ago

If this isn't a library issue, can it be closed?

shivaniumredkar1102 commented 2 years ago

Yes

On Wed, 24 Nov 2021, 9:33 pm Sara Damiano, @.***> wrote:

If this isn't a library issue, can it be closed?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/vshymanskyy/TinyGSM/issues/570#issuecomment-978014434, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHCGGVXJXCIVUMKPLDXBW63UNUEFRANCNFSM5A66BVTQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.