Filedownload Example Issues

[ajinkyaw]

[x ] I have read the Troubleshooting section of the ReadMe

What type of issues is this?

[ ] Request to support a new module [ ] Bug or problem compiling the library [x ] Bug or issue with library functionality (ie, sending data over TCP/IP) [ ] Question or request for help

What are you working with?

Modem: Sim800c Main processor board: ESP32 TinyGSM version: 0.10.6 Code:

Scenario, steps to reproduce

Executing Filedownload example

Expected result

Complete File download

Actual result

Client connection getting disconnected.

I am trying to download 1.4 mb file from tinygsm module, and for that i am using FileDownload example. however my device downloads 96,000 bytes and then stops. After debugging the prpgram i found out that client.connected() condition is turning false and that is why i am not be able to download the file further.

I am sharing the code snippet below, Please guide me resolving this issue.

TinyGsmClientSecure client(modemGSM);

if (client.connect(host, port)) {

         SerialDebug.println("connected");
     // Make a HTTP GET request:
     SerialDebug.println("Performing HTTP GET request...");
     client.print(String("GET ") + path + " HTTP/1.1\r\n");
     client.print(String("Host: ") + host + "\r\n");
     client.print("Connection: close\r\n\r\n");
     client.println();

}else {

    SerialDebug.println("connection failed");
        return;

}

while (client.connected() && millis() - timeout < 10000L) {
    while (client.available()) {
        char c = client.read();
        readBytes++;
        if(readBytes % 1000 == 0)
            SerialDebug.println(readBytes);
        timeout = millis();
    }
}

[Mr-HaleYa]

This can happen when this timeout is reached timeout < 10000L) meaning it hasn't received any data from the client for 10 seconds you could have poor signal so try moving to better signal or the server may be slow that you are connecting to. I use 5-sec timeout so the fact that you are getting this with 10 sec may be a problem elsewhere than this code block

first, I would try changing it to this

while (client.connected() && millis() - timeout < 30000L) {    // longer timeout
    while (client.available()) {
        char c = client.read();
        readBytes++;
        if(readBytes % 1000 == 0)
            SerialDebug.println(readBytes);
        timeout = millis();
    }
}

if that doesn't work try

while (client.connected() && millis() - timeout < 60000L) {    // even longer timeout
    while (client.available()) {
        char c = client.read();
        readBytes++;
        if(readBytes % 1000 == 0)
            SerialDebug.println(readBytes);
        timeout = millis();
    }
}

and if that doesn't work for the last effort you can remove the timeout, this may cause the code to get stuck here tho if it doesn't complete

while (client.connected()) {    // no timeout
    while (client.available()) {
        char c = client.read();
        readBytes++;
        if(readBytes % 1000 == 0)
            SerialDebug.println(readBytes);
    }
}

[ajinkyaw]

@Mr-HaleYa Hi, Thanks for the reply on the issue. Currently i am using the delay of 60000L. I am downloading an OTA file of 1.137060 MB ( thats approx 1.13 MB) . The gsm device disconnects multiple times ( at least 62 times ), and also at the gsm device disconnects at 1.136000 and then consecutively takes 30 to 40 retries to download remaining last few ( 1060 bytes ); And the chances of the gsm device to download that last chunk of data is literally 50%. It sometimes downloads or sometimes not, even after waiting for an hour.

Can you please guide me to resolve this issue as well.

[Mr-HaleYa]

what in the world.... ya that's not supposed to happen... I have NEVER had a download failed once started and I have 30 devices across my city that I push OTA updates to regularly. So what exactly are you downloading? because I'm downloading firmware yes I use an HTTP GET req to get the data from my server and just feed it to some updating code. explain your setup and how large your sketch is and all that stuff so I can try and help better. Maybe some pictures?

[ajinkyaw]

Hi . Wow that really great and encouraging me to hear that its working smoothly at your side. I am downloading the bin file uisng HTTP GET request only. My sketch is having below things included,

1. Wifi AP mode.
2. TinyGSM module.
3. Storing failed requests in txt file using SPIFFS. Here i am restricting that more than 500kb of space should not be used by these logs.

I cannot post the entire log here but i have reduced it so that you can get a picture. Let me explain the log given below, I have shared it from very initial stage of the response, where it prints the total length of the file 1137060. Now i am printing the total downloaded bytes , which is 1000,2000..... The statement "outside of loop" is printed when the execution comes out of

while (client.connected() && millis() - timeout < 60000L) {
}

loop. The statement 'Remaining Content Length: 1067864' indicates that how much data is downloaded and how much we are expecting in further chunks.

Content Length: 1137060 Remaining Content Length: 1137060 Reading Response.... 1000 2000 ...... 18000 19000 21000 22000 23000 outside of loop Remaining Content Length: 1113992 Reading Response.... 24000 25000 ...... 40000 41000 44000 45000 46000 outside of loop Remaining Content Length: 1090928 Reading Response.... 47000 48000 49000 ..... 66000 67000 68000 69000 outside of loop Remaining Content Length: 1067864 Reading Response.... 70000 71000 72000 .... 89000 90000 91000 93000 94000 95000 ..... ...... 1133000 1134000 outside of loop Remaining Content Length: 2413 Reading Response.... 1135000 1136000 outside of loop Request Timeout Request Timeout Connection to host failed... Request Timeout Request Timeout Request Timeout Request Timeout Request Timeout Request Timeout Request Timeout Request Timeout Request Timeout Request Timeout Request Timeout Request Timeout Request Timeout Remaining Content Length: 832 Reading Response.... outside of loop

Here the loop does the re-connection again and again , and downloading the last chunk has the probability of 50%..

[Mr-HaleYa]

Are you using Wi-Fi to download the firmware or are you using cellular?

[ajinkyaw]

I am using 2G network cellular to download using GSM Module.

[Mr-HaleYa]

can you share your sketch? because as far as I know the example was trash and I'm the only one that has GSMFOTA updates working because I wrote my own.

[ajinkyaw]

@Mr-HaleYa i will try to share the minimum version of the sketch.

[ajinkyaw]

@Mr-HaleYa https://codeshare.io/50yMxn Sorry for the delay, this is the best i could share.

[Mr-HaleYa]

A very important question is how much of the storage does the sketch take up when its compiled in Arduino IDE because if it's over or around 50% you CANNOT do this method because you will have nowhere to store the data as it writes it so it would probably crash but idk...

It is looking a lot like either you

1. run out of space bc sketch is too big
2. bad service so the cellular data is disconnecting
3. your client-server is not easily accessible for your device (their server sucks?)

or the code you use is just not working right...

also, I think it may be because you are trying with an HTTPS website that uses the stupid handshake data transfer protocol with I hate and never got to work.

YOU ALSO NEED THIS AT VERY TOP OF THE SKETCH #define TINY_GSM_RX_BUFFER 1024 // Set RX buffer to 1Kb or full HTTP response won't be received

[Mr-HaleYa]

I did say I wrote my own way of doing a GSMFOTA update that works super well. I was reluctant to share bc it took forever but if it helps you then your welcome 🙂

it actually modifies the Arduino OTA update code to "feed" the data from the server into it synthesizing a WiFi-connected client. to do that I had to modify their library so it is 2 files + the .ino

So this is the code that goes in the .ino file. I cut it down to the Raw but it works, you just have to connect to set up your connection on whatever modem you're using.

#define TINY_GSM_MODEM_SIM7000                    // My modem is SIM7000
#define TINY_GSM_RX_BUFFER   1024                 // Set RX buffer to 1Kb or full HTTP response wont be received

#include <TinyGsmClient.h>                        // https://github.com/vshymanskyy/TinyGSM

#include <Arduino.h>                              // used for GSMFOTA update
#include "GSMFOTAUpdate.h"                        // Updates firmware (this library is modified to work with GSM)

#define SerialAT  Serial1                         // Set serial for AT commands (to the module)
TinyGsm modem(SerialAT);                          // sets GSM modem to serial1
TinyGsmClient GSMclient(modem);                   // client for GSM

String fwImageURL     = "google.com/firmware/your.bin";   //example of your .bin location MUST BE HTTP SITE!!! NOT HTTPS
const int port        = 80;                               // HTTP = 80
const char server[]   = "google.com";                     //  head server address (MUST BE HTTP!)

long contentLength;                                       // How many bytes of data the .bin is
bool isValidContentType = false;                          // checks if the .bin is valid

/*---------------------------SETUP-----------------------------*/

void setup() {
  Serial.begin(9600);                                         // Start serial monitor at a baud rate of 9600
  Serial.println("\n------STARTING-UP------");

}

void connectClient() {    // Connect to your client server, for example
  while (!!!GSMclient.connect(server, port)) {
    Serial.print("*");
  }
}

/*-----------------------------LOOP-------------------------------*/

void loop() {

}

/*-------------------------GET-UPDATE-----------------------------*/

// used to extract header value from headers for ota update
String getHeaderValue(String header, String headerName) {
  return header.substring(strlen(headerName.c_str()));
}

void GSMOTA() {
  Serial.println("\n♦ ♦ UPDATE AVALIBLE ♦ ♦\n");
  connectClient();                                // Connect to your client
  if (GSMclient.connect(server, port)) {
    Serial.println("Fetching bin file at: " + String(fwImageURL));    // tells where its going to get the .bin and the name its looking for

    // Get the contents of the bin file
    GSMclient.print(String("GET ") + fwImageURL + " HTTP/1.1\r\n" +
                    "Host: " + String(server) + "\r\n" +
                    "Cache-Control: no-cache\r\n" +
                    "Connection: close\r\n\r\n");

    unsigned long timeout = millis();
    while (GSMclient.available() == 0) {
      if (millis() - timeout > 5000) {
        Serial.println("Client Timeout !");
        GSMclient.stop();
        return;
      }
    }
    // Once the response is available start reading reply
    while (GSMclient.available()) {

      // read line till /n
      String line = GSMclient.readStringUntil('\n');
      // remove space, to check if the line is end of headers
      line.trim();

      // if the the line is empty, this is end of headers
      // break the while and feed the remaining client to the Update.writeStream();
      if (!line.length()) {
        // headers ended
        break; // and get the OTA update started
      }

      // Check if the HTTP Response is 200 if not break and Exit Update
      if (line.startsWith("HTTP/1.1")) {
        if (line.indexOf("200") < 0) {
          Serial.println("Got a non 200 status code from server. Exiting OTA Update.");
          break;
        }
      }

      // extracting headers starting with content length
      if (line.startsWith("Content-Length: ")) {
        contentLength = atol((getHeaderValue(line, "Content-Length: ")).c_str());
        Serial.println("Got " + String(contentLength) + " bytes from server");
      }

      // Next, the content type
      if (line.startsWith("Content-Type: ")) {
        String contentType = getHeaderValue(line, "Content-Type: ");
        Serial.println("Got " + contentType + " payload.");
        if (contentType == "application/octet-stream") {
          isValidContentType = true;
        }
      }
    }
  } else {
    Serial.println("Connection to " + String(server) + " failed. Please check your setup");
  }
  // Check what is the contentLength and if content type is `application/octet-stream`
  Serial.println("contentLength : " + String(contentLength) + ", isValidContentType : " + String(isValidContentType));

  // check contentLength and content type
  if (contentLength && isValidContentType) {
    // Check if there is enough to OTA Update
    bool canBegin = Update.begin(contentLength);

    // If yes, begin
    if (canBegin) {
      Serial.println("Begin GSMFOTA update. This may take 2 - 25 minutes to complete depending on the connection and file size. Patience!!!");
      Serial.println("Firmware updating...");
      size_t written = Update.writeStream(GSMclient);

      if (written == contentLength) {
        Serial.println("Written : " + String(written) + " successfully");
      } else {
        Serial.println("Written only : " + String(written) + "/" + String(contentLength));
      }

      if (Update.end()) {
        Serial.println("OTA done!");
        if (Update.isFinished()) {
          Serial.println("Update successfully completed. Rebooting to activate.");
          ESP.restart();
        } else {
          Serial.println("Update not finished... Something went wrong!");
        }
      } else {
        Serial.println("Error Occurred! \nError: " + String(Update.getError()) + " " + String(Update.errorString()) );
        Serial.println("Will try to update again at a different time.\n");
      }
    } else {
      // not enough space to begin OTA, check .bin size
      Serial.println("Not enough space to begin OTA\n");
      GSMclient.flush();
    }
  } else {
    Serial.println("There was no content in the response\n");
    GSMclient.flush();
  }
}

You will need these two files also. MAKE SURE TO REMOVE THE .TXT at the end of the file name

GSMFOTAUpdate.h.txt GSMFOTAUpdater.cpp.txt

they need to be in the same folder as the .ino sketch like this

Inside of file GSMFOTAUpdate.h on line 46, you can change the LED PIN so to whatever you want so that it blinks when you receive data bytes, good to know if it's working

---

Other than that should get you going. Please let me know the progress you have. If you don't have an HTTP server then message me and ill upload your .bin to mine so you can test if it works.

- Regards Hale

[ajinkyaw]

@Mr-HaleYa can i contact you on your email. or somewhere else ?

[taufikterdidik]

@Mr-HaleYa Hi, Thanks for the reply on the issue. Currently i am using the delay of 60000L. I am downloading an OTA file of 1.137060 MB ( thats approx 1.13 MB) . The gsm device disconnects multiple times ( at least 62 times ), and also at the gsm device disconnects at 1.136000 and then consecutively takes 30 to 40 retries to download remaining last few ( 1060 bytes ); And the chances of the gsm device to download that last chunk of data is literally 50%. It sometimes downloads or sometimes not, even after waiting for an hour.

Can you please guide me to resolve this issue as well.

Me too.. I use SIM800L with ESP8266 (Wemos D1 Mini), the example use TINY_GSM_RX_BUFFER 1024, and my result with downloading 100kb (102400 byte = test_100k.bin file) is not completed, just stop at around 102100 bytes. I try with 2048 TINY_GSM_RX_BUFFER and the result more worst. So I try with TINY_GSM_RX_BUFFER 256 and it SUCCESS to download it with around 583 seconds, I try again with 512 and success too with around 544 seconds. I dont know why, but try it with lower buffer below 1024

[parthbhat13]

I did say I wrote my own way of doing a GSMFOTA update that works super well. I was reluctant to share bc it took forever but if it helps you then your welcome 🙂

it actually modifies the Arduino OTA update code to "feed" the data from the server into it synthesizing a WiFi-connected client. to do that I had to modify their library so it is 2 files + the .ino

So this is the code that goes in the .ino file. I cut it down to the Raw but it works, you just have to connect to set up your connection on whatever modem you're using.

#define TINY_GSM_MODEM_SIM7000                    // My modem is SIM7000
#define TINY_GSM_RX_BUFFER   1024                 // Set RX buffer to 1Kb or full HTTP response wont be received

#include <TinyGsmClient.h>                        // https://github.com/vshymanskyy/TinyGSM

#include <Arduino.h>                              // used for GSMFOTA update
#include "GSMFOTAUpdate.h"                        // Updates firmware (this library is modified to work with GSM)

#define SerialAT  Serial1                         // Set serial for AT commands (to the module)
TinyGsm modem(SerialAT);                          // sets GSM modem to serial1
TinyGsmClient GSMclient(modem);                   // client for GSM

String fwImageURL     = "google.com/firmware/your.bin";   //example of your .bin location MUST BE HTTP SITE!!! NOT HTTPS
const int port        = 80;                               // HTTP = 80
const char server[]   = "google.com";                     //  head server address (MUST BE HTTP!)

long contentLength;                                       // How many bytes of data the .bin is
bool isValidContentType = false;                          // checks if the .bin is valid

/*---------------------------SETUP-----------------------------*/

void setup() {
  Serial.begin(9600);                                         // Start serial monitor at a baud rate of 9600
  Serial.println("\n------STARTING-UP------");

}

void connectClient() {    // Connect to your client server, for example
  while (!!!GSMclient.connect(server, port)) {
    Serial.print("*");
  }
}

/*-----------------------------LOOP-------------------------------*/

void loop() {

}

/*-------------------------GET-UPDATE-----------------------------*/

// used to extract header value from headers for ota update
String getHeaderValue(String header, String headerName) {
  return header.substring(strlen(headerName.c_str()));
}

void GSMOTA() {
  Serial.println("\n♦ ♦ UPDATE AVALIBLE ♦ ♦\n");
  connectClient();                                // Connect to your client
  if (GSMclient.connect(server, port)) {
    Serial.println("Fetching bin file at: " + String(fwImageURL));    // tells where its going to get the .bin and the name its looking for

    // Get the contents of the bin file
    GSMclient.print(String("GET ") + fwImageURL + " HTTP/1.1\r\n" +
                    "Host: " + String(server) + "\r\n" +
                    "Cache-Control: no-cache\r\n" +
                    "Connection: close\r\n\r\n");

    unsigned long timeout = millis();
    while (GSMclient.available() == 0) {
      if (millis() - timeout > 5000) {
        Serial.println("Client Timeout !");
        GSMclient.stop();
        return;
      }
    }
    // Once the response is available start reading reply
    while (GSMclient.available()) {

      // read line till /n
      String line = GSMclient.readStringUntil('\n');
      // remove space, to check if the line is end of headers
      line.trim();

      // if the the line is empty, this is end of headers
      // break the while and feed the remaining client to the Update.writeStream();
      if (!line.length()) {
        // headers ended
        break; // and get the OTA update started
      }

      // Check if the HTTP Response is 200 if not break and Exit Update
      if (line.startsWith("HTTP/1.1")) {
        if (line.indexOf("200") < 0) {
          Serial.println("Got a non 200 status code from server. Exiting OTA Update.");
          break;
        }
      }

      // extracting headers starting with content length
      if (line.startsWith("Content-Length: ")) {
        contentLength = atol((getHeaderValue(line, "Content-Length: ")).c_str());
        Serial.println("Got " + String(contentLength) + " bytes from server");
      }

      // Next, the content type
      if (line.startsWith("Content-Type: ")) {
        String contentType = getHeaderValue(line, "Content-Type: ");
        Serial.println("Got " + contentType + " payload.");
        if (contentType == "application/octet-stream") {
          isValidContentType = true;
        }
      }
    }
  } else {
    Serial.println("Connection to " + String(server) + " failed. Please check your setup");
  }
  // Check what is the contentLength and if content type is `application/octet-stream`
  Serial.println("contentLength : " + String(contentLength) + ", isValidContentType : " + String(isValidContentType));

  // check contentLength and content type
  if (contentLength && isValidContentType) {
    // Check if there is enough to OTA Update
    bool canBegin = Update.begin(contentLength);

    // If yes, begin
    if (canBegin) {
      Serial.println("Begin GSMFOTA update. This may take 2 - 25 minutes to complete depending on the connection and file size. Patience!!!");
      Serial.println("Firmware updating...");
      size_t written = Update.writeStream(GSMclient);

      if (written == contentLength) {
        Serial.println("Written : " + String(written) + " successfully");
      } else {
        Serial.println("Written only : " + String(written) + "/" + String(contentLength));
      }

      if (Update.end()) {
        Serial.println("OTA done!");
        if (Update.isFinished()) {
          Serial.println("Update successfully completed. Rebooting to activate.");
          ESP.restart();
        } else {
          Serial.println("Update not finished... Something went wrong!");
        }
      } else {
        Serial.println("Error Occurred! \nError: " + String(Update.getError()) + " " + String(Update.errorString()) );
        Serial.println("Will try to update again at a different time.\n");
      }
    } else {
      // not enough space to begin OTA, check .bin size
      Serial.println("Not enough space to begin OTA\n");
      GSMclient.flush();
    }
  } else {
    Serial.println("There was no content in the response\n");
    GSMclient.flush();
  }
}

You will need these two files also. MAKE SURE TO REMOVE THE .TXT at the end of the file name

GSMFOTAUpdate.h.txt GSMFOTAUpdater.cpp.txt

they need to be in the same folder as the .ino sketch like this

Inside of file GSMFOTAUpdate.h on line 46, you can change the LED PIN so to whatever you want so that it blinks when you receive data bytes, good to know if it's working

Other than that should get you going. Please let me know the progress you have. If you don't have an HTTP server then message me and ill upload your .bin to mine so you can test if it works.

• Regards Hale

Hey, you have been a real savior, i had spent over 2 weeks to figure this out and i always came on this page and never really went down. thank you so damn very much for sharing it. now i wanna add more features to it like rollback if it fails and bluetooth config and so on.

[Mr-HaleYa]

Hey, you have been a real savior, i had spent over 2 weeks to figure this out and i always came on this page and never really went down. thank you so damn very much for sharing it. now i wanna add more features to it like rollback if it fails and bluetooth config and so on.

No problem, I'm glad it worked out for you.

If the download fails it will automatically rollback to the previous version on the device. this is so that if it is interrupted it will not brick the device. if you look I also have it take an MD5 hash as it downloads then one when it's done and if those don't match it deletes the file, also to prevent bricking. I added a lot of safety measures since this is the code in my companies products that I've built.

adding Bluetooth config should be super easy with the number of tutorials but personally, I prefer wifi config since more devices support wifi than Bluetooth. If you use wifi config I can strongly recommend WiFi-Manager Its a little outdated in the documentation but super easy to add and use.

[FreVanBE]

@Mr-HaleYa , Thank you for sharing your OTA functions! Very useful I tested it last week and it worked liked a charm. Unfortunately, since today I continuously receive a Stream Read Timeout. 02:15:41.717 -> Written only : 31005/214272 02:15:41.717 -> Error Occurred! 02:15:41.717 -> Error: 6 Stream Read Timeout 02:15:41.717 -> Will try to update again at a different time. he fails always around 31kb out of the 214kb Test sketch. The ping tests of the http server haven't changed. My node is connected via LTE CAT-M instead of NB-IoT. Any Idea how to tackle this issue?

[taufikterdidik]

You can implement range download file to continue download that file, I have done that with

• download file
• check if downloaded size if same with file size, if not
• (in for loop) download and concatenate file, starting with last successfull byte to end byte (file size). Implement this http request https://developer.mozilla.org/en-US/docs/Web/HTTP/Range_requests

[a9965]

Written only : 106394/1178016
Error Occurred!
Error: 6 Stream Read Timeout
Will try to update again at a different time.

After many attempts I can't get it to update. I have tried everything. Any suggestion? Thank you very much for sharing your effort and work. I am willing to pay for your work. Greetings and thank you.

[woodlist]

@Mr-HaleYa Are you able to make us free from toxic WiFi connection in another OTA library, available here: https://github.com/chrisjoyce911/esp32FOTA The library itself very good and advanced, but the author seems is very busy nor has anymore interest in maintenance of library. As I write in appropriate issue in respective repository, the declaration of network connection should look like this: `#include

include someSSL.h

include esp32fota.h

TinyGSM modem(Serial1); TinyGSMClient baseclient(modem); someSSLClient sslclient(baseclient); esp32Fota (sslclient);`

[woodlist]

I did say I wrote my own way of doing a GSMFOTA update that works super well. I was reluctant to share bc it took forever but if it helps you then your welcome

it actually modifies the Arduino OTA update code to "feed" the data from the server into it synthesizing a WiFi-connected client. to do that I had to modify their library so it is 2 files + the .ino

So this is the code that goes in the .ino file. I cut it down to the Raw but it works, you just have to connect to set up your connection on whatever modem you're using.

#define TINY_GSM_MODEM_SIM7000                    // My modem is SIM7000
#define TINY_GSM_RX_BUFFER   1024                 // Set RX buffer to 1Kb or full HTTP response wont be received

#include <TinyGsmClient.h>                        // https://github.com/vshymanskyy/TinyGSM

#include <Arduino.h>                              // used for GSMFOTA update
#include "GSMFOTAUpdate.h"                        // Updates firmware (this library is modified to work with GSM)

#define SerialAT  Serial1                         // Set serial for AT commands (to the module)
TinyGsm modem(SerialAT);                          // sets GSM modem to serial1
TinyGsmClient GSMclient(modem);                   // client for GSM

String fwImageURL     = "google.com/firmware/your.bin";   //example of your .bin location MUST BE HTTP SITE!!! NOT HTTPS
const int port        = 80;                               // HTTP = 80
const char server[]   = "google.com";                     //  head server address (MUST BE HTTP!)

long contentLength;                                       // How many bytes of data the .bin is
bool isValidContentType = false;                          // checks if the .bin is valid

/*---------------------------SETUP-----------------------------*/

void setup() {
  Serial.begin(9600);                                         // Start serial monitor at a baud rate of 9600
  Serial.println("\n------STARTING-UP------");

}

void connectClient() {    // Connect to your client server, for example
  while (!!!GSMclient.connect(server, port)) {
    Serial.print("*");
  }
}

/*-----------------------------LOOP-------------------------------*/

void loop() {

}

/*-------------------------GET-UPDATE-----------------------------*/

// used to extract header value from headers for ota update
String getHeaderValue(String header, String headerName) {
  return header.substring(strlen(headerName.c_str()));
}

void GSMOTA() {
  Serial.println("\n♦ ♦ UPDATE AVALIBLE ♦ ♦\n");
  connectClient();                                // Connect to your client
  if (GSMclient.connect(server, port)) {
    Serial.println("Fetching bin file at: " + String(fwImageURL));    // tells where its going to get the .bin and the name its looking for

    // Get the contents of the bin file
    GSMclient.print(String("GET ") + fwImageURL + " HTTP/1.1\r\n" +
                    "Host: " + String(server) + "\r\n" +
                    "Cache-Control: no-cache\r\n" +
                    "Connection: close\r\n\r\n");

    unsigned long timeout = millis();
    while (GSMclient.available() == 0) {
      if (millis() - timeout > 5000) {
        Serial.println("Client Timeout !");
        GSMclient.stop();
        return;
      }
    }
    // Once the response is available start reading reply
    while (GSMclient.available()) {

      // read line till /n
      String line = GSMclient.readStringUntil('\n');
      // remove space, to check if the line is end of headers
      line.trim();

      // if the the line is empty, this is end of headers
      // break the while and feed the remaining client to the Update.writeStream();
      if (!line.length()) {
        // headers ended
        break; // and get the OTA update started
      }

      // Check if the HTTP Response is 200 if not break and Exit Update
      if (line.startsWith("HTTP/1.1")) {
        if (line.indexOf("200") < 0) {
          Serial.println("Got a non 200 status code from server. Exiting OTA Update.");
          break;
        }
      }

      // extracting headers starting with content length
      if (line.startsWith("Content-Length: ")) {
        contentLength = atol((getHeaderValue(line, "Content-Length: ")).c_str());
        Serial.println("Got " + String(contentLength) + " bytes from server");
      }

      // Next, the content type
      if (line.startsWith("Content-Type: ")) {
        String contentType = getHeaderValue(line, "Content-Type: ");
        Serial.println("Got " + contentType + " payload.");
        if (contentType == "application/octet-stream") {
          isValidContentType = true;
        }
      }
    }
  } else {
    Serial.println("Connection to " + String(server) + " failed. Please check your setup");
  }
  // Check what is the contentLength and if content type is `application/octet-stream`
  Serial.println("contentLength : " + String(contentLength) + ", isValidContentType : " + String(isValidContentType));

  // check contentLength and content type
  if (contentLength && isValidContentType) {
    // Check if there is enough to OTA Update
    bool canBegin = Update.begin(contentLength);

    // If yes, begin
    if (canBegin) {
      Serial.println("Begin GSMFOTA update. This may take 2 - 25 minutes to complete depending on the connection and file size. Patience!!!");
      Serial.println("Firmware updating...");
      size_t written = Update.writeStream(GSMclient);

      if (written == contentLength) {
        Serial.println("Written : " + String(written) + " successfully");
      } else {
        Serial.println("Written only : " + String(written) + "/" + String(contentLength));
      }

      if (Update.end()) {
        Serial.println("OTA done!");
        if (Update.isFinished()) {
          Serial.println("Update successfully completed. Rebooting to activate.");
          ESP.restart();
        } else {
          Serial.println("Update not finished... Something went wrong!");
        }
      } else {
        Serial.println("Error Occurred! \nError: " + String(Update.getError()) + " " + String(Update.errorString()) );
        Serial.println("Will try to update again at a different time.\n");
      }
    } else {
      // not enough space to begin OTA, check .bin size
      Serial.println("Not enough space to begin OTA\n");
      GSMclient.flush();
    }
  } else {
    Serial.println("There was no content in the response\n");
    GSMclient.flush();
  }
}

You will need these two files also. MAKE SURE TO REMOVE THE .TXT at the end of the file name

GSMFOTAUpdate.h.txt GSMFOTAUpdater.cpp.txt

they need to be in the same folder as the .ino sketch like this

Inside of file GSMFOTAUpdate.h on line 46, you can change the LED PIN so to whatever you want so that it blinks when you receive data bytes, good to know if it's working

Other than that should get you going. Please let me know the progress you have. If you don't have an HTTP server then message me and ill upload your .bin to mine so you can test if it works.

• Regards Hale

In String fwImageURL = "google.com/firmware/your.bin"; the google.com should be deleted. I did a try over SSL connection, the log is below: 20:52:14.707 -> ♦ ♦ UPDATE AVALIBLE ♦ ♦ 20:52:14.707 -> 20:52:17.428 -> (SSLClient)(SSL_WARN)(connect): Arduino client is already connected? Continuing anyway... 20:52:18.108 -> Fetching bin file at: /bin/GFOTA_Althinker.ino.esp32.bin 20:52:18.710 -> Got application/octet-stream payload. 20:52:18.710 -> Got 504368 bytes from server 20:52:18.710 -> contentLength : 504368, isValidContentType : 1 20:52:18.710 -> Begin GSMFOTA update. This may take 2 - 25 minutes to complete depending on the connection and file size. Patience!!! 20:52:18.710 -> Firmware updating... 20:52:19.350 -> abort() was called at PC 0x40104425 on core 1 20:52:19.350 -> 20:52:19.350 -> ELF file SHA256: 0000000000000000 20:52:19.350 -> 20:52:19.350 -> Backtrace: 0x4008a9e8:0x3ffb1e00 0x4008ac61:0x3ffb1e20 0x40104425:0x3ffb1e40 0x4008807f:0x3ffb1e60 0x400e0091:0x3ffb1e80 0x400d5250:0x3ffb1ea0 0x400d54d0:0x3ffb1ec0 0x400d2d2a:0x3ffb1ee0 0x400d4c9e:0x3ffb1f80 0x400e1486:0x3ffb1fb0 0x4008c8f2:0x3ffb1fd0

[woodlist]

Perhaps my MC is an ESP32 CAM Al-thinker, which need to have set the GPIO3 grounded for firmware update mode. Who has experience on Al-thinker OTA?

[woodlist]

Decoding stack results: 0x4008a9e8: invoke_abort at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp32/panic.c line 156 0x4008ac61: abort at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/esp32/panic.c line 171 0x40104425: is_safe_write_address at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/spi_flash/flash_ops.c line 129 0x4008807f: spi_flash_erase_sector at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/spi_flash/flash_ops.c line 214 0x400e0091: EspClass::flashEraseSector(unsigned int) at C:\Users\User\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.6\cores\esp32\Esp.cpp line 325 0x400d5250: UpdateClass::_writeBuffer() at C:\Users\User\AppData\Local\Temp\arduino_build_767032\sketch\GSMFOTAUpdater.cpp line 197 0x400d54d0: UpdateClass::writeStream(Stream&) at C:\Users\User\AppData\Local\Temp\arduino_build_767032\sketch\GSMFOTAUpdater.cpp line 359 0x400d2d2a: GSMOTA() at C:\Users\User\Documents\Arduino\ESP\ESPCAM\GFOTA_Althinker/GFOTA_Althinker.ino line 363 0x400d4c9e: setup() at C:\Users\User\Documents\Arduino\ESP\ESPCAM\GFOTA_Althinker/GFOTA_Althinker.ino line 261 0x400e1486: loopTask(void*) at C:\Users\User\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.6\cores\esp32\main.cpp line 18 0x4008c8f2: vPortTaskWrapper at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/freertos/port.c line 143

[gunter72]

Hi @Mr-HaleYa , your code works fine, I have a problem however, after the first OTA update, subsequent OTA updates fail and always return the following errors:

23: 22: 59.030 -> Firmware updating ...
23: 23: 40.176 -> Written: 314416 successfully
23: 23: 40.176 -> Calculating Checksum ...
23: 23: 40.176 -> Checksum Passed!
23: 23: 40.342 -> E (116159) esp_image: Image hash failed - image is corrupt
23: 23: 40.342 -> Error Occurred!
23: 23: 40.342 -> Error: 9 Could Not Activate The Firmware
23: 23: 40.342 -> Will try to update again at a different time.

or this:

23: 21: 02.313 -> Firmware updating ...
23: 21: 43.864 -> Written: 314416 successfully
23: 21: 43.864 -> Calculating Checksum ...
23: 21: 43.864 -> Checksum Passed!
23: 21: 44.029 -> E (121517) esp_image: Checksum failed. Calculated 0xd read 0x5b
23: 21: 44.029 -> Error Occurred!
23: 21: 44.029 -> Error: 9 Could Not Activate The Firmware
23: 21: 44.029 -> Will try to update again at a different time.

Do you need to delete some memory partitions first?

thank you

[Ollebolle]

@a9965 @FreVanBE Did you ever get this working? I'm seeing the same exact error and not really sure what's going on. It worked perfectly once on my first try with one board and now it just throws this error consistently across different boards...

[shivaniumredkar1102]

I did say I wrote my own way of doing a GSMFOTA update that works super well. I was reluctant to share bc it took forever but if it helps you then your welcome slightly_smiling_face

it actually modifies the Arduino OTA update code to "feed" the data from the server into it synthesizing a WiFi-connected client. to do that I had to modify their library so it is 2 files + the .ino

So this is the code that goes in the .ino file. I cut it down to the Raw but it works, you just have to connect to set up your connection on whatever modem you're using.

#define TINY_GSM_MODEM_SIM7000                    // My modem is SIM7000
#define TINY_GSM_RX_BUFFER   1024                 // Set RX buffer to 1Kb or full HTTP response wont be received

#include <TinyGsmClient.h>                        // https://github.com/vshymanskyy/TinyGSM

#include <Arduino.h>                              // used for GSMFOTA update
#include "GSMFOTAUpdate.h"                        // Updates firmware (this library is modified to work with GSM)

#define SerialAT  Serial1                         // Set serial for AT commands (to the module)
TinyGsm modem(SerialAT);                          // sets GSM modem to serial1
TinyGsmClient GSMclient(modem);                   // client for GSM

String fwImageURL     = "google.com/firmware/your.bin";   //example of your .bin location MUST BE HTTP SITE!!! NOT HTTPS
const int port        = 80;                               // HTTP = 80
const char server[]   = "google.com";                     //  head server address (MUST BE HTTP!)

long contentLength;                                       // How many bytes of data the .bin is
bool isValidContentType = false;                          // checks if the .bin is valid

/*---------------------------SETUP-----------------------------*/

void setup() {
  Serial.begin(9600);                                         // Start serial monitor at a baud rate of 9600
  Serial.println("\n------STARTING-UP------");

}

void connectClient() {    // Connect to your client server, for example
  while (!!!GSMclient.connect(server, port)) {
    Serial.print("*");
  }
}

/*-----------------------------LOOP-------------------------------*/

void loop() {

}

/*-------------------------GET-UPDATE-----------------------------*/

// used to extract header value from headers for ota update
String getHeaderValue(String header, String headerName) {
  return header.substring(strlen(headerName.c_str()));
}

void GSMOTA() {
  Serial.println("\n♦ ♦ UPDATE AVALIBLE ♦ ♦\n");
  connectClient();                                // Connect to your client
  if (GSMclient.connect(server, port)) {
    Serial.println("Fetching bin file at: " + String(fwImageURL));    // tells where its going to get the .bin and the name its looking for

    // Get the contents of the bin file
    GSMclient.print(String("GET ") + fwImageURL + " HTTP/1.1\r\n" +
                    "Host: " + String(server) + "\r\n" +
                    "Cache-Control: no-cache\r\n" +
                    "Connection: close\r\n\r\n");

    unsigned long timeout = millis();
    while (GSMclient.available() == 0) {
      if (millis() - timeout > 5000) {
        Serial.println("Client Timeout !");
        GSMclient.stop();
        return;
      }
    }
    // Once the response is available start reading reply
    while (GSMclient.available()) {

      // read line till /n
      String line = GSMclient.readStringUntil('\n');
      // remove space, to check if the line is end of headers
      line.trim();

      // if the the line is empty, this is end of headers
      // break the while and feed the remaining client to the Update.writeStream();
      if (!line.length()) {
        // headers ended
        break; // and get the OTA update started
      }

      // Check if the HTTP Response is 200 if not break and Exit Update
      if (line.startsWith("HTTP/1.1")) {
        if (line.indexOf("200") < 0) {
          Serial.println("Got a non 200 status code from server. Exiting OTA Update.");
          break;
        }
      }

      // extracting headers starting with content length
      if (line.startsWith("Content-Length: ")) {
        contentLength = atol((getHeaderValue(line, "Content-Length: ")).c_str());
        Serial.println("Got " + String(contentLength) + " bytes from server");
      }

      // Next, the content type
      if (line.startsWith("Content-Type: ")) {
        String contentType = getHeaderValue(line, "Content-Type: ");
        Serial.println("Got " + contentType + " payload.");
        if (contentType == "application/octet-stream") {
          isValidContentType = true;
        }
      }
    }
  } else {
    Serial.println("Connection to " + String(server) + " failed. Please check your setup");
  }
  // Check what is the contentLength and if content type is `application/octet-stream`
  Serial.println("contentLength : " + String(contentLength) + ", isValidContentType : " + String(isValidContentType));

  // check contentLength and content type
  if (contentLength && isValidContentType) {
    // Check if there is enough to OTA Update
    bool canBegin = Update.begin(contentLength);

    // If yes, begin
    if (canBegin) {
      Serial.println("Begin GSMFOTA update. This may take 2 - 25 minutes to complete depending on the connection and file size. Patience!!!");
      Serial.println("Firmware updating...");
      size_t written = Update.writeStream(GSMclient);

      if (written == contentLength) {
        Serial.println("Written : " + String(written) + " successfully");
      } else {
        Serial.println("Written only : " + String(written) + "/" + String(contentLength));
      }

      if (Update.end()) {
        Serial.println("OTA done!");
        if (Update.isFinished()) {
          Serial.println("Update successfully completed. Rebooting to activate.");
          ESP.restart();
        } else {
          Serial.println("Update not finished... Something went wrong!");
        }
      } else {
        Serial.println("Error Occurred! \nError: " + String(Update.getError()) + " " + String(Update.errorString()) );
        Serial.println("Will try to update again at a different time.\n");
      }
    } else {
      // not enough space to begin OTA, check .bin size
      Serial.println("Not enough space to begin OTA\n");
      GSMclient.flush();
    }
  } else {
    Serial.println("There was no content in the response\n");
    GSMclient.flush();
  }
}

You will need these two files also. MAKE SURE TO REMOVE THE .TXT at the end of the file name

GSMFOTAUpdate.h.txt GSMFOTAUpdater.cpp.txt

they need to be in the same folder as the .ino sketch like this

Inside of file GSMFOTAUpdate.h on line 46, you can change the LED PIN so to whatever you want so that it blinks when you receive data bytes, good to know if it's working

Other than that should get you going. Please let me know the progress you have. If you don't have an HTTP server then message me and ill upload your .bin to mine so you can test if it works.

• Regards Hale

I am trying with Quectale MC60 module, to update firmware in ESP32 using tinyGSM library. Similar issue I am facing while downloading the firmware. The file size of my .bin file is almost 1.1MB and the connection break consistently in the middle of downloading process. I checked my signal strength it is -92DB not good, but still is there any way to do OTA? How can I improve signal strength? Is the same code which you given can I use for my usecase ?

[taufikterdidik]

I did say I wrote my own way of doing a GSMFOTA update that works super well. I was reluctant to share bc it took forever but if it helps you then your welcome slightly_smiling_face it actually modifies the Arduino OTA update code to "feed" the data from the server into it synthesizing a WiFi-connected client. to do that I had to modify their library so it is 2 files + the .ino So this is the code that goes in the .ino file. I cut it down to the Raw but it works, you just have to connect to set up your connection on whatever modem you're using.

#define TINY_GSM_MODEM_SIM7000                    // My modem is SIM7000
#define TINY_GSM_RX_BUFFER   1024                 // Set RX buffer to 1Kb or full HTTP response wont be received

#include <TinyGsmClient.h>                        // https://github.com/vshymanskyy/TinyGSM

#include <Arduino.h>                              // used for GSMFOTA update
#include "GSMFOTAUpdate.h"                        // Updates firmware (this library is modified to work with GSM)

#define SerialAT  Serial1                         // Set serial for AT commands (to the module)
TinyGsm modem(SerialAT);                          // sets GSM modem to serial1
TinyGsmClient GSMclient(modem);                   // client for GSM

String fwImageURL     = "google.com/firmware/your.bin";   //example of your .bin location MUST BE HTTP SITE!!! NOT HTTPS
const int port        = 80;                               // HTTP = 80
const char server[]   = "google.com";                     //  head server address (MUST BE HTTP!)

long contentLength;                                       // How many bytes of data the .bin is
bool isValidContentType = false;                          // checks if the .bin is valid

/*---------------------------SETUP-----------------------------*/

void setup() {
  Serial.begin(9600);                                         // Start serial monitor at a baud rate of 9600
  Serial.println("\n------STARTING-UP------");

}

void connectClient() {    // Connect to your client server, for example
  while (!!!GSMclient.connect(server, port)) {
    Serial.print("*");
  }
}

/*-----------------------------LOOP-------------------------------*/

void loop() {

}

/*-------------------------GET-UPDATE-----------------------------*/

// used to extract header value from headers for ota update
String getHeaderValue(String header, String headerName) {
  return header.substring(strlen(headerName.c_str()));
}

void GSMOTA() {
  Serial.println("\n♦ ♦ UPDATE AVALIBLE ♦ ♦\n");
  connectClient();                                // Connect to your client
  if (GSMclient.connect(server, port)) {
    Serial.println("Fetching bin file at: " + String(fwImageURL));    // tells where its going to get the .bin and the name its looking for

    // Get the contents of the bin file
    GSMclient.print(String("GET ") + fwImageURL + " HTTP/1.1\r\n" +
                    "Host: " + String(server) + "\r\n" +
                    "Cache-Control: no-cache\r\n" +
                    "Connection: close\r\n\r\n");

    unsigned long timeout = millis();
    while (GSMclient.available() == 0) {
      if (millis() - timeout > 5000) {
        Serial.println("Client Timeout !");
        GSMclient.stop();
        return;
      }
    }
    // Once the response is available start reading reply
    while (GSMclient.available()) {

      // read line till /n
      String line = GSMclient.readStringUntil('\n');
      // remove space, to check if the line is end of headers
      line.trim();

      // if the the line is empty, this is end of headers
      // break the while and feed the remaining client to the Update.writeStream();
      if (!line.length()) {
        // headers ended
        break; // and get the OTA update started
      }

      // Check if the HTTP Response is 200 if not break and Exit Update
      if (line.startsWith("HTTP/1.1")) {
        if (line.indexOf("200") < 0) {
          Serial.println("Got a non 200 status code from server. Exiting OTA Update.");
          break;
        }
      }

      // extracting headers starting with content length
      if (line.startsWith("Content-Length: ")) {
        contentLength = atol((getHeaderValue(line, "Content-Length: ")).c_str());
        Serial.println("Got " + String(contentLength) + " bytes from server");
      }

      // Next, the content type
      if (line.startsWith("Content-Type: ")) {
        String contentType = getHeaderValue(line, "Content-Type: ");
        Serial.println("Got " + contentType + " payload.");
        if (contentType == "application/octet-stream") {
          isValidContentType = true;
        }
      }
    }
  } else {
    Serial.println("Connection to " + String(server) + " failed. Please check your setup");
  }
  // Check what is the contentLength and if content type is `application/octet-stream`
  Serial.println("contentLength : " + String(contentLength) + ", isValidContentType : " + String(isValidContentType));

  // check contentLength and content type
  if (contentLength && isValidContentType) {
    // Check if there is enough to OTA Update
    bool canBegin = Update.begin(contentLength);

    // If yes, begin
    if (canBegin) {
      Serial.println("Begin GSMFOTA update. This may take 2 - 25 minutes to complete depending on the connection and file size. Patience!!!");
      Serial.println("Firmware updating...");
      size_t written = Update.writeStream(GSMclient);

      if (written == contentLength) {
        Serial.println("Written : " + String(written) + " successfully");
      } else {
        Serial.println("Written only : " + String(written) + "/" + String(contentLength));
      }

      if (Update.end()) {
        Serial.println("OTA done!");
        if (Update.isFinished()) {
          Serial.println("Update successfully completed. Rebooting to activate.");
          ESP.restart();
        } else {
          Serial.println("Update not finished... Something went wrong!");
        }
      } else {
        Serial.println("Error Occurred! \nError: " + String(Update.getError()) + " " + String(Update.errorString()) );
        Serial.println("Will try to update again at a different time.\n");
      }
    } else {
      // not enough space to begin OTA, check .bin size
      Serial.println("Not enough space to begin OTA\n");
      GSMclient.flush();
    }
  } else {
    Serial.println("There was no content in the response\n");
    GSMclient.flush();
  }
}

You will need these two files also. MAKE SURE TO REMOVE THE .TXT at the end of the file name GSMFOTAUpdate.h.txt GSMFOTAUpdater.cpp.txt they need to be in the same folder as the .ino sketch like this Inside of file GSMFOTAUpdate.h on line 46, you can change the LED PIN so to whatever you want so that it blinks when you receive data bytes, good to know if it's working Other than that should get you going. Please let me know the progress you have. If you don't have an HTTP server then message me and ill upload your .bin to mine so you can test if it works.

• Regards Hale

I am trying with Quectale MC60 module, to update firmware in ESP32 using tinyGSM library. Similar issue I am facing while downloading the firmware. The file size of my .bin file is almost 1.1MB and the connection break consistently in the middle of downloading process. I checked my signal strength it is -92DB not good, but still is there any way to do OTA? How can I improve signal strength? Is the same code which you given can I use for my usecase ?

yeah, I also got the problem before.. Here I solved that: wrap a download code in a loop (example with "for") and check while connection break (download not completed) to resume download. implement resume download

[roysG]

Can you share you code please, it will be better to understand you, thanks

[danilo-mw]

Hello everybody! I was having problems doing OTA updates with GSM at SIMCOM A7672E module and ESP32. After many attempts, I deleted the line:

#define TINY_GSM_YIELD() { delay(2); }

and now everything is fine!

my problem was that OTA download was very slow.

I believe the TINY_GSM_YIELD setting is not necessary on ESP32 in most cases.

Thanks!

[eevora]

Hello everybody! I was having problems doing OTA updates with GSM at SIMCOM A7672E module and ESP32. After many attempts, I deleted the line:

#define TINY_GSM_YIELD() { delay(2); }

and now everything is fine!

my problem was that OTA download was very slow.

I believe the TINY_GSM_YIELD setting is not necessary on ESP32 in most cases.

Thanks!

Hello @danilo-mw did you updated ota with HTTPS??

[danilo-mw]

Hello everybody! I was having problems doing OTA updates with GSM at SIMCOM A7672E module and ESP32. After many attempts, I deleted the line: #define TINY_GSM_YIELD() { delay(2); } and now everything is fine! my problem was that OTA download was very slow. I believe the TINY_GSM_YIELD setting is not necessary on ESP32 in most cases. Thanks!

Hello @danilo-mw did you updated ota with HTTPS??

Hello! i didn't try HTTPS, only HTTP :(

[GerryDush]

@Mr-HaleYa Hi, Thanks for the reply on the issue. Currently i am using the delay of 60000L. I am downloading an OTA file of 1.137060 MB ( thats approx 1.13 MB) . The gsm device disconnects multiple times ( at least 62 times ), and also at the gsm device disconnects at 1.136000 and then consecutively takes 30 to 40 retries to download remaining last few ( 1060 bytes ); And the chances of the gsm device to download that last chunk of data is literally 50%. It sometimes downloads or sometimes not, even after waiting for an hour. Can you please guide me to resolve this issue as well.

Me too.. I use SIM800L with ESP8266 (Wemos D1 Mini), the example use TINY_GSM_RX_BUFFER 1024, and my result with downloading 100kb (102400 byte = test_100k.bin file) is not completed, just stop at around 102100 bytes. I try with 2048 TINY_GSM_RX_BUFFER and the result more worst. So I try with TINY_GSM_RX_BUFFER 256 and it SUCCESS to download it with around 583 seconds, I try again with 512 and success too with around 544 seconds. I dont know why, but try it with lower buffer below 1024

Thank you very much, I used the same way also success