fast WiFi file download

[RafigRzayev]

I am trying to download a large file(e.g. 100MB) from a given download link into ESP32 (SD card).

The download speed that I see is very low (about 30 kB/sec).

I am using following simplified code:

void download(String download_url) {
  // Init SD card and create a new file
  SD_MMC.begin();
  FILE* file = fopen("/sdcard/new_file.wav", "ab");

  // Get file stream from internet
  HTTPClient audio_download_handler;
  audio_download_handler.begin(download_url);
  int httpCode = audio_download_handler.GET();
  WiFiClient* stream = audio_download_handler.getStreamPtr();

  // Download data and write into SD card
  size_t downloaded_data_size = 0;
  const size_t SONG_SIZE = audio_download_handler.getSize();
  while (downloaded_data_size < SONG_SIZE) {
    size_t available_data_size = stream->available();
    if (available_data_size > 0) {
      uint8_t* audio_data = (uint8_t*)malloc(available_data_size);
      stream->readBytes(audio_data, available_data_size);
      write(fileno(file), audio_data, available_data_size);
      downloaded_data_size += available_data_size;
      free(audio_data);
    }
  }

  fclose(file);
  SD_MMC.end();
}

It uses HTTPClient. Of course, time used for memory allocation and SD card write operations will increase the download time, but not at such scale.

As I have understood from https://github.com/espressif/arduino-esp32/issues/1648 , it looks like HTTPClient cannot offer high speed, people were talking about few kilobytes per second.

1) So firstly I wanted to ask, has anyone reached high download speed ( ~1MB/sec) by using HTTPClient?

2) If no, would it be possible to optimize the download speed in future releases?

3) If yes, can you share how you did it?

4) Meanwhile, can someone direct me towards an alternative approach that can offer high speed download?

Download link example: https://www2.cs.uic.edu/~i101/SoundFiles/PinkPanther60.wav.

[ToyboxZach]

You definitely can get to over 1Mb/s I do it consistently.

My rough implementation is as follows:

while(success &&
        http.connected() &&
        (downloadRemaining > 0 || downloadRemaining == -1) ) {
        // get available data size
                auto size = _getStream().available();
        if (size > 0) {
            auto c = _getStream().read(cur_buffer, ((size > CHUNK_SIZE) ? CHUNK_SIZE : size));
            cur_buffer = cur_buffer + c;

            auto totalSize = cur_buffer - buff;
            if (totalSize + 2 * CHUNK_SIZE > buffSize)
            {

                success &= _writeToFile(buff, totalSize);
            }
            if (downloadRemaining > 0) {
                downloadRemaining -= c;
            }
        }
               vTaskDelay(1);

Quickly looking at your code some obvious problems are: You are mallocing every loop, that is a lot of extra processing when you could just pre-allocate it.

Your write to the SD card every time you have any data. Writing to the SD card is pretty slow and you need to minimize the amount of writes you do and the number of different block chunks. With my implementation I actually scrapped any file system and just write directly to blocks on the SD card.

Your loop is constantly running so it is taking processing time to check stream->available() constantly, when that time could be better used by the chip actually filling up the buffer.

Fixing these issues should give you massive improvements.

[RafigRzayev]

@ToyboxZach Thank you very much for help.

I am 100% sure that frequent mallocing and SD write operation is not the bottleneck. I created a new project as per your recommendations. It has single pre-allocated buffer, and SD write only takes place once CHUNK_SIZE amount of data has been received. There was very small improvement in the download speed (maybe 5%). I have never reached a speed above 40kB/sec. Also, I measured the time it takes for SD write operations, it is very small. In fact, my original code wasn't using FS library for write operation. For example, for a 1MB file, a total of 52 seconds went into the download process, out of which just 0.7 seconds was spent on SD write. I even tried by completely commenting out SD write code, and change in total download time was not noticeable. You can find the code below:

#include "FS.h"
#include "SD_MMC.h"
#include <WiFi.h>
#include <HTTPClient.h>

#define CHUNK_SIZE 65536

// ENTER YOUR WIFI and optionally a download link
const char* ssid = "";
const char* password = "";
String download_url = "https://www2.cs.uic.edu/~i101/SoundFiles/preamble.wav";

struct download_info_t {
  String file_name;
  String url;
};

download_info_t Download_info;

void setup() {
  Serial.begin(115200);
  delay(1000);

  // Initiate SD card
  if (!SD_MMC.begin()) {
    Serial.println("Card Mount Failed");
  }

  // Connect to wifi
  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("Wifi connected");

  // Enter download url and name for new file
  Download_info.file_name = "new_file.wav";
  Download_info.url = download_url;

  // Create download task
  xTaskCreate(download_task, "downloader", 10000, &Download_info, 55, NULL);
}

void loop() {
  vTaskDelay(10000);
}

void download_task(void* params) {
  for (;;) {
    download( ((download_info_t*)params)->file_name, ((download_info_t*)params)->url);
    vTaskDelay(pdMS_TO_TICKS(2000));
  }
}

bool download(String file_name, String download_url) {
  // ********************* INIT HTTP *********************
  HTTPClient http;
  if (!http.begin(download_url) ) {
    Serial.println("http.begin(download_url) error");
    return false;
  }

  // ********************* SEND GET REQUEST *********************
  size_t try_counter = 0;
  const size_t TRY_LIMIT = 20;
  int httpCode = -1;
  do {
    httpCode = http.GET();
    Serial.print(".");
    vTaskDelay(pdMS_TO_TICKS(250));
    if (try_counter++ == TRY_LIMIT) {
      Serial.println("Connection timeout");
      return false;
    }
  } while (httpCode != HTTP_CODE_OK);
  Serial.println("GET Success");

  // ********************* RECEIVE FILE STREAM *********************
  WiFiClient* stream = http.getStreamPtr();
  try_counter = 0;
  do {
    stream = http.getStreamPtr();
    vTaskDelay(pdMS_TO_TICKS(250));
    Serial.print(".");
    if (try_counter++ == TRY_LIMIT) {
      Serial.println("Connection timeout");
      return false;
    }
  } while (!stream->available());
  Serial.println("File stream received");

  // ********************* CREATE NEW FILE *********************
  String fullpath = String("/sdcard/") + file_name;
  FILE* file = fopen(fullpath.c_str(), "ab");
  if (file == NULL) {
    Serial.println("Error opening file");
    return false;
  }
  Serial.println("Opened empty file");

  // ********************* DOWNLOAD PROCESS *********************
  uint8_t* buffer_ = (uint8_t*)malloc(CHUNK_SIZE);
  uint8_t* cur_buffer = buffer_;
  const size_t TOTAL_SIZE = http.getSize();
  Serial.print("TOTAL SIZE : ");
  Serial.println(TOTAL_SIZE);
  size_t downloadRemaining = TOTAL_SIZE;
  Serial.println("Download START");

  auto start_ = millis();
  while ( downloadRemaining > 0 && http.connected() ) {
    auto data_size = stream->available();
    if (data_size > 0) {
      auto available_buffer_size = CHUNK_SIZE - (cur_buffer - buffer_);
      auto read_count = stream->read(cur_buffer, ((data_size > available_buffer_size) ? available_buffer_size : data_size));
      cur_buffer += read_count;
      downloadRemaining -= read_count;
      // If one chunk of data has been accumulated, write to SD card
      if (cur_buffer - buffer_ == CHUNK_SIZE) {
        write(fileno(file), buffer_, CHUNK_SIZE);
        cur_buffer = buffer_;
      }
    }
    vTaskDelay(1);
  }
  auto end_ = millis();

  Serial.println("Download END");

  size_t time_ = (end_ - start_) / 1000;
  String speed_ = String(TOTAL_SIZE / time_);
  Serial.println("Speed: " + speed_ + " bytes/sec");

  fclose(file);
  free(buffer_);
  return true;
}

[RafigRzayev]

According to https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-guides/wifi.html#esp32-wi-fi-throughput some modifications should be made in sdkconfig in order to optimize wifi settings for better throughput. As I have understood, majority of these configurations cannot be changed for the Arduino IDE. If I am not mistaken, the only way is to change the configurations in esp-idf, recompile the libraries with updated settings, and afterwards replace libraries in Arduino with new ones. I will try and let you know.

[RafigRzayev]

I was able to increase the speed ten times by changing configurations in esp-idf and updating arduino libraries. I will post instructions soon.

[RafigRzayev]

Edit: In bottom of this page, there are updated alternative instructions which I believe are a better way.

Below you can find instructions, with which you can increase WiFi speed on Arduino. Instructions are for Windows OS.

1) Create a folder called ESP32 2) Inside ESP32 folder, create two subfolders: esp-idf, and tools 3) Download esp-idf-tools-setup from https://dl.espressif.com/dl/esp-idf-tools-setup-2.3.exe 4) Installation process via esp-idf-tools: 4.1) Install Python or choose an already installed version on your PC. I had problems when used my own Python version, it lead to pip version errors which I was not able to solve. Installing suggested 3.7 version instead of using existing Python version solved my issues.

4.2) Install Git or choose an already installed version

4.3) Select esp-idf version to download. I have used v3.3.4 (release version)

4.4) Select directory for tools - we will choose the folder that we created in step 2

4.5) Select additional options. In this step, selecting last option led to some warnings for me, though as far as I remember build was successful. I am unselecting it for any case.

4.6) Start the process and wait for the downoad.

5) Go to ESP32/esp-idf/examples/get-started directory. Copy and paste blink example in the same directory for backup purposes.

6) Go the the blink example. Inside it, create a folder called components.

7) Go the the components directory and run the following command via gitbash/cmd or any other tool that you use for git. This command will copy the contents of arduino-esp32 repository to arduino folder git clone https://github.com/espressif/arduino-esp32 arduino

8) Go to the arduino folder. Run the following command git submodule update --init --recursive

9) Close the console. Go to C:\ESP32\esp-idf\examples\get-started\blink\components\arduino\tools\sdk directory. Copy the sdkconfig file

10) Go to C:\ESP32\esp-idf\examples\get-started\blink. Paste the copied sdkconfig file here.

11) In the Desktop, you can find a shortcut to esp-idf command promt. Open it.

12) If everything has been ok until this step, you should see following start message inside the console.

13) Go to the blink example and invoke project configuration tool with following commands:

cd examples\get-started\blink
idf.py menuconfig

14) Following GUI will appear:

15) Go to Arduino Configuration. Deselect option Autostart Arduino setup and loop on boot (Press Y/N for select-unselect). It will enable us to compile without having to modify main source code.

16) Go to Compiler Options. Set Optimization Level to Release instead of Debug to increase performance.

After this point, all changes are made inside Component config.

17) Go to Wi-Fi. Change configuration so that they match with iperf example settings https://github.com/espressif/esp-idf/blob/178b122c145c19e94ac896197a3a4a9d379cd618/examples/wifi/iperf/sdkconfig.ci.99

• Change Max number of WiFi dynamic RX buffers from 32 to 64
• Change Max number of WiFi dynamic TX buffers from 32 to 64
• Change WiFi AMPDU TX BA window size from 6 to 32
• Change WiFi AMPDU RX BA window size from 16 to 32
• Enable WiFi IRAM speed optimization
• Enable WiFi RX IRAM speed optimization

18) Go to LWIP. Do following changes:

• Select Enable LWIP IRAM optimization
• Change TCPIP task receive mail box size from 32 to 64

19) Inside LWIP, go to TCP sub-section. Make following changes:

• Change Default send buffer size from 5744 to 65534
• Change Default receive window size from 5744 to 65534
• Change Default TCP receive mail box size from 6 to 64

20) Exit and choose to save the configuration

21) Inside esp-idf console, type following command: idf.py build Wait for build to complete.

Close esp-idf console

22) Find the esp32 directory used by your Arduino IDE. Mine is located at C:\Users\rafig\AppData\Local\Arduino15\packages. After locating the folder, go to following subdirectory: C:\Users\rafig\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4\tools\sdk

The lib folder here containts precompiled esp32 libraries.

We need to update some of these libraries with those that we generated in esp-idf in order to improve WiFi speed. Since we will change the libraries, you can take a backup of whole lib folder.

23) Go to C:\ESP32\esp-idf\examples\get-started\blink\build\esp-idf directory. This directory holds all of generated libraries in esp-idf.

Go to lwip directory. Copy the lwip.a file.

No go to Arduino's library directory, C:\Users\rafig\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4\tools\sdk\lib, paste the copied file.

Done. Now WiFi speed should be good.

Notes: 1) Some of library names in arduino differ from esp-idf. For example, libfr.a vs libfreertos.a and etc. Don't know the proper way to replace them. 2) Replacing libesp32.a led to error during my experiments. Probably some additional files such as partition table and etc. should be moved for it to work as well. 3) You could play with wifi configurations in esp-idf according to this guide https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-guides/wifi.html#how-to-improve-wi-fi-performance 4) There is a tool for Mac/Linux that automates this process, but when I tried on Linux, in the last step where I had to replace libraries, the script gave error that Arduino folder was not found. Link to lib builder: https://github.com/espressif/arduino-esp32/blob/master/docs/lib_builder.md 5) I am not expert in this topic, so I can't tell you if there will be any issues after replacing. 6) Important thing is versions compatibility between esp32-arduino and esp-idf. Deviations lead to many errors. 7) I am attaching the generated library so you can move it directly to your Arduino folder, and test that you have higher speed. liblwip.zip

EDIT: Even though download example provided above works great now, my own project stopped working properly. I will share future updates.

[Ben79543]

@RafigRzayev This is very interesting. I did try several times to do what you are doing now, for the same purpose (wifi speed), and I finally gave up. But you are already beyond where I stopped. So keep us posted, thanks.

[RafigRzayev]

Finally got everything working. Will post detailed instructions soon.

[RafigRzayev]

New Windows 10 instructions:

Here I have used alternative approach. Since there is an official lib-builder tool which is supported for Mac/Linux, we will download a tool called WSL (Windows subsystem for Linux), use the lib-builder inside this environment, and then copy the generated files to our Arduino directory.

1) Follow the manual installation instructions for WSL from the following link: https://docs.microsoft.com/en-us/windows/wsl/install-win10 . Before starting, you need to enable virtualization in your computer's BIOS. The screenshot below demonstrates how I did it in my PC.

I selected Ubuntu 18 in Step 6 - Install your Linux distribution of choice, after that there is no need to proceed further to the step Install Windows Terminal (optional)

If Ubuntu is not opening properly after installation, wait couple of minutes so the environment is ready and relaunch. If it doesn't help, try reinstalling Ubuntu.

2) Now, inside Ubuntu terminal, type following command to install the basic Linux prerequisities, it will take some time: sudo apt-get update && sudo apt-get upgrade -y

3) Now type following to download pre-requisities for lib-builder tool (instructions from https://github.com/espressif/esp32-arduino-lib-builder): sudo apt-get install git wget curl libssl-dev libncurses-dev flex bison gperf python python-pip python-setuptools python-serial python-click python-cryptography python-future python-pyparsing python-pyelftools cmake ninja-build ccache

when asked to continue type "y". It will also take some time.

4) Type sudo pip install --upgrade pip

5) After it ends, type:

git clone https://github.com/espressif/esp32-arduino-lib-builder
cd esp32-arduino-lib-builder

This should be result, you can check contents of folder with "ls" command

6) As I have understood, this tool by default will generate libraries for the latest esp32-arduino release. So before proceeding, we will upgrade our esp32 board in Arduino IDE to match the latest version. Go to https://github.com/espressif/arduino-esp32/releases

Hover over package_esp32_dev_index.json to see the download link. You can also use browser's inspect element tool to extract the link.

In my case the download url is https://github.com/espressif/arduino-esp32/releases/download/1.0.5-rc4/package_esp32_dev_index.json

7) Now go to Arduino-IDE->File->Preferences and paste the json download link to Additional Boards Manager URLs.

8) Afterwards, go to Arduino-IDE->Tools->Board->Board Manager Install the latest ESP32 version.

9) Now make sure that Ubuntu terminal is opened. Now in Windows File Explorer, type following to see Ubuntu contents via GUI: \\wsl$

Go to Ubuntu->home->your_user_id->esp32-arduino-lib-builder

10) Now edit the sdkconfig file as per your needs.

You can try following changes:

Inside WiFi: CONFIG_ESP32_WIFI_DYNAMIC_RX_BUFFER_NUM to 64 CONFIG_ESP32_WIFI_DYNAMIC_TX_BUFFER_NUM to 64 CONFIG_ESP32_WIFI_TX_BA_WIN to 32 CONFIG_ESP32_WIFI_RX_BA_WIN to 32 CONFIG_ESP32_WIFI_IRAM_OPT to y CONFIG_ESP32_WIFI_RX_IRAM_OPT to y

Inside LWIP: CONFIG_TCPIP_RECVMBOX_SIZE to 64 CONFIG_LWIP_IRAM_OPTIMIZATION to y

Inside TCP: CONFIG_TCP_SND_BUF_DEFAULT to 65534 CONFIG_TCP_WND_DEFAULT to 65534 CONFIG_TCP_RECVMBOX_SIZE to 64

NOTE: I tried switching compiler optimization to release instead of debug, and in this case the tool doesn't generate output. So better don't change following lines:

CONFIG_OPTIMIZATION_LEVEL_DEBUG=y CONFIG_OPTIMIZATION_LEVEL_RELEASE=

after you have finished with your changes type following in Ubuntu terminal and wait for build to finish. ./build.sh

First build will take a lot of time since it will download esp-idf toolchain. Afterwards builds will be fast.

After the build is completed, a new folder called "out" will emerge. Open it with file explorer, go to tools->sdk. Copy the lib folder.

Then paste this folder into your Arduino sdk folder, mine is at C:\Users\rafig\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4\tools\sdk As a result of this replacement, Arduino IDE will use new libraries. (Edit: Replace whole SDK folder instead of just lib folder, this solved issue when I tried this process with Arduino 1.0.5 RC6. https://github.com/espressif/esp32-arduino-lib-builder/issues/24)

If you run the download example above, it should work without problems. I reached a speed of 480kB/s compared to 50kB/s with raw Arduino libs.

Even though the demo example works ok, my own project stopped working, It was giving "Store Prohibitied" error in runtime. The reason is that all our changes to WiFi parameters increase RAM consumption. Since my application was also using a lot of RAM, there was no memory left and program crashed. Afterwards, I modified some of sdkconfig parameters in lib-builder to use less RAM, repeated the process again and as a result my project now works without any problem. I am getting 320kB/s now. My latest configs for your reference:

Inside WiFi: CONFIG_ESP32_WIFI_DYNAMIC_RX_BUFFER_NUM to 64 CONFIG_ESP32_WIFI_DYNAMIC_TX_BUFFER_NUM to 64 CONFIG_ESP32_WIFI_TX_BA_WIN to 32 CONFIG_ESP32_WIFI_RX_BA_WIN to 32 CONFIG_ESP32_WIFI_IRAM_OPT - leave default, no optimization CONFIG_ESP32_WIFI_RX_IRAM_OPT - leave default, no optimization

Inside LWIP: CONFIG_TCPIP_RECVMBOX_SIZE to 20 CONFIG_LWIP_IRAM_OPTIMIZATION - leave default, no optimization

Inside TCP: CONFIG_TCP_SND_BUF_DEFAULT to 25848 CONFIG_TCP_WND_DEFAULT to 25848 CONFIG_TCP_RECVMBOX_SIZE to 20

This was all I was able to learn so far. Hope you find it useful. And please share if you succeed to further increase the speed.

[RafigRzayev]

@HKGK @ScruffR @Ben79543 fyi

[pablofr918]

Hello @RafigRzayev. I've tried the second instructions, but when I replace the SDK folder in _C:\Users\myuser\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4\tools\sdk and run the code that I have for downloading a file, it gives me some errors due to the replacement of the folder. I don't know if you had any problems you solved.

[RafigRzayev]

@pablofr918 I will check it today on my computer again and let you know. Can you send a screenshot of your errors please?

[pablofr918]

@RafigRzayev So, I first tried to do this to make WiFi faster, but I couldn't. After that I'm doing something similar but to enable nat feature in esp32 arduino framework, but the lasts steps are the ones I think I'm doing wrong because I get some errors like this:

But this errors seem to disappear when I move those files to the right directory. Maybe I'm copying the folder/file I need to another directory or something. The steps I followed are:

1. Modify menuconfig to enable NAT as this repo indicates: https://github.com/paclema/esp32_lwip_nat_example
2. Then I make the build.sh
3. (this is the step I don't understand at all) I have to copy and paste the modified files to my sdk folder.

I have tried to replace the liblwip.a (which I don't know what is). But those errors ocurred. Then I tried to copy the \wsl$\Ubuntu-18.04\home\tecnico7\esp32-arduino-lib-builder\out\tools\sdk\esp32\include\lwip folder and paste it somehow into the arduino sdk folder, but it still happens. The thing is that I don't know how this folders/libraries are structed and work, so thats why I'm copying/pasting with no sense.

Any help will be welcome!! :)

[RafigRzayev]

@pablofr918 Hello again. I removed WSL and Arduino completely from my computer to test instructions from the scratch.

At the time of writing these instructions, the latest version of Arduino was 1.0.5 RC4, and lib-builder was updating according to it. At the moment the last version is 2.0, and lib-builder should be working according to it. Since you were using 1.0.5RC4, there was incompatibility between Arduino version of your IDE and lib builder.

So, in clean Arduino, I tried to install the latest ESP32-Arduino version 2.0, but was not able to install because of error. As a workaround, I needed to first install older version, and then upgrade to latest version.

Afterwards, I started editing sdkconfig, but some of old options didn't exist there, and other old options now caused errors in build step. Anyway I changed some sdkconfig parameters until it worked and built successfully.

Later replaced sdk folder and now also getting errors... Some of them were related to mbedtls option, but some were unrelated. Unfortunately I don't have much time to investigate it at the moment, maybe espressif staff can investigate.

I don't know if it ok for you, but as a temporary help I can offer following: I have saved the lib output folder that I have generated in the instructions for personal use. You can use Arduino 1.0.5RC4 and just replace the lib folder to have increased speed. Follow the below instructions:

Instructions for changing Arduino IDE to increase the download speed:

1) Go to File->Preferences->Additional Boards Manager URLs 2) Paste following link there https://github.com/espressif/arduino-esp32/releases/download/1.0.5-rc4/package_esp32_dev_index.json 3) Go to Tools->Board->Board Manager. 4) Search for ESP32, and install/upgrade the version to 1.0.5 - RC4 5) Download a compressed file from https://www.dropbox.com/s/qf8vsexax03zeg6/lib.rar?dl=0 6) Extract the file to your Desktop 7) Locate Arduino IDE's ESP directory. Open the tools->sdk folder. Example path - C:\Users\rafig\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.5-rc4\tools\sdk 8) Replace the lib folder in the directory with the downloaded one

[RafigRzayev]

Also, during discussion here https://github.com/espressif/arduino-esp32/issues/4833#issuecomment-796374311 there was one more suggestion for building on windows using docker. I didn't have a chance to test it yet, maybe you can try. https://hub.docker.com/r/lbernstone/esp32-arduino-lib-builder

[pablofr918]

Thank you so much! You helped me understand a LOT of things with this. I really appreciate it @RafigRzayev!

[skylli]

here is the reason why http stream so slow

[0x0fe]

here is the reason why http stream so slow

What did you use instead? DMA? @skylli

[fjmsouza]

Hello, Me too, trying improve the wifi transfer, but... I've been trying to use Arduino as a component of ESP-IDF for days, but the behavior is completely different from the one previously compiled in Arduino IDE, a struggle with the sdkconfig. Without success, I resorted to ESP32-arduino-lib-builder, followed their tutorial, but each attempt to adjust the compilation takes long time. Even the CONFIGs, currently, are no longer the same as the ones mentioned by @RafigRzayev.

Does anyone have a suggestion beyound to code in ESP-IDF?