Flash Encryption (FLASH_CRYPT_CNT)

[Buffalchill]

Hardware:

Board: ESP32 Dev Module Core Installation/update date: 11/jul/2017 IDE name: Arduino IDE 1.8.3. Flash Frequency: 40Mhz Upload Speed: 115200

Hi, I'm trying to enable Flash Encryption. The ESP IDF documentation says (http://esp-idf.readthedocs.io/en/latest/security/flash-encryption.html)

"Flash Encryption Initialisation The bootloader must be compiled with flash encryption support enabled. In make menuconfig, navigate to “Security Features” and select “Yes” for “Enable flash encryption on boot”.

how is this possible in Arduino IDE ?

[WebDust21]

One last comment on the problem: either you have to buy ESP32s pre-programmed with a security key (that you have to supply to the manufacturer), or you have to find some other solution.

You have to realize that if the manufacturer starts with a blank ESP32, and ends with a fully-programmed ESP32...there is no security or encryption that can stop them from programming a bazillion copies with which to rip off your product. Period. At best you can make it more difficult for them (i.e. the "downloader stub" that they FLASH, which then goes to a specific web address to download and program the actual firmware file.) They can still Internet-sleuth the ESP32's file request and get a copy of the actual firmware file--brownie points if you encrypt or hash it in some way that makes it unuseable. But if they are very crafty, it's not impossible to program a router to serve a local copy of the "firmware file" to the stub downloader...and once again you're right back at square 1. Like I said, if they can fully program the device, they can also rip it off.

[workpage2]

• и иначе) приведет к загрузке с ошибками, так как невозможно снова вызвать функции шифрования в загрузчике (из-за предохранителей FLASH_CRYPT_CNT, защищенных от записи).
• Обратите внимание, что безопасную загрузку невозможно реализовать (без особых трудностей) с Arduino. Однако это не проблема, если в

I encrypted esp32 according to your description. Everything works. However, I am unable to update my esp32 via the web, even though everything worked before. I am using esp32FOTA library. (use update.h) Pre-encoded the fw.bin file. error:

Fetching Bin: /fw.bin
Got 1177456 bytes from server
Got application/octet-stream payload.
contentLength : 1177456, isValidContentType : 1
Begin OTA. This may take 2 - 5 mins to complete. Things might be quite for a while.. 
Patience!
Written only : 0/1177456. Retry?
Error Occurred. Error #: 8

I created a topic https://github.com/chrisjoyce911/esp32FOTA/issues/40 Error 8, it looks like "UPDATE_ERROR_MAGIC_BYTE" I think in the update.h libraries, as well as in httpupdate.h, you need to disable all MD5 checks, magic bytes ... Is there someone who has found a solution for how to update esp over the internet? (arduino) Any thoughts.

[WebDust21]

Ahh, the delightful boundless wisdom of designers using the ONLY function that is limited to non-encrypted ESP32s. What in the world they were thinking, I do not know. The Arduino-ESP32 built-in "update.h" library is completely and totally at fault here. Yes, I have successfully gotten OTA firmware updates working with an encrypted ESP32. Had to dig the "update.h" (and "updater.c") files out of the Arduino-ESP32 core, and rewrite them to use the required functions. I know you'll say, "post them here"--but I am notoriously terrible at writing relocatable/library code. It will not work as a drop-in-replacement. Instead, I can provide the necessary pointers:

• All "ESP.flash####" functions are NOT compatible with ESP32 Flash encryption. Notice here the use of such incompatible function in the Updater.cpp (Update.h library): https://github.com/espressif/arduino-esp32/blob/f98fc7ee9f266a0a1d10ec711e05184688c6de30/libraries/Update/src/Updater.cpp#L204
• Instead, you will have to use the "esp_partition_write", "esp_partition_erase_range" and other associated functions that DO support encrypted ESP32 FLASH.

Might want to do some research online and try to find a library utilizing the proper commands. As the same commands work equally well on both encrypted and unencrypted ESP32s, you can debug the OTA update library (using the right commands) with an unencrypted ESP32--and once it is working perfectly, then you can encrypt the ESP32 and enjoy functional OTA.
Maybe it's just me, but I completely do not understand why the developers would go through the effort of making half-useless routines, when perfectly functional routines are just an arms-reach away?

[WebDust21]

Looking into it further, noticing a discrepancy between the files I'd copied and rewritten, and the Arduino-ESP32 core on GitHub. Seems they've fixed the egregious problem, BUT if your Arduino-ESP32 is still using the old files, it won't work.

Back when I copied it, the offending functions were "ESP.flashEraseSector", "ESP.flashWrite", and "ESP.flashRead". Which are clearly indicated as not compatible with encrypted FLASH...

bool EspClass::flashEraseSector(uint32_t sector)
{
    return spi_flash_erase_sector(sector) == ESP_OK;
}

// Warning: These functions do not work with encrypted flash
bool EspClass::flashWrite(uint32_t offset, uint32_t *data, size_t size)
{
    return spi_flash_write(offset, (uint32_t*) data, size) == ESP_OK;
}

bool EspClass::flashRead(uint32_t offset, uint32_t *data, size_t size)
{
    return spi_flash_read(offset, (uint32_t*) data, size) == ESP_OK;
}

But right below it in "ESP.c" are the functions that DO use the right commands for writing to encrypted FLASH:

bool EspClass::partitionEraseRange(const esp_partition_t *partition, uint32_t offset, size_t size) 
{
    return esp_partition_erase_range(partition, offset, size) == ESP_OK;
}

bool EspClass::partitionWrite(const esp_partition_t *partition, uint32_t offset, uint32_t *data, size_t size) 
{
    return esp_partition_write(partition, offset, data, size) == ESP_OK;
}

bool EspClass::partitionRead(const esp_partition_t *partition, uint32_t offset, uint32_t *data, size_t size) 
{
    return esp_partition_read(partition, offset, data, size) == ESP_OK;
}

Turns out that this was fixed on September 30th.

Not sure if you just need to update the Arduino-ESP32...or just yank the new "Updater.c" file from GitHub and shoehorn it into your project.

[workpage2]

Did I understand correctly that even with a new updater.cpp file, I have to download an unprotected file?

[WebDust21]

Yes. Unless you add an encryption level of your own.

[workpage2]

I have replaced updater.cpp and update.h. Sket has stopped compiling. I have the following errors:

C:\Users\wp\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4\libraries\Update\src\Updater.cpp:204:14: error: 'class EspClass' has no member named 'partitionWrite'
 if (!ESP.partitionWrite(_partition, _progress + skip, (uint32_t*)_buffer + skip/sizeof(uint32_t), _bufferLen - skip)) {

[workpage2]

I completely downloaded the new esp32 core and it worked.

[workpage2]

Изучив его дальше, я заметил несоответствие между файлами, которые я скопировал и переписал, и ядром Arduino-ESP32 на GitHub. Кажется, они устранили вопиющую проблему, НО если ваш Arduino-ESP32 все еще использует старые файлы, это не сработает.

Когда я его скопировал, вызывали нарушение функции «ESP.flashEraseSector», «ESP.flashWrite» и «ESP.flashRead». Которые явно указаны как несовместимые с зашифрованной FLASH ...

bool EspClass::flashEraseSector(uint32_t sector)
{
    return spi_flash_erase_sector(sector) == ESP_OK;
}

// Warning: These functions do not work with encrypted flash
bool EspClass::flashWrite(uint32_t offset, uint32_t *data, size_t size)
{
    return spi_flash_write(offset, (uint32_t*) data, size) == ESP_OK;
}

bool EspClass::flashRead(uint32_t offset, uint32_t *data, size_t size)
{
    return spi_flash_read(offset, (uint32_t*) data, size) == ESP_OK;
}

Но прямо под ним в "ESP.c" находятся функции, которые ДЕЙСТВИТЕЛЬНО используют правильные команды для записи в зашифрованную FLASH:

bool EspClass::partitionEraseRange(const esp_partition_t *partition, uint32_t offset, size_t size) 
{
    return esp_partition_erase_range(partition, offset, size) == ESP_OK;
}

bool EspClass::partitionWrite(const esp_partition_t *partition, uint32_t offset, uint32_t *data, size_t size) 
{
    return esp_partition_write(partition, offset, data, size) == ESP_OK;
}

bool EspClass::partitionRead(const esp_partition_t *partition, uint32_t offset, uint32_t *data, size_t size) 
{
    return esp_partition_read(partition, offset, data, size) == ESP_OK;
}

Оказывается, это было исправлено 30 сентября.

Не уверен, нужно ли вам просто обновить Arduino-ESP32 ... или просто вытащить новый файл Updater.c из GitHub и вставить его в свой проект.

Please tell me the standard HTTPUpdate example can write new firmware to esp32 encoded? It doesn't use updater.c ...

[WebDust21]

Please tell me the standard HTTPUpdate example can write new firmware to esp32 encoded? It doesn't use updater.c ...

Yes it does use update.c. Right here in HTTPUpdate.h. Looked good until I found references to the Update library. Thusly HTTPUpdate can only write to an encrypted ESP32 if the Update library supports encrypted FLASH. Which seems to be Arduino-ESP32 core dependent.

[Sergio-Navarro-M]

Final report: in the above bootloader file, I had not checked "Disable Plaintext Flashing" in the bootloader configuration, resulting in it not fully locking down the ESP32. Due to some other issues, I was unable to use my ESP-IDF installation to recompile the bootloader with the setting enabled...and I had to start over.

Attached to this post is a ZIP file with a bootloader compiled and tested with the following configuration:

• ESP-WROOM-32
• 4MB FLASH
• QIO, 80MHz
• FLASH encryption enabled on boot, Release mode (NOT Development)
• Bootloader output: error only (bootloader is too big if left on the default "Info" setting)

Assuming your Arduino ESP32 project fits the above requirements, use the bootloader as follows:

1. Find the bootloader files in your Arduino installation. In my case, on Ubuntu 18.04 LTS 64-bit, they can be found at "/home/USERNAME/.arduino15/packages/esp32/hardware/esp32/1.0.4/tools/sdk/bin/"
2. Rename the original "bootloader_qio_80m.bin" to something else ("ORIGINALbootloader_qio_80m.bin") so you can swap it back for regular debug testing
3. Extract the ZIP file, so "bootloader_qio_80m.bin" is in the folder.
4. Go to Arduino IDE, and upload your sketch like normal.
5. Cycle power to the ESP32, and wait. In my case, with a 1.4MB app, it takes about 25 seconds to fully encrypt and automatically start my program. If you connect over the serial monitor @ 115200 baud (standard), you should see something like the following output:

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT) configsip: 0, SPIWP:0xee clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00 mode:DIO, clock div:1 load:0x3fff0018,len:4 load:0x3fff001c,len:5232 ho 0 tail 12 room 4 load:0x40078000,len:12792 ho 0 tail 12 room 4 load:0x40080400,len:5960 entry 0x400806a4 {ESP32 IS ENCRYPTING WITH A SELF-GENERATED KEY} �[0;31mE (27445) esp_image: image at 0x1f0000 has invalid magic byte�[0m �[0;31mE (27445) boot_comm: mismatch chip ID, expected 0, found 18770�[0m ets Jun 8 2016 00:22:57 rst:0x3 (SW_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT) configsip: 0, SPIWP:0xee clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00 mode:DIO, clock div:1 load:0x3fff0018,len:4 load:0x3fff001c,len:5232 ho 0 tail 12 room 4 load:0x40078000,len:12792 ho 0 tail 12 room 4 load:0x40080400,len:5960 entry 0x400806a4 {YOUR APPLICATION IS RUNNING}

And last but not least, SWAP THE ORIGINAL ARDUINO BOOTLOADER BACK BEFORE YOU FORGET!

NOTES:

• If you accidentally interrupt power to the ESP32 while it is encrypting, simply re-upload your sketch, and try again. I did that by accident once (while programming and encrypting 18 ESP32s), and it worked fine.
• If you are in a semi-production environment, and have an SPIFFS upload to make, don't worry. Simply upload the sketch to the ESP32, then restart it in programming mode (hold EN, or however you access programming mode on your project)...don't let it enter "run" mode, or it will start to encrypt. Upload the SPIFFS data, then restart the ESP32 in "run" mode.
• You will not be able to upload another sketch to the ESP32 via Arduino IDE (or even ESP-IDF). The only way to update the ESP32 is via internal update, where your application receives the binary, and calls the appropriate ESP flash write functions to update it. Any attempt to upload code via RS-232 (Arduino or otherwise) will result in a bootloop with errors, as it is impossible to call the encryption functions in the bootloader again (due to the write-protected FLASH_CRYPT_CNT fuses.)
• Note that secure boot is not possible to implement (without a lot of difficulty) with Arduino. However, this is not a problem if you devise your own protection against updating with "non-official" firmwares. Basically, secure boot prevents "non-signed" binaries from running if they get OTA'd into the ESP. But if your OTA code simply rejects "non-official" binaries, you have a better end result than Secure Boot.

Enjoy ;-)

bootloader_qio_80m.zip

Thank you very much for sharing your efforts! You saved me weeks of work porting a 1500-line perfectly working application to ESP-IDF. I followed your descriptions to generate bootloader files and use them to upload them with the Arduino IDE. I am using the partition scheme "Huge APP (3MB no OTA / 1MB SPIFFS)". Arduino 1.0.6 and ESP-IDF v3.3.5.

I support the idea of providing a straightforward way to encrypt Arduino firmwares on the ESP32 too.

[WebDust21]

I completely do not understand the resistance for the ESP dev team to simply add the option for an encrypted bootloader for use in Arduino. The bootloaders are already pre-generated, and there are NO connections between the user application and the bootloader. (All functions that in an ESP-IDF context would access the bootloader, are nothing but empty stubs for the Arduino port.)

At any rate, those of us who want the functionality can access it with a minor hack. And if you're doing a batch production programming job by making a commandline programming command (bypassing the Arduino requirements for recompiling your project each time), it's relatively trivial to point the programming tool to an encrypted bootloader, without modifying any Arduino files. (Dumping the SPIFFS write onto the single command isn't difficult either.)

info from: https://www.youtube.com/watch?v=Lwpxrq9v1Yw https://forum.arduino.cc/t/programming-esp8266-without-compiling-every-time/472971/13

[halilozdgn]

Thank you for porting bootloader @WebDust21. I've done all steps you given but i get this error continuously:

18:35:31.031 -> rst:0x10 (RTCWDT_RTC_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT) 18:35:31.031 -> flash read err, 1000 18:35:31.031 -> ets_main.c 371

So i did some stuff wrong apparently but i dont know what is this error about. Probably my esp is broken. Gonna get new one but i don't want to get same error again. I'm using custom partition btw:

Name, Type, SubType, Offset, Size, Flags

nvs, data, nvs, 0x9000, 0x5000, otadata, data, ota, 0xe000, 0x2000, app0, app, ota_0, 0x10000, 0x180000, app1, app, ota_1, 0x190000,0x140000, spiffs, data, spiffs, 0x2D0000,0x130000,

But default partition not works either.

Is there any way something gone wrong in encryption proccess? After you done with flashing how do you recognize encryption is finished? I might unplug before encryption done because i can't see anything in monitor after flash.

espefuse.py summary output:

EFUSE_NAME (Block) Description = [Meaningful Value] [Readable/Writeable] (Hex Value)

Calibration fuses: BLK3_PART_RESERVE (BLOCK0): BLOCK3 partially served for ADC calibration data = False R/W (0b0) ADC_VREF (BLOCK0): Voltage reference calibration = 1107 R/W (0b00001)

Config fuses: XPD_SDIO_FORCE (BLOCK0): Ignore MTDI pin (GPIO12) for VDD_SDIO on reset = False R/W (0b0) XPD_SDIO_REG (BLOCK0): If XPD_SDIO_FORCE, enable VDD_SDIO reg on reset = False R/W (0b0) XPD_SDIO_TIEH (BLOCK0): If XPD_SDIO_FORCE & XPD_SDIO_REG = 1.8V R/W (0b0) CLK8M_FREQ (BLOCK0): 8MHz clock freq override = 54 R/W (0x36) SPI_PAD_CONFIG_CLK (BLOCK0): Override SD_CLK pad (GPIO6/SPICLK) = 0 R/W (0b00000) SPI_PAD_CONFIG_Q (BLOCK0): Override SD_DATA_0 pad (GPIO7/SPIQ) = 0 R/W (0b00000) SPI_PAD_CONFIG_D (BLOCK0): Override SD_DATA_1 pad (GPIO8/SPID) = 0 R/W (0b00000) SPI_PAD_CONFIG_HD (BLOCK0): Override SD_DATA_2 pad (GPIO9/SPIHD) = 0 R/W (0b00000) SPI_PAD_CONFIG_CS0 (BLOCK0): Override SD_CMD pad (GPIO11/SPICS0) = 0 R/W (0b00000) DISABLE_SDIO_HOST (BLOCK0): Disable SDIO host = False R/W (0b0)

Efuse fuses: WR_DIS (BLOCK0): Efuse write disable mask = 132 R/W (0x0084) RD_DIS (BLOCK0): Efuse read disable mask = 1 R/W (0x1) CODING_SCHEME (BLOCK0): Efuse variable block length scheme = NONE (BLK1-3 len=256 bits) R/W (0b00) KEY_STATUS (BLOCK0): Usage of efuse block 3 (reserved) = False R/W (0b0)

Identity fuses: MAC (BLOCK0): Factory MAC Address = ac:xx:xx:xx:xx:e4 (CRC 0xcd OK) R/W MAC_CRC (BLOCK0): CRC8 for factory MAC address = 205 R/W (0xcd) CHIP_VER_REV1 (BLOCK0): Silicon Revision 1 = True R/W (0b1) CHIP_VER_REV2 (BLOCK0): Silicon Revision 2 = False R/W (0b0) CHIP_VERSION (BLOCK0): Reserved for future chip versions = 2 R/W (0b10) CHIP_PACKAGE (BLOCK0): Chip package identifier = 0 R/W (0b000) MAC_VERSION (BLOCK3): Version of the MAC field = 0 R/W (0x00)

Security fuses: FLASH_CRYPT_CNT (BLOCK0): Flash encryption mode counter = 1 R/- (0b0000001) UART_DOWNLOAD_DIS (BLOCK0): Disable UART download mode (ESP32 rev3 only) = False R/- (0b0) FLASH_CRYPT_CONFIG (BLOCK0): Flash encryption config (key tweak bits) = 15 R/W (0xf) CONSOLE_DEBUG_DISABLE (BLOCK0): Disable ROM BASIC interpreter fallback = True R/W (0b1) ABS_DONE_0 (BLOCK0): Secure boot V1 is enabled for bootloader image = False R/W (0b0) ABS_DONE_1 (BLOCK0): Secure boot V2 is enabled for bootloader image = False R/W (0b0) JTAG_DISABLE (BLOCK0): Disable JTAG = True R/W (0b1) DISABLE_DL_ENCRYPT (BLOCK0): Disable flash encryption in UART bootloader = True R/W (0b1) DISABLE_DL_DECRYPT (BLOCK0): Disable flash decryption in UART bootloader = True R/W (0b1) DISABLE_DL_CACHE (BLOCK0): Disable flash cache in UART bootloader = True R/W (0b1) BLOCK1 (BLOCK1): Flash encryption key = ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? -/- BLOCK2 (BLOCK2): Secure boot key = 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 R/W BLOCK3 (BLOCK3): Variable Block 3 = 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 R/W

Flash voltage (VDD_SDIO) determined by GPIO12 on reset (High for 1.8V, Low/NC for 3.3V).

[WebDust21]

"After you done with flashing how do you recognize encryption is finished? I might unplug before encryption done because i can't see anything in monitor after flash."

As I mentioned above, after being programmed with an encrypted bootloader, the ESP32 will provide a power-up bootloader string...and then be silent for easily 30 seconds (longer if the memory space is bigger). During this time, it is encrypting. After it's FINISHED encrypting, it will automatically restart, and run your program.

If you interrupt power before it FINISHES encrypting, the ESP32 will bootloop with the "err:1000". This is not fatal; as I mentioned above, simply reflash it from Arduino, restart the ESP, and this time wait longer.

If the ESP32 is already fully encrypted, any attempt to reflash via Arduino IDE (or ESP-IDF) will result in an unusable bootloop ("err:1000"). Noticing the FLASH_CRYPT_CNT being an odd number seems to indicate it's finished...not sure there.

And remember, you WILL NOT be able to reflash an ESP32 that's been encrypted via serial. ANY update must be via internal update with the appropriate memory handling commands.

I've encrypted ESP32 chips by accident if I had the wrong bootloader file in the Arduino folder. Only ran into troubles when I didn't realize it and tried to reflash the ESP32 which had already been encrypted.

[WebDust21]

EDIT: Will note that after flashing the ESP32, you'll need to reset it (and not hold EN, so it can boot normally)...and then wait. Wait until your application starts. No status is provided during the encryption, but it will provide a bootloader string after the encryption completes.

[halilozdgn]

"After you done with flashing how do you recognize encryption is finished? I might unplug before encryption done because i can't see anything in monitor after flash."

As I mentioned above, after being programmed with an encrypted bootloader, the ESP32 will provide a power-up bootloader string...and then be silent for easily 30 seconds (longer if the memory space is bigger). During this time, it is encrypting. After it's FINISHED encrypting, it will automatically restart, and run your program.

If you interrupt power before it FINISHES encrypting, the ESP32 will bootloop with the "err:1000". This is not fatal; as I mentioned above, simply reflash it from Arduino, restart the ESP, and this time wait longer.

If the ESP32 is already fully encrypted, any attempt to reflash via Arduino IDE (or ESP-IDF) will result in an unusable bootloop ("err:1000"). Noticing the FLASH_CRYPT_CNT being an odd number seems to indicate it's finished...not sure there.

And remember, you WILL NOT be able to reflash an ESP32 that's been encrypted via serial. ANY update must be via internal update with the appropriate memory handling commands.

I've encrypted ESP32 chips by accident if I had the wrong bootloader file in the Arduino folder. Only ran into troubles when I didn't realize it and tried to reflash the ESP32 which had already been encrypted.

So i should use "--after hard_reset" not "--after no_reset" right?

EDIT: "--after hard_reset" works fine. Thanks.

[halilozdgn]

Hey there @WebDust21 after a long time. I was triying to encrypt esp32 again but i had some issues. I though it's about esp idf's version. I'm triying encryption with esp-idf v3.0 but im going to update ASAP. My question is how did you ported arduino's bootloader. Can you give me details so i can port for new version and share? Or if you have ported new versions bootloader can you share it? I'm waiting for your response. Have a nice day.

[WebDust21]

I did not do any "porting" of sorts. See, the Arduino ESP32 implementation basically replaces all "bootloader function calls" with stubs that either return nothing, or try to ascertain the data some other way. (For example, you cannot read the bootloader version number in the Arduino ESP32 system.) In short, there is no connection between the bootloader and the application code (unlike the ESP-IDF environment, where if you don't pull all your hair out the first day or 2, you can do function calls to the bootloader itself). This makes it very easy to just throw in a different bootloader--nothing ever is called in it, so if it boots your sketch, the bootloader's job is DONE.

It's been a little bit, but AFAIK all I did was open a basic project example on ESP-IDF, adjust the bootloader settings (Encryption enabled, etc.), compile the project--and then dig into the project output folders looking for the bootloader file. Copied it from there and "replaced" the appropriate Arduino bootloader file with the encrypted one. Your biggest challenge will probably be turning off enough settings in the bootloader so it'll actually fit in the partition... My biggest challenge was getting absolutely frustrated bonkers with the ESP-IDF's GIT version requirements that'd work one day and be completely useless the next. If you can survive that, you should be able to make your own bootloader.

Nowadays what I do is copy the Arduino "programming command", modify the bootloader one to point to the encrypted bootloader, and run the programming sequence from the command prompt. Much easier for "assembly line" style programming (I don't lose time recompiling the project 50 times)--AND I don't risk bricking ESP32's by accident when I forget to change the bootloader file back to the stock one.

[workpage2]

Final report: in the above bootloader file, I had not checked "Disable Plaintext Flashing" in the bootloader configuration, resulting in it not fully locking down the ESP32. Due to some other issues, I was unable to use my ESP-IDF installation to recompile the bootloader with the setting enabled...and I had to start over.

Attached to this post is a ZIP file with a bootloader compiled and tested with the following configuration:

• ESP-WROOM-32
• 4MB FLASH
• QIO, 80MHz
• FLASH encryption enabled on boot, Release mode (NOT Development)
• Bootloader output: error only (bootloader is too big if left on the default "Info" setting)

Assuming your Arduino ESP32 project fits the above requirements, use the bootloader as follows:

1. Find the bootloader files in your Arduino installation. In my case, on Ubuntu 18.04 LTS 64-bit, they can be found at "/home/USERNAME/.arduino15/packages/esp32/hardware/esp32/1.0.4/tools/sdk/bin/"
2. Rename the original "bootloader_qio_80m.bin" to something else ("ORIGINALbootloader_qio_80m.bin") so you can swap it back for regular debug testing
3. Extract the ZIP file, so "bootloader_qio_80m.bin" is in the folder.
4. Go to Arduino IDE, and upload your sketch like normal.
5. Cycle power to the ESP32, and wait. In my case, with a 1.4MB app, it takes about 25 seconds to fully encrypt and automatically start my program. If you connect over the serial monitor @ 115200 baud (standard), you should see something like the following output:

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT) configsip: 0, SPIWP:0xee clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00 mode:DIO, clock div:1 load:0x3fff0018,len:4 load:0x3fff001c,len:5232 ho 0 tail 12 room 4 load:0x40078000,len:12792 ho 0 tail 12 room 4 load:0x40080400,len:5960 entry 0x400806a4 {ESP32 IS ENCRYPTING WITH A SELF-GENERATED KEY} �[0;31mE (27445) esp_image: image at 0x1f0000 has invalid magic byte�[0m �[0;31mE (27445) boot_comm: mismatch chip ID, expected 0, found 18770�[0m ets Jun 8 2016 00:22:57 rst:0x3 (SW_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT) configsip: 0, SPIWP:0xee clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00 mode:DIO, clock div:1 load:0x3fff0018,len:4 load:0x3fff001c,len:5232 ho 0 tail 12 room 4 load:0x40078000,len:12792 ho 0 tail 12 room 4 load:0x40080400,len:5960 entry 0x400806a4 {YOUR APPLICATION IS RUNNING}

And last but not least, SWAP THE ORIGINAL ARDUINO BOOTLOADER BACK BEFORE YOU FORGET!

NOTES:

• If you accidentally interrupt power to the ESP32 while it is encrypting, simply re-upload your sketch, and try again. I did that by accident once (while programming and encrypting 18 ESP32s), and it worked fine.
• If you are in a semi-production environment, and have an SPIFFS upload to make, don't worry. Simply upload the sketch to the ESP32, then restart it in programming mode (hold EN, or however you access programming mode on your project)...don't let it enter "run" mode, or it will start to encrypt. Upload the SPIFFS data, then restart the ESP32 in "run" mode.
• You will not be able to upload another sketch to the ESP32 via Arduino IDE (or even ESP-IDF). The only way to update the ESP32 is via internal update, where your application receives the binary, and calls the appropriate ESP flash write functions to update it. Any attempt to upload code via RS-232 (Arduino or otherwise) will result in a bootloop with errors, as it is impossible to call the encryption functions in the bootloader again (due to the write-protected FLASH_CRYPT_CNT fuses.)
• Note that secure boot is not possible to implement (without a lot of difficulty) with Arduino. However, this is not a problem if you devise your own protection against updating with "non-official" firmwares. Basically, secure boot prevents "non-signed" binaries from running if they get OTA'd into the ESP. But if your OTA code simply rejects "non-official" binaries, you have a better end result than Secure Boot.

Enjoy ;-)

bootloader_qio_80m.zip

Please make a file for this parameter chip: ESP-WROOM-32 16MB FLASH QIO, 80MHz CPU freq. 240MHz 3mb APP/9mb FATFS

Thanks

[WebDust21]

Please make a file for this parameter chip: ESP-WROOM-32 16MB FLASH QIO, 80MHz CPU freq. 240MHz 3mb APP/9mb FATFS

Thanks

No difference. Notice in my quoted post that I said it was TESTED on a particular config. As the exact same Arduino bootloader is used irrespective of partition table or FLASH size, the "encrypted bootloader" will also work in the same situations. The only time a different bootloader would be necessary is if the FLASH communication (QIO/DIO/QOUT/DOUT) or FLASH speed (40M/80M) need to be different.

And as always...remember that after an ESP32 is encrypted with this bootloader, you will NOT be able to reprogram it via anything except OTA updates utilizing the encryption-safe FLASH write functions.

[torntrousers]

This is a really interesting ticket, thanks for sharing all you have done.

On this comment:

One last comment on the problem: either you have to buy ESP32s pre-programmed with a security key (that you have to supply to the manufacturer), or you have to find some other solution.

You have to realize that if the manufacturer starts with a blank ESP32, and ends with a fully-programmed ESP32...there is no security or encryption that can stop them from programming a bazillion copies with which to rip off your product. Period. At best you can make it more difficult for them (i.e. the "downloader stub" that they FLASH, which then goes to a specific web address to download and program the actual firmware file.) They can still Internet-sleuth the ESP32's file request and get a copy of the actual firmware file--brownie points if you encrypt or hash it in some way that makes it unuseable. But if they are very crafty, it's not impossible to program a router to serve a local copy of the "firmware file" to the stub downloader...and once again you're right back at square 1. Like I said, if they can fully program the device, they can also rip it off.

what are your thoughts on using a secure element like the ATECC608 for the private keys in conjunction with this flash encryption that you've got to work?

[WebDust21]

@torntrousers That's entirely beyond the scope of this issue. Note that the stock ESP32 bootloader does not provide any ability for you to specify an encryption key--it's randomly generated and burned to OTP memory. The stock bootloader also does not allow you to specify how the encryption is done--not to mention that whatever encryption is provided has to be decryptable by the hardware-level FLASH access engine (if I understand the ESP32 silicon correctly.) I haven't been able to turn up specifics on that chip, but at the very least you'd need to write a custom ESP32 bootloader--if not also having to design your own ESP32 module.

And at the end of the day, my original comment stands the same: if an overseas factory can fully program your chip in a production-level environment, you have no security...no matter how fancy the parts you use or elaborate the code you write.

[torntrousers]

Thats not exactly what I was suggesting, it was more about addressing the comment about a manufacturer making a bazillion copies with which to rip off your product - the flash encryption keys would still be the OTP ones but you could use a secure element for the device identity which could then help prevent cloning a device.

Anyway, thanks for sharing your work here, really interesting.

[workpage2]

A new esp32-WROOM-UE chip has been released. Judging by the documentation, the encryption algorithm has changed. Can I still use this file for encryption? https://www.espressif.com/en/news/ESP32_FIA_Analysis

[lbernstone]

This question will be more appropriate at https://esp32.com, since it has nothing to do with arduino.

[WebDust21]

@workpage2 If you can program the WROOM-UE chip via Arduino IDE, and Arduino is using the original (i.e. not "S2" or "S3") bootloaders, then in THEORY, yes, the above file I provided should still work.

I'm getting into the ESP32-S3 chips here shortly, and the Arduino port utilizes a different bootloader file for those chips--which means that I would need to generate a new "encryption" bootloader with ESP-IDF for Arduino port projects.

[simogaspa84]

Ahh, the delightful boundless wisdom of designers using the ONLY function that is limited to non-encrypted ESP32s. What in the world they were thinking, I do not know. The Arduino-ESP32 built-in "update.h" library is completely and totally at fault here. Yes, I have successfully gotten OTA firmware updates working with an encrypted ESP32. Had to dig the "update.h" (and "updater.c") files out of the Arduino-ESP32 core, and rewrite them to use the required functions. I know you'll say, "post them here"--but I am notoriously terrible at writing relocatable/library code. It will not work as a drop-in-replacement. Instead, I can provide the necessary pointers:

• All "ESP.flash####" functions are NOT compatible with ESP32 Flash encryption. Notice here the use of such incompatible function in the Updater.cpp (Update.h library): https://github.com/espressif/arduino-esp32/blob/f98fc7ee9f266a0a1d10ec711e05184688c6de30/libraries/Update/src/Updater.cpp#L204
• Instead, you will have to use the "esp_partition_write", "esp_partition_erase_range" and other associated functions that DO support encrypted ESP32 FLASH.

Might want to do some research online and try to find a library utilizing the proper commands. As the same commands work equally well on both encrypted and unencrypted ESP32s, you can debug the OTA update library (using the right commands) with an unencrypted ESP32--and once it is working perfectly, then you can encrypt the ESP32 and enjoy functional OTA. Maybe it's just me, but I completely do not understand why the developers would go through the effort of making half-useless routines, when perfectly functional routines are just an arms-reach away?

Hi @WebDust21 .. Thanks for the amazing job you did with encryption..

just 2 questions-..

1- I have lost the link where you explain how to create a custom new bootloader if our esp32 is a bit different from the one you use.... my case is HARDWARE: ESP32 240MHz, 320KB RAM, 4MB Flash can you provide again the link ? 2- Regarding the issue of using the esp32fota lib ..
https://github.com/chrisjoyce911/esp32FOTA

is it true that it should work if you modify the the files update.cpp using the supported esp32 functions ?

Thanks a lot

[simogaspa84]

• и иначе) приведет к загрузке с ошибками, так как невозможно снова вызвать функции шифрования в загрузчике (из-за предохранителей FLASH_CRYPT_CNT, защищенных от записи).
• Обратите внимание, что безопасную загрузку невозможно реализовать (без особых трудностей) с Arduino. Однако это не проблема, если в

I encrypted esp32 according to your description. Everything works. However, I am unable to update my esp32 via the web, even though everything worked before. I am using esp32FOTA library. (use update.h) Pre-encoded the fw.bin file. error:

Fetching Bin: /fw.bin
Got 1177456 bytes from server
Got application/octet-stream payload.
contentLength : 1177456, isValidContentType : 1
Begin OTA. This may take 2 - 5 mins to complete. Things might be quite for a while.. 
Patience!
Written only : 0/1177456. Retry?
Error Occurred. Error #: 8

I created a topic chrisjoyce911/esp32FOTA#40 Error 8, it looks like "UPDATE_ERROR_MAGIC_BYTE" I think in the update.h libraries, as well as in httpupdate.h, you need to disable all MD5 checks, magic bytes ... Is there someone who has found a solution for how to update esp over the internet? (arduino) Any thoughts.

Hi @workpage2

Were you able to modify the esp32fota lib as suggested and make working the update ?

Thanks a lot for the info ..

Thanks

[workpage2]

Were you able to modify the esp32fota lib as suggested and make working the update ?

To update an encrypted esp32 I use the following: #include <HTTPUpdate.h>

const char* ca ="-----BEGIN CERTIFICATE-----\n" \ Put your certificate here. "-----END CERTIFICATE-----";

WiFiClientSecure Sclient; Sclient.setCACert(ca); Serial.println("Update SPIFFS..."); t_httpUpdate_return ret = httpUpdate.updateSpiffs(Sclient,"https://LOGIN:PASSWORD@youhost.ru/spiffs.bin"); Serial.println("Update firmware. Please wait..."); ret = httpUpdate.update(Sclient, "https://LOGIN:PASSWORD@youhost.ru/fw.bin"); switch (ret) { case HTTP_UPDATE_FAILED: Serial.printf("HTTP_UPDATE_FAILED Error (%d): %s\n", httpUpdate.getLastError(), httpUpdate.getLastErrorString().c_str()); break; case HTTP_UPDATE_NO_UPDATES: Serial.println("HTTP_UPDATE_NO_UPDATES"); break; case HTTP_UPDATE_OK: Serial.println("HTTP_UPDATE_OK"); break; }

[workpage2]

Pre-encoded the fw.bin

Yes. I also fell into this trap. The file does not need to be pre-encoded!!!

[simogaspa84]

Hi @workpage2 ... sorry i didn't understand.. so with the code you posted you were able to update the fw of esp32 but with the flash encrypted as described (replacing the bootloader)... But the code you posted is it still using the esp32fota lib or not ? Thanks

[workpage2]

I don't use esp32fota. HTTPUpdate.h only.

[workpage2]

The fw.bin update file must be unencrypted! esp32 will re-encrypt it!

[simogaspa84]

ok clear @workpage2... but it is a pity not using esp32fota lib... i was using it also for uploading a signed fw.. .basically the lib lets you update the esp32 with a signed fw.. it is a pity because you miss this useful thing... any ideas for that?

[workpage2]

I am updating the firmware via SSL. The site is protected by login/password. It will not work to change the site, because the ssl certificate is being checked.

[simogaspa84]

yes... but the point is that the bin you are uploading is not validated from the esp32.. you should sign it.. the esp32fota lib was very good for that..

https://github.com/chrisjoyce911/esp32FOTA/blob/master/examples/HTTP/HTTP_signature_check/HTTP_signature_check.ino

[workpage2]

Try to update to use UNENCRYPTED file. Maybe esp32fota will work. I also want to say that it is impossible to download someone else's or a replacement file from a secure site for updating.

[simogaspa84]

Hi @workpage2 ... they have already tried but it seems that it is not easy

I will try to update fw with flash encryption using the code you posted and then maybe adding some signing.. or try to modify the esp32fota lib

[workpage2]

Yes. I remembered. A couple of years ago I used esp32fota, but I had to abandon this idea, because it is impossible to update the encrypted firmware.

[simogaspa84]

@workpage2 because it is impossible to update the fw when the flash is encrypted you mean?

[workpage2]

because it is impossible to update the fw when the flash is encrypted you mean?

Yes

[simogaspa84]

@workpage2 so you have already tried to do this without success ?

[workpage2]

Yes. I will try to find correspondence on this issue.

[workpage2]

https://github.com/chrisjoyce911/esp32FOTA/issues/40#issuecomment-719958103

[simogaspa84]

clear @workpage2 ... thanks for sharing... so i will follow your approach and maybe i will add something for signing the file to be updated..

[simogaspa84]

@workpage2 maybe you can think too a simply way for validating the update before to flash it in the esp32..

[workpage2]

What is the purpose of the update check? File upload error? Or authentication? httpupdate.h will not update the firmware if there are errors while uploading the file.

[workpage2]

Also, before downloading the file, you can read it byte by byte and calculate the checksum. I am using 3 files. 1 - fw.bin 2 - ver.txt 3 - checksum.txt checksum.txt contains checksum information. If the checksum does not match, the update does not occur.

[simogaspa84]

ok do you upload the checksum.txt before to ship the iot board i guess

[simogaspa84]

What is the purpose of the update check? File upload error? Or authentication? httpupdate.h will not update the firmware if there are errors while uploading the file.

integrity of the update.. if i am able to enter in your cloud platform .. I can upload what i want and it will always work.. instead if you sign the fw it is impossible