httpUpdateSigned don't work if used gzipped firmware

krc-soft commented 4 years ago

Basic Infos

[ x] This issue complies with the issue POLICY doc.
[ x] I have read the documentation at readthedocs and the issue is not addressed there.
[ ] I have tested that the issue is present in current master branch (aka latest git).
[ x] I have searched the issue tracker for a similar issue.
[ ] If there is a stack dump, I have decoded it.
[ x] I have filled out all fields below.

Platform

Hardware: [ESP-12]
Core Version: [2.7.4]
Development Env: [Arduino IDE]
Operating System: [Windows]

Settings in IDE

Module: [Nodemcu]
Flash Mode: [dio]
Flash Size: [4MB (FS:1MB OTA:~1019KB)]
lwip Variant: [v2 Lower Memory]
Reset Method: [ck|nodemcu]
Flash Frequency: [40Mhz]
CPU Frequency: [80Mhz]
Upload Using: [OTA]

Problem Description

Signed http update don't work correctly if using gzipped firmware. During update no error appears and the signature check worked. But the esp boots to the old firmware and when i reset the esp it hangs and cannot boot anymore.

I used the httpUpdateSigned example build it with Arduino IDE v1.8.11. After compiling i gzipped the firmware.bin and signed it with the signing.py script according to the documentation.

An unsigned gzipped firmware works and signed unzipped firmware works also.

MCVE Sketch


/*
   httpUpdateSigned.ino - Earle F. Philhower, III
   Released into the Public Domain

   For use while building under Linux or Mac.

   Automatic code signing is not supported on Windows, so this example
   DOES NOT WORK UNDER WINDOWS.

   Shows how to use a public key extracted from your private certificate to
   only allow updates that you have signed to be applied over HTTP.  Remote
   updates will require your private key to sign them, but of course
   **ANYONE WITH PHYSICAL ACCESS CAN UPDATE THE 8266 VIA THE SERIAL PORT**.
*/

#include <Arduino.h>

#include <ESP8266WiFi.h>
#include <ESP8266WiFiMulti.h>

#include <ESP8266HTTPClient.h>
#include <ESP8266httpUpdate.h>

#ifndef STASSID
#define STASSID "ssid"
#define STAPSK  "pass"
#endif

ESP8266WiFiMulti WiFiMulti;

#define MANUAL_SIGNING 1

// This example is now configured to use the automated signing support
// present in the Arduino IDE by having a "private.key" and "public.key"
// in the sketch folder.  You can also programmatically enable signing
// using the method shown here.

// This key is taken from the server public certificate in BearSSL examples
// You should make your own private/public key pair and guard the private
// key (never upload it to the 8266).
const char pubkey[] PROGMEM = R"EOF(
-----BEGIN PUBLIC KEY-----
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAu1Pt7yEk/xI+6cozLj5B
u4xV8gXDXcHS0rSJFfl4wBTk4UXpaJRaLfR1k0juEEa5LBRZaoA0iLj2e6kfCibO
Nx0VVoWmeqN2HBc3zkA1eqCksI0QUudzto4KhKHp0odiZ2zo6c/2Tn1zqD/m3OLo
SjVTbsJmGuwx8RGMBXozpg/uL0hHflihX+HND4Xfw92QXv7SaPBhgvM9xyRxn0/w
3J2nNjtuPuVN5vcQkd8ncMexVfy9AWp+HSA5AT5N8CJ/EeIsdDMY1US28bUePzj1
WIo75bZHKZNFw/iXe2xoPpm74qriMNSlW2craFP2K3KYnI28vJeUU6t9I6LS9zt2
zQIDAQAB
-----END PUBLIC KEY-----
)EOF";
#if MANUAL_SIGNING
BearSSL::PublicKey *signPubKey = nullptr;
BearSSL::HashSHA256 *hash;
BearSSL::SigningVerifier *sign;
#endif

void setup() {

  Serial.begin(115200);
  // Serial.setDebugOutput(true);

  Serial.println();
  Serial.println();
  Serial.println();

  for (uint8_t t = 4; t > 0; t--) {
    Serial.printf("[SETUP] WAIT %d...\n", t);
    Serial.flush();
    delay(1000);
  }

  WiFi.mode(WIFI_STA);
  WiFiMulti.addAP(STASSID, STAPSK);

  #if MANUAL_SIGNING
  signPubKey = new BearSSL::PublicKey(pubkey);
  hash = new BearSSL::HashSHA256();
  sign = new BearSSL::SigningVerifier(signPubKey);
  #endif
}

void loop() {
  // wait for WiFi connection
  if ((WiFiMulti.run() == WL_CONNECTED)) {

    WiFiClient client;

    #if MANUAL_SIGNING
    // Ensure all updates are signed appropriately.  W/o this call, all will be accepted.
    Update.installSignature(hash, sign);
    #endif
    // If the key files are present in the build directory, signing will be
    // enabled using them automatically

    ESPhttpUpdate.setLedPin(LED_BUILTIN, LOW);

    t_httpUpdate_return ret = ESPhttpUpdate.update(client, "http://192.168.0.2/firmware.bin");

    switch (ret) {
      case HTTP_UPDATE_FAILED:
        Serial.printf("HTTP_UPDATE_FAILED Error (%d): %s\n", ESPhttpUpdate.getLastError(), ESPhttpUpdate.getLastErrorString().c_str());
        break;

      case HTTP_UPDATE_NO_UPDATES:
        Serial.println("HTTP_UPDATE_NO_UPDATES");
        break;

      case HTTP_UPDATE_OK:
        Serial.println("HTTP_UPDATE_OK");
        break;
    }
  }
  delay(10000);
}

Debug Messages

[SETUP] WAIT 4... [SETUP] WAIT 3... [SETUP] WAIT 2... [SETUP] WAIT 1...

ets Jan 8 2013,rst cause:2, boot mode:(3,6)

load 0x4010f000, len 3584, room 16 tail 0 chksum 0xb0 csum 0xb0 v2843a5ac @cp:0 ld

[SETUP] WAIT 4... [SETUP] WAIT 3... [SETUP] WAIT 2... [SETUP] WAIT 1...

earlephilhower commented 4 years ago

Those debug messages don't really give much info. Can you bump up to full debug and re-run?

Sign checks are done in Updater.cpp before a reboot, and it doesn't care/know if the blob is signed or raw bin. It just looks for the proper magic bytes at the end and a signed hash of the bits uploaded (i.e. you should first gzip, then sign the update...the other way won't work). The exact steps you used to make the signed binaries and a demo public/private keypar would be helpful, too.

krc-soft commented 4 years ago

I use the public/private keypar included in the httpUpdateSigned example.

I build it with Arduino IDE v1.8.11 on Windows and use Debian on Windows (WSL) and call

gzip -9 firmware.bin
signing.py --mode sign --privatekey private.key --bin firmware.bin.gz --out firmware.bin.gz.signed
after that i rename firmware.bin.gz.signed to firmware.bin and upload it to my webserver

Debug Messages

SDK:2.2.2-dev(38a443e)/Core:2.7.3-3-g2843a5ac=20703003/lwIP:STABLE-2_1_2_RELEASE/glue:1.2-30-g92add50/BearSSL:5c771be

[SETUP] WAIT 4...
scandone
state: 0 -> 2 (b0)
state: 2 -> 3 (0)
state: 3 -> 5 (10)
add 0
aid 1
cnt

connected with test, channel 1
dhcp client start...
ip:192.168.0.114,mask:255.255.255.0,gw:192.168.0.1
[SETUP] WAIT 3...
[SETUP] WAIT 2...
[SETUP] WAIT 1...
[httpUpdate] Header read fin.
[httpUpdate] Server header:
[httpUpdate]  - code: 200
[httpUpdate]  - len: 304594
[httpUpdate] ESP8266 info:
[httpUpdate]  - free Space: 2801664
[httpUpdate]  - current Sketch Size: 343232
[httpUpdate] runUpdate flash...
sleep disable
[begin] roundedSize:       0x0004B000 (307200)
[begin] updateEndAddress:  0x00300000 (3145728)
[begin] currentSketchSize: 0x00054000 (344064)
[begin] _startAddress:     0x002B5000 (2838528)
[begin] _currentAddress:   0x002B5000 (2838528)
[begin] _size:             0x0004A5D2 (304594)
pm open,type:0 0
[Updater] sigLen: 256
[Updater] Adjusted binsize: 304334
[Updater] Computed Hash: ea 6e 9e 92 a2 0d 1c 9f 45 02 bf 36 de 3d 04 64 d6 fd 83 06 d8 35 94 79 c9 81 cf 09 bb 5f 1e e9
[Updater] Received Signature: 69 39 fe 7a 1c df 7b 1b 4b 11 48 11 e5 72 bb 22 f5 d8 de 8a 78 c0 1a c9 84 f3 11 ff 0a 55 2a 0e 7c bb 5d 58 40 ce e0 08 52 32 66 90 70 31 d9 88 c0 97 4c 21 cb 21 99 b8 85 72 9d 1d a4 26 eb 99 fa 
11 23 d7 80 40 39 b8 ef 47 79 2d 42 ee 77 55 06 47 6e 0a 01 b1 3c d7 c3 8e 20 84 38 4b 02 28 32 ce 72 e6 48 e5 9c fe 39 86 c7 bd 97 37 f3 c9 4a c3 7f f9 d9 eb 6f c0 46 83 a0 c6 f2 09 03 3b 36 4f 96 12 de de a2 46 84 b3 b6 d5 
a5 6d 6a 00 f6 fa 50 5f 3a 2c ce 2d d4 ea de 0f a0 0d ec 1b 78 9a 60 68 c4 af 87 3a 9c fd bf 7e a3 ae bb 2a 3c a0 ab 5e 28 3d 43 fa 9c 36 9e 68 58 36 b6 9d f2 f4 9a e4 2b 5b 29 cb f8 e9 8e 53 3c 75 54 ec 62 75 9c 03 37 48 d4 
90 e3 45 5a 8f a5 39 71 22 2d f9 a0 ec 83 42 32 32 42 4a 3a 87 82 dc f2 32 7a 0b 45 6d dd ec d8 03 d9 41 b7 17 b0 51 62 23
[Updater] Signature matches
Staged: address:0x002B5000, size:0x0004A5D2
[httpUpdate] Update ok
state: 5 -> 0 (0)
rm 0
pm close 7
del if0
usl

 ets Jan  8 2013,rst cause:2, boot mode:(3,6)

load 0x4010f000, len 3584, room 16
tail 0
chksum 0xb0
csum 0xb0
v2843a5ac
@cp:0
ld

earlephilhower commented 4 years ago

@cp:0 means the firmware was copied over by the bootloader.

Check https://github.com/esp8266/Arduino/blob/e636a6587c17d9ce20464ac6af140fb5e61bd1e5/bootloaders/eboot/eboot.c#L236-L238

So, I'm seeing no issue here. The new firmware was copied the it was loaded (ld).

Make sure you're uploading the file you want and not another one (http cache or something weird like that).

earlephilhower commented 4 years ago

Also, if the original bootloader does not have GZIP support, you could end up having it write gzip'd data instead of opcodes when it copies the new app over. You need to ensure the bootloader has gzip support (i.e. upload one FW using uncompressed mode), and once that's up only then try and use gzip'd bins.

The signing stuff should be orthogonal to any gzip support. Signing is just doing a signed hash compare over a set of bits, it doesn't care what those bits are. Once the signing is validated, it should be exactly the same as unsigned.

earlephilhower commented 4 years ago

Quick check of Update.cpp shows it might be getting the wrong bin size to copy, meaning the eboot gzip decompressor could abort because the len was invalid.

Try this patch, at line https://github.com/esp8266/Arduino/blob/e636a6587c17d9ce20464ac6af140fb5e61bd1e5/cores/esp8266/Updater.cpp#L284 add

    _size = binSize;

after the free() and report back.

krc-soft commented 4 years ago

It works 👍 thanks for your help!

esp8266 / Arduino