Request for Guidance on Multi-Stage Testing for OTA Updates and Proposal for an Advanced Example (RDT-798)

[hayschan]

I am currently implementing an OTA update feature for ESP32 devices, using an OTA server. This feature is crucial as it allows for remote updates and bug fixes post-deployment. The testing strategy is designed to ensure robustness in real-world scenarios, where a device must handle firmware updates seamlessly.

Test Setup Description

I have set up a CI pipeline that includes building the firmware, flashing it onto an ESP32, and conducting OTA updates through the OTA server. The ESP32 is pre-registered on the OTA server as ota_testing_machine with a special OTA package (v999) assigned to simulate an update with a significantly higher version number. The sequence of the pipeline is as follows:

graph TD;
    A[Start CI Pipeline] --> B[Build Firmware];
    B --> C[Flash Firmware to ESP32];
    C --> D{ESP32 Connects to the OTA server};
    D -->|Yes| E[Device Registers as 'ota_testing_machine'];
    D -->|No| F[Fail Test];
    E --> G[Assign OTA Package 'v999'];
    G --> H[Trigger OTA Update];
    H --> I[ESP32 Downloads and Installs 'v999' firmware];
    I --> J[ESP32 Reboots];
    J --> K[ESP32 Prints New Version to Console];
    K -->|Version is 'v999'| L[Pass Test];
    K -->|Version not 'v999'| M[Fail Test];
    L --> N[End CI Pipeline];
    M --> N;

Request for Information

1. Support for Multi-Stage Testing: Does pytest-embedded support tests that require device resets as part of the testing process? This is akin to multi-stage testing in Unity, where a device may undergo resets to validate different stages of firmware handling, especially post-OTA update.
2. Example Code for Multi-Stage Testing: If multi-stage testing is supported, could you provide guidance or example code that integrates this methodology with pytest-embedded? This is particularly relevant for testing OTA updates where the device needs to reboot to apply new firmware and continue testing post-reboot.

Proposed Contribution

Given the complexity of implementing realistic OTA update scenarios, I have noticed the examples in the repository, like hello_world, may not fully represent the challenges in such applications. I propose to contribute a detailed example based on my setup to enhance the utility of pytest-embedded for developers facing similar challenges. I am eager to contribute for the repo.

Additional Resources and Documentation

For context on multi-stage testing, here's how Espressif describes it for ESP-IDF: Espressif Multi-Stage Testing Guide

Your guidance will greatly enhance the effectiveness of our testing strategy and contribute to a more robust implementation of OTA updates in IoT devices.

Thank you for your support and consideration.

[igrr]

Does pytest-embedded support tests that require device resets as part of the testing process

You can call dut.reset() or dut.serial.hard_reset() to toggle a reset.

This is particularly relevant for testing OTA updates where the device needs to reboot to apply new firmware and continue testing post-reboot.

You can refer to this test of ESP-IDF OTA example:

https://github.com/espressif/esp-idf/blob/0453e8608bde98133a427a74ae61d272770b1bfd/examples/system/ota/advanced_https_ota/pytest_advanced_ota.py#L368-L374

[hayschan]

Thanks for the quicky reply. I'll try them out immediately.