Twister doesn't work with Espressif platforms when splitting build and run across machines

[sam-golioth]

Describe the bug Twister has the ability to build tests on one machine and run them on another (useful in CI when building on a cloud-hosted machine and running on a local machine with hardware attached). This is accomplished by using --prep-artifacts-for-testing --package-artifacts when building the tests to produce a tar archive, moving the tar file to the test machine and extracting it, and then running twister with --test-only. This works great for most platforms, but it fails when running on Espressif based platforms (e.g. esp32_devkitc_wrover). There appear to be 2 related issues:

• The runners.yaml file generated during the build includes hardcoded absolute paths for the esp32 runner. When extracting the archive file on another machine or even in another folder, these paths are incorrect.
• The --package-artifacts argument in Twister doesn't package up the additional files that are required to flash Espressif targets, including the bootloader and partition table. So even if the path issue above was fixed, the actual files would be missing.

This was tested on Zephyr v3.5.0

To Reproduce Steps to reproduce the behavior:

1. Build twister tests for ESP32, e.g.: zephyr/scripts/twister --platform esp32_devkitc_wrover -T zephyr/samples/hello_world --prep-artifacts-for-testing --package-artifacts artifacts.tar
2. Extract artifacts in a different folder: mkdir twister-test && mv artifacts.tar twister-test && cd twister-test && tar -xvf artifacts.tar
3. Run Twister: ../zephyr/scripts/twister --test-only --platform esp32_devkitc_wrover -T ../zephyr/samples/hello_world --device-testing --device-serial <serial_port> --west-flash

This produce the following in device.log:

                           [--flash_freq {keep,80m,60m,48m,40m,30m,26m,24m,20m,16m,15m,12m}]
                           [--flash_mode {keep,qio,qout,dio,dout}]
                           [--flash_size {detect,keep,256KB,512KB,1MB,2MB,2MB-c1,4MB,4MB-c1,8MB,16MB,32MB,64MB,128MB}]
                           [--spi-connection SPI_CONNECTION] [--no-progress]
                           [--verify] [--encrypt]
                           [--encrypt-files <address> <filename> [<address> <filename> ...]]
                           [--ignore-flash-encryption-efuse-setting] [--force]
                           [--compress | --no-compress]
                           <address> <filename> [<address> <filename> ...]
esptool write_flash: error: argument <address> <filename>: [Errno 2] No such file or directory: '/Users/samfriedman/golioth/test_cp_sdk_zephyr/twister-out/esp32_devkitc_wrover/samples/hello_world/sample.basic.helloworld/esp-idf/build/bootloader/bootloader.bin'
FATAL ERROR: command exited with status 2: /Users/samfriedman/miniconda3/envs/golioth/bin/python /Users/samfriedman/golioth/test_cp_sdk_zephyr/modules/hal/espressif/components/esptool_py/esptool/esptool.py --chip auto --baud 921600 --before default_reset --after hard_reset write_flash -u --flash_mode dio --flash_freq 40m --flash_size detect 0x1000 /Users/samfriedman/golioth/test_cp_sdk_zephyr/twister-out/esp32_devkitc_wrover/samples/hello_world/sample.basic.helloworld/esp-idf/build/bootloader/bootloader.bin 0x8000 /Users/samfriedman/golioth/test_cp_sdk_zephyr/twister-out/esp32_devkitc_wrover/samples/hello_world/sample.basic.helloworld/esp-idf/build/partitions_singleapp.bin 0x10000 /Users/samfriedman/golioth/test_cp_sdk_zephyr/twister-test/twister-out/esp32_devkitc_wrover/samples/hello_world/sample.basic.helloworld/zephyr/zephyr.bin

Expected behavior Twister should be able run tests that were built on another machine on Espressif platforms.

Impact Our company develops an SDK on top of Zephyr and tests it against a variety of platforms. Because of this bug, we aren't testing against Espressif targets in CI, reducing our coverage.

[sylvioalves]

Hi @sam-golioth, sorry for taking so long to reply.

There are 2 issues in there:

1) Current error exists due to missing bootloader files, which are built as external project in Zephyr. To solve this, one option would be removing the --prep-artifacts-for-testing, which would add into the .tar file all the necessary files:
./scripts/twister --platform esp32_devkitc_wrover -T samples/hello_world --package-artifacts artifacts.tar

2) Twister is not able to run device tests in ESP32 due to serial port conflict. (flashing and reading log output). We have submitted a draft PR as a suggestion to fix it: #66534

Let me know if this works for you somehow.

Sylvio

[github-actions[bot]]

This issue has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this issue will automatically be closed in 14 days. Note, that you can always re-open a closed issue at any time.