Unit Testing not working on ESP32 (test port not automatically recognized)

[maxgerhardt]

What kind of issue is this?

• [X] PlatformIO Core. If you’ve found a bug, please provide an information below.

You can erase any parts of this template not applicable to your Issue.

---

Configuration

Operating system: Win10 x64

PlatformIO Version (platformio --version): version 5.2.1b4

Description of problem

Unit testing for an ESP32 hangs at the "Testing..." phase.

Steps to Reproduce

1. Create new blank ESP32 project
2. Add test/simple_test.cpp from below
3. Use the project task "Advanced -> Test"

Actual Results

Writing at 0x0003fafb... (100 %)
Wrote 210992 bytes (108452 compressed) at 0x00010000 in 3.1 seconds (effective 536.3 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...
Testing...
If you don't see any output for the first 10 secs, please reset board (press reset button)

(hangs forever)

Expected Results

Successfull unit test result.

If problems with PlatformIO Build System:

The content of platformio.ini:

[env:esp32dev]
platform = espressif32
board = esp32dev
framework = arduino

Source file to reproduce issue:

test\simple_test.cpp

#include <Arduino.h>
#include <unity.h>

void test_led_builtin_pin_number(void) {
    TEST_ASSERT_EQUAL(13, 13);
}

void setup() {
    delay(3000);
    UNITY_BEGIN();
    RUN_TEST(test_led_builtin_pin_number);
    UNITY_END(); // stop unit testing
}

void loop() { }

Additional info

1. The board is a standard https://www.amazon.de/ESP-32S-ESP32-Development-Bluetooth-Antennenmodul/dp/B071JR9WS9
2. pio test locks up so that it can't even killed by Ctrl+C, must be killed in the taskmanager
3. Increasing the delay() does not help
4. Explicitly setting test_speed = 115200 does not help
5. Modifying setup() to Serial.begin(115200); Serial.print("\n\n"); does not help (separates bootloader output from test output)
6. Setting DTR / RTS states with pio test --monitor-dtr <0/1> --monitor-rts <0/1> does not help
7. Pressing the reset button on the board does not help
8. Looking at the serial output after the testing has been aborted at the waiting phase gives

[valeros]

Hi @maxgerhardt ! I guess it works fine, except that the esp32dev board doesn't have any hwids in its manifest so PIO couldn't find a proper serial port and hence waits for a timeout to exceed. If you wait a bit longer it should print the Error: Please specify test_port for environment or use global --test-port option message. Could you please try to manually specify the test_port option in your platformio.ini?

Although, during some attempts to reproduce the issue I indeed bumped into situations when the test command sporadically became unresponsive even to the Ctrl+C signal, but it seems like a more generic problem that has nothing to do with ESP32 specifically.

[maxgerhardt]

That was indeed the problem.

Uploading...
Testing...
If you don't see any output for the first 10 secs, please reset board (press reset button)

ets Jun  8 2016 00:22:57
rst:0x10 (RTCWDT_RTC_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0018,len:4
load:0x3fff001c,len:1044
load:0x40078000,len:10124
load:0x40080400,len:5828
entry 0x400806a8
test\simple_test.cpp:12:test_led_builtin_pin_number     [PASSED]
-----------------------
1 Tests 0 Failures 0 Ignored
===================== [PASSED] Took 30.94 seconds =====================
Test    Environment    Status    Duration
------  -------------  --------  ------------
*       esp32dev       PASSED    00:00:30.943
===================== 1 succeeded in 00:00:30.943 =====================

That was quite baffling because I thought the detection of the test_port would be the same as the automatic detection of the upload_port, which worked flawlessly for me. I only have my motherboard's COM port and the ESP32 dev board as serial ports.

[ankostis]

@maxgerhardtI would edit the title to reflect the content of this report, as a courtesy to future bug-reporters.

[maxgerhardt]

I'm not sure what title would be better? The problem of unit testing not working was resolved by setting test_port.

[ankostis]

Maybe "test_port automatic detection fails"?

[maxgerhardt]

People who will experience the problem will first think of "Unit testing not working" instead of knowing the solution that test port detection fails though :/

[ankostis]

The message i'm conveying is that if all kinds of issues have generic titles "foo don't work", "bar crashes", it's difficult to orient when scanning through them. Eg when looking at search results, the results are matched by GitHub based on the contents of the issues (not just the titles), but it's only the titles that's present in the result-list, and these better help humans to remember which one is which, and avoid revisiting the same issue again and again.

Certainly it's not that important all this discussion, still you could extend the current title with some identifying info, if you agree with the above perspective. Thank you for your time.

[maxgerhardt]

Updated

[HareshPrajapati]

Hi @maxgerhardt , Thank you for solving problem , but setting test_port in the platformio.ini file solved problem only for unit test not for google test ! Currently I m doing Google test using 'esp32dev' board , some how test run in the board but it hangs on one task I mean I try to test serval unit tests and google tests in same project and when google test executed other tests not executing after that. surprisingly , All tests on native environment works perfectly one by one but not working in esp32dev board

I m using google/googletest@^1.11.0 library.

This is my configuration for both env (See below image ):

This is the problem (See below image):

[ivankravets]

1. Please remove googletest from lib_deps
2. Switch to the latest PlatformIO Core 6.0 (pio upgrade --dev)
3. Set test_framework = googletest
4. See new docs https://docs.platformio.org/en/latest/advanced/unit-testing/frameworks/googletest.html

Does it work for you?

[HareshPrajapati]

I tried it ! I think After upgradation of PlatformIO Core 6.0.0rc2 I get more troubles !!! This time I removed all unit test examples and try to build google test with your suggestions but still it hangs on google test

[ivankravets]

Have you completed 4) from the list above? Please read again the Get Started section. There is an example of how to implement the main or setup/loop entry points.

Does it work now?

[HareshPrajapati]

It Works !!!! @ivankravets

I just changed my test file names to test_main.cpp and Set port in platformIO.ini file. You need to set all files names as a test_main.cpp in folders where you want your test code you still need to set test_port in platformIO.ini file.

I suggest to try this example for getting started. I mean if you have a two code for test than your structure should look like this :

above steps are needed after latest platformIO update (6.0.0) my old configuration of platformIO runs with any type of file names in test folder

One more thing ..

Don't forget to put below functions in unity test codes otherwise you will get error.

void setUp(void) {
    // set stuff up here
}

void tearDown(void) {
    // clean stuff up here
}

[ivankravets]

@HareshPrajapati

1. You don't need to specify the test port manually. PlatformIO detects it automatically. Could you share the output of pio device list command?
2. You don't need to declare setUp/tearDown. They are intended for the Unity testing framework.

[HareshPrajapati]

@ivankravets

1. this is my output for pio device list :

F:\1_Hello_World\UnitTest>pio device list
COM4
----
Hardware ID: USB VID:PID=10C4:EA60 SER=0001 LOCATION=1-1.2
Description: Silicon Labs CP210x USB to UART Bridge (COM4)

COM1
----
Hardware ID: ACPI\PNP0501\1
Description: Communications Port (COM1)

COM5
----
Hardware ID: PCI\VEN_8086&DEV_1E3D&SUBSYS_339A103C&REV_04\3&11583659&0&B3
Description: Intel(R) Active Management Technology - SOL (COM5)

2. You are right ,but some how it throwing errors while I was building unity tests.

[ivankravets]

I think we can improve 1). Could file an enhancement request at https://github.com/platformio/platformio-core/issues ?

[HareshPrajapati]

@ivankravets Created 4263

[ivankravets]

Hi guys,

Could I ask you to re-test the latest pio upgrade --dev? It has significant improvements regarding automatic serial port detection of test_port and monito_port.

[HareshPrajapati]

@ivankravets

Works perfectly 💯 🥇

[kapanmc]

@ivankravets may I also ask for your kind help for similar issue

I'm building a project on ESP32-S3 and lately we decided to add tests into the codes and ran into the exact same problem as @maxgerhardt My terminal was stuck at If you don't see any output for the first 10 secs, please reset board (press reset button) line forever.

Then after reading this thread I added test_port = /dev/cu.usbserial-2130 into my .ini file and it's not stuck anymore yet it doesn't run the tests either!

What would be the problem and how may I get it solved if anyone can help?

Below is my platformio.ini lines;

[platformio]
src_dir = src
boards_dir = .

[env:esp32-s3-devkitc-1-myboard]
platform = espressif32
board = esp32-s3-devkitc-1-myboard
framework = arduino
monitor_speed = 115200
platform_packages = framework-arduinoespressif32 @ https://github.com/espressif/arduino-esp32#2.0.3
build_flags = 
    -D LV_LVGL_H_INCLUDE_SIMPLE
    -I./include
lib_deps = 
    lovyan03/LovyanGFX@^1.1.12
    tamctec/TAMC_GT911@^1.0.2
    lvgl/lvgl@8.3.6
    fbiego/ESP32Time@^2.0.6
    throwtheswitch/Unity@^2.5.2
board_build.partitions = huge_app.csv

test_build_src = true  
test_port = /dev/cu.usbserial-2130

and here is my test_main.cpp codes:

#include <Arduino.h>
#include <unity.h>
#include "events_init.h"

//conflicts with "setup" and "loop" in main, therefore I modified below 2 as 
void setup_test(); 
void loop_test(); 

void test_parlaklik_initial_value(void) {
    TEST_ASSERT_EQUAL(4, parlaklik_seviye); //testing a static variable from my project to see how testing works with my own parameters.
}

int add(int a, int b) {
    return a + b;
}

void test_add() { //testing some general stuff to learn test framework in general
    TEST_ASSERT_EQUAL(5, add(2, 3));
    TEST_ASSERT_EQUAL(0, add(-2, 2));
    TEST_ASSERT_EQUAL(-5, add(-3, -2));
}

void setup_test() {
    // Note: delay is needed to allow the serial monitor to initialize
    delay(5000);
    UNITY_BEGIN();
    RUN_TEST(test_parlaklik_initial_value);
    RUN_TEST(test_add);
    UNITY_END();
}

void loop_test() {
    // Leave empty
}

finally here is my terminal output:

➜  touchscreen git:(main) ✗ pio test -vv
Collected 1 tests (*)

Processing * in esp32-s3-devkitc-1-myboard environment
---------------------------------------------------------------------------------------------------------------------------------------------

Building & Uploading...
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif32/esp32-s3-devkitc-1-myboard.html
PLATFORM: Espressif 32 (6.7.0) > Espressif ESP32-S3-DevKitC-1-N8 -ELECROW
HARDWARE: ESP32S3 240MHz, 320KB RAM, 8MB Flash
DEBUG: Current (esp-builtin) On-board (esp-builti
.
.
.
.
.
.
Writing at 0x0023a3b5... (92 %)
Writing at 0x00241e30... (94 %)
Writing at 0x0024b19a... (97 %)
Writing at 0x00252688... (100 %)
Wrote 2374640 bytes (609456 compressed) at 0x00010000 in 20.0 seconds (effective 948.3 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...

Testing...
If you don't see any output for the first 10 secs, please reset board (press reset button)

[  1663][E][Wire.cpp:159] begin(): Bus already started in Master Mode.
[  1663][E][esp32-hal-gpio.c:95] __pinMode(): Invalid pin selected
[  1664][E][esp32-hal-gpio.c:95] __pinMode(): Invalid pin selected
E (1252) gpio: gpio_set_level(226): GPIO output gpio_num error
E (1257) gpio: gpio_set_level(226): GPIO output gpio_num error
E (1273) gpio: gpio_set_level(226): GPIO output gpio_num error
E (1274) gpio: gpio_set_level(226): GPIO output gpio_num error
E (1279) gpio: gpio_set_level(226): GPIO output gpio_num error
[  1747][E][esp32-hal-gpio.c:95] __pinMode(): Invalid pin selected

[maxgerhardt]

//conflicts with "setup" and "loop" in main, therefore I modified below 2 as 
void setup_test(); 
void loop_test(); 

so how are the Unit tests supposed to be executed when it doesn't reach execution of that function? Who occupies the real setup() and loop() functions? The events_init.h lirary? If yes, fix that library.

In any case, your issue is with your code, not PlatformIO itself.

[kapanmc]

Thanks for very fast response @maxgerhardt main.cpp occupies the real setup() and loop() functions.

Because it's my first time trying to implement a test environment, I may get confused easily.

From examples and tutorials I read, all of them has setup() and loop() inside test_main.cpp ; however if i proceed like that, compiler complains that duplicate declaration.

[maxgerhardt]

Don't set test_build_src = true when the code in there defines setup() and loop(). All to-be-tested components should be in lib/<your library name> folders, so that they can be accessed independently by either the regular firmware (src) or tests.

[kapanmc]

[UPDATE: I've added lib_deps = Wire and the dependency issue got resolved] The reason why i set test_build_src = true is because compiler is terminated due to a dependency(attached is the SS of terminal output)

However, if I just compile and upload the code as it is, there's no problem with this dependency because Wire.h is already under .pio/libdeps/

[maxgerhardt]

If a subdependency can't be found, you might need to increase the lib_ldf_mode. In any case, this issue is getting filled with lots of off-topic content. The issue about test port recognition was long solved.

[kapanmc]

Yes, thank you very much for your kind and very quick answers @maxgerhardt