Remote unit testing only uploads a single test in v6+

[noisymime]

• [ 🟢 ] PlatformIO Core. If you’ve found a bug, please provide an information below.

You can erase any parts of this template not applicable to your Issue.

---

Configuration

Operating system: Multiple, have confirmed on MacOS and linux

PlatformIO Version (platformio --version): 6.0.2rc2

Description of problem

Since upgrading to a v6.x.x version, remote unit testing appears to only upload and test the last unit test if there are multiple present. Local unit testing works fine, however remote testing only performs a single compile and then uploads the same hex file to the remote multiple times (once per test) rather than performing a new compile between each upload

I have verified that this was working ok in v5.x, but not since moving to 6.0.2rc2 (I have not tried any of the other 6.x releases). I suspect this maybe related to the hierarchical unit testing that was added in 6.0, but cannot verify this.

Actual Results

Log of an attempt to perform a remote test:

xxx@yyy speeduino % pio remote test
Building project locally
Verbosity level can be increased via `-v, -vv, or -vvv` option
Collected 6 tests

Processing test_misc2 in megaatmega2560 environment
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Building...
--------------------------------------------------------------------------------------- megaatmega2560:test_misc2 [PASSED] Took 7.88 seconds ---------------------------------------------------------------------------------------

Processing test_decoders in megaatmega2560 environment
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Building...
-------------------------------------------------------------------------------------- megaatmega2560:test_decoders [PASSED] Took 7.72 seconds --------------------------------------------------------------------------------------

Processing test_misc in megaatmega2560 environment
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Building...
--------------------------------------------------------------------------------------- megaatmega2560:test_misc [PASSED] Took 10.22 seconds ---------------------------------------------------------------------------------------

Processing test_fuel in megaatmega2560 environment
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Building...
---------------------------------------------------------------------------------------- megaatmega2560:test_fuel [PASSED] Took 9.91 seconds ----------------------------------------------------------------------------------------

Processing test_schedules in megaatmega2560 environment
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Building...
------------------------------------------------------------------------------------- megaatmega2560:test_schedules [PASSED] Took 12.17 seconds -------------------------------------------------------------------------------------

============================================================================================= 0 test cases: 0 succeeded in 00:00:47.902 =============================================================================================
Testing project remotely
Verbose mode can be enabled via `-v, --verbose` option
Collected 6 tests

Processing test_decoders in megaatmega2560 environment
--------------------------------------------------------------------------------
Uploading...

Warning! Please install `99-platformio-udev.rules`. 
More details: https://docs.platformio.org/page/faq.html#platformio-udev-rules

avrdude: AVR device initialized and ready to accept instructions

Reading | ################################################## | 100% 0.01s

avrdude: Device signature = 0x1e9801 (probably m2560)
avrdude: reading input file ".pio/build/megaatmega2560/firmware.hex"
avrdude: writing flash (124334 bytes):

Writing | ################################################## | 100% 19.91s

avrdude: 124334 bytes of flash written
avrdude: verifying flash memory against .pio/build/megaatmega2560/firmware.hex:
avrdude: load data flash data from input file .pio/build/megaatmega2560/firmware.hex:
avrdude: input file .pio/build/megaatmega2560/firmware.hex contains 124334 bytes
avrdude: reading on-chip flash data:

Reading | ################################################## | 100% 15.73s

avrdude: verifying ...
avrdude: 124334 bytes of flash verified

avrdude done.  Thank you.

Testing...
If you don't see any output for the first 10 secs, please reset board (press reset button)

test/test_schedules/test_schedules.cpp:106: test_status_initial_off_inj1    [PASSED]
test/test_schedules/test_schedules.cpp:107: test_status_initial_off_inj2    [PASSED]
test/test_schedules/test_schedules.cpp:108: test_status_initial_off_inj3    [PASSED]
test/test_schedules/test_schedules.cpp:109: test_status_initial_off_inj4    [PASSED]
test/test_schedules/test_schedules.cpp:110: test_status_initial_off_inj5    [PASSED]
test/test_schedules/test_schedules.cpp:111: test_status_initial_off_inj6    [PASSED]
test/test_schedules/test_schedules.cpp:112: test_status_initial_off_inj7    [PASSED]
test/test_schedules/test_schedules.cpp:113: test_status_initial_off_inj8    [PASSED]
test/test_schedules/test_schedules.cpp:115: test_status_initial_off_ign1    [PASSED]
test/test_schedules/test_schedules.cpp:116: test_status_initial_off_ign2    [PASSED]
test/test_schedules/test_schedules.cpp:117: test_status_initial_off_ign3    [PASSED]
test/test_schedules/test_schedules.cpp:118: test_status_initial_off_ign4    [PASSED]
test/test_schedules/test_schedules.cpp:119: test_status_initial_off_ign5    [PASSED]
test/test_schedules/test_schedules.cpp:120: test_status_initial_off_ign6    [PASSED]
test/test_schedules/test_schedules.cpp:121: test_status_initial_off_ign7    [PASSED]
test/test_schedules/test_schedules.cpp:122: test_status_initial_off_ign8    [PASSED]
test/test_schedules/test_schedules.cpp:187: test_accuracy_timeout_inj1  [PASSED]
test/test_schedules/test_schedules.cpp:188: test_accuracy_timeout_inj2  [PASSED]
test/test_schedules/test_schedules.cpp:189: test_accuracy_timeout_inj3  [PASSED]
test/test_schedules/test_schedules.cpp:190: test_accuracy_timeout_inj4  [PASSED]
test/test_schedules/test_schedules.cpp:191: test_accuracy_timeout_inj5  [PASSED]
test/test_schedules/test_schedules.cpp:192: test_accuracy_timeout_inj6  [PASSED]
test/test_schedules/test_schedules.cpp:193: test_accuracy_timeout_inj7  [PASSED]
test/test_schedules/test_schedules.cpp:194: test_accuracy_timeout_inj8  [PASSED]
test/test_schedules/test_schedules.cpp:196: test_accuracy_timeout_ign1  [PASSED]
test/test_schedules/test_schedules.cpp:197: test_accuracy_timeout_ign2  [PASSED]
test/test_schedules/test_schedules.cpp:198: test_accuracy_timeout_ign3  [PASSED]
test/test_schedules/test_schedules.cpp:199: test_accuracy_timeout_ign4  [PASSED]
test/test_schedules/test_schedules.cpp:200: test_accuracy_timeout_ign5  [PASSED]
test/test_schedules/test_schedules.cpp:201: test_accuracy_timeout_ign6  [PASSED]
test/test_schedules/test_schedules.cpp:202: test_accuracy_timeout_ign7  [PASSED]
test/test_schedules/test_schedules.cpp:203: test_accuracy_timeout_ign8  [PASSED]
test/test_schedules/test_schedules.cpp:187: test_accuracy_duration_inj1 [PASSED]
test/test_schedules/test_schedules.cpp:188: test_accuracy_duration_inj2 [PASSED]
test/test_schedules/test_schedules.cpp:189: test_accuracy_duration_inj3 [PASSED]
test/test_schedules/test_schedules.cpp:190: test_accuracy_duration_inj4 [PASSED]
test/test_schedules/test_schedules.cpp:191: test_accuracy_duration_inj5 [PASSED]
test/test_schedules/test_schedules.cpp:192: test_accuracy_duration_inj6 [PASSED]
test/test_schedules/test_schedules.cpp:193: test_accuracy_duration_inj7 [PASSED]
test/test_schedules/test_schedules.cpp:194: test_accuracy_duration_inj8 [PASSED]
test/test_schedules/test_schedules.cpp:196: test_accuracy_duration_ign1 [PASSED]
test/test_schedules/test_schedules.cpp:197: test_accuracy_duration_ign2 [PASSED]
test/test_schedules/test_schedules.cpp:198: test_accuracy_duration_ign3 [PASSED]
test/test_schedules/test_schedules.cpp:199: test_accuracy_duration_ign4 [PASSED]
test/test_schedules/test_schedules.cpp:200: test_accuracy_duration_ign5 [PASSED]
test/test_schedules/test_schedules.cpp:201: test_accuracy_duration_ign6 [PASSED]
test/test_schedules/test_schedules.cpp:202: test_accuracy_duration_ign7 [PASSED]
test/test_schedules/test_schedules.cpp:203: test_accuracy_duration_ign8 [PASSED]
----------- megaatmega2560:test_decoders [PASSED] Took 39.94 seconds -----------

Processing test_fuel in megaatmega2560 environment
--------------------------------------------------------------------------------
Uploading...

Warning! Please install `99-platformio-udev.rules`. 
More details: https://docs.platformio.org/page/faq.html#platformio-udev-rules

avrdude: AVR device initialized and ready to accept instructions

Reading | ################################################## | 100% 0.01s

avrdude: Device signature = 0x1e9801 (probably m2560)
avrdude: reading input file ".pio/build/megaatmega2560/firmware.hex"
avrdude: writing flash (124334 bytes):

Writing | ################################################## | 100% 19.91s

avrdude: 124334 bytes of flash written
avrdude: verifying flash memory against .pio/build/megaatmega2560/firmware.hex:
avrdude: load data flash data from input file .pio/build/megaatmega2560/firmware.hex:
avrdude: input file .pio/build/megaatmega2560/firmware.hex contains 124334 bytes
avrdude: reading on-chip flash data:

Reading | ################################################## | 100% 15.68s

avrdude: verifying ...
avrdude: 124334 bytes of flash verified

avrdude done.  Thank you.

Testing...
If you don't see any output for the first 10 secs, please reset board (press reset button)

test/test_schedules/test_schedules.cpp:106: test_status_initial_off_inj1    [PASSED]
test/test_schedules/test_schedules.cpp:107: test_status_initial_off_inj2    [PASSED]
test/test_schedules/test_schedules.cpp:108: test_status_initial_off_inj3    [PASSED]
test/test_schedules/test_schedules.cpp:109: test_status_initial_off_inj4    [PASSED]
test/test_schedules/test_schedules.cpp:110: test_status_initial_off_inj5    [PASSED]
test/test_schedules/test_schedules.cpp:111: test_status_initial_off_inj6    [PASSED]
test/test_schedules/test_schedules.cpp:112: test_status_initial_off_inj7    [PASSED]
test/test_schedules/test_schedules.cpp:113: test_status_initial_off_inj8    [PASSED]
test/test_schedules/test_schedules.cpp:115: test_status_initial_off_ign1    [PASSED]
test/test_schedules/test_schedules.cpp:116: test_status_initial_off_ign2    [PASSED]
test/test_schedules/test_schedules.cpp:117: test_status_initial_off_ign3    [PASSED]
test/test_schedules/test_schedules.cpp:118: test_status_initial_off_ign4    [PASSED]
test/test_schedules/test_schedules.cpp:119: test_status_initial_off_ign5    [PASSED]
test/test_schedules/test_schedules.cpp:120: test_status_initial_off_ign6    [PASSED]
test/test_schedules/test_schedules.cpp:121: test_status_initial_off_ign7    [PASSED]
test/test_schedules/test_schedules.cpp:122: test_status_initial_off_ign8    [PASSED]
test/test_schedules/test_schedules.cpp:187: test_accuracy_timeout_inj1  [PASSED]
test/test_schedules/test_schedules.cpp:188: test_accuracy_timeout_inj2  [PASSED]
test/test_schedules/test_schedules.cpp:189: test_accuracy_timeout_inj3  [PASSED]
test/test_schedules/test_schedules.cpp:190: test_accuracy_timeout_inj4  [PASSED]
test/test_schedules/test_schedules.cpp:191: test_accuracy_timeout_inj5  [PASSED]
test/test_schedules/test_schedules.cpp:192: test_accuracy_timeout_inj6  [PASSED]
test/test_schedules/test_schedules.cpp:193: test_accuracy_timeout_inj7  [PASSED]
test/test_schedules/test_schedules.cpp:194: test_accuracy_timeout_inj8  [PASSED]
test/test_schedules/test_schedules.cpp:196: test_accuracy_timeout_ign1  [PASSED]
test/test_schedules/test_schedules.cpp:197: test_accuracy_timeout_ign2  [PASSED]
test/test_schedules/test_schedules.cpp:198: test_accuracy_timeout_ign3  [PASSED]
test/test_schedules/test_schedules.cpp:199: test_accuracy_timeout_ign4  [PASSED]
test/test_schedules/test_schedules.cpp:200: test_accuracy_timeout_ign5  [PASSED]
test/test_schedules/test_schedules.cpp:201: test_accuracy_timeout_ign6  [PASSED]
test/test_schedules/test_schedules.cpp:202: test_accuracy_timeout_ign7  [PASSED]
test/test_schedules/test_schedules.cpp:203: test_accuracy_timeout_ign8  [PASSED]
test/test_schedules/test_schedules.cpp:187: test_accuracy_duration_inj1 [PASSED]
test/test_schedules/test_schedules.cpp:188: test_accuracy_duration_inj2 [PASSED]
test/test_schedules/test_schedules.cpp:189: test_accuracy_duration_inj3 [PASSED]
test/test_schedules/test_schedules.cpp:190: test_accuracy_duration_inj4 [PASSED]
test/test_schedules/test_schedules.cpp:191: test_accuracy_duration_inj5 [PASSED]
test/test_schedules/test_schedules.cpp:192: test_accuracy_duration_inj6 [PASSED]
test/test_schedules/test_schedules.cpp:193: test_accuracy_duration_inj7 [PASSED]
test/test_schedules/test_schedules.cpp:194: test_accuracy_duration_inj8 [PASSED]
test/test_schedules/test_schedules.cpp:196: test_accuracy_duration_ign1 [PASSED]
test/test_schedules/test_schedules.cpp:197: test_accuracy_duration_ign2 [PASSED]
test/test_schedules/test_schedules.cpp:198: test_accuracy_duration_ign3 [PASSED]
test/test_schedules/test_schedules.cpp:199: test_accuracy_duration_ign4 [PASSED]
test/test_schedules/test_schedules.cpp:200: test_accuracy_duration_ign5 [PASSED]
test/test_schedules/test_schedules.cpp:201: test_accuracy_duration_ign6 [PASSED]
test/test_schedules/test_schedules.cpp:202: test_accuracy_duration_ign7 [PASSED]
test/test_schedules/test_schedules.cpp:203: test_accuracy_duration_ign8 [PASSED]
------------- megaatmega2560:test_fuel [PASSED] Took 39.90 seconds -------------

Note:

• I have only included the first 2 upload cycles of this test, but it demonstrates the problem
• Despite this claiming to perform the test_decoders and test_fuel tests, it is actually the test_schedules hex file that is uploaded both times (Note that the uploaded file size is the same both times)
• At the beginning you can see all tests being built, but the remote upload only takes place after all of them have been built, meaning that it is only the last test that is run (over and over).

If problems with PlatformIO Build System:

The content of platformio.ini:

[env:megaatmega2560]
platform=atmelavr
board=megaatmega2560
framework=arduino
build_unflags = -Os
build_flags = -O3 -ffast-math -fshort-enums -funroll-loops -Wall -Wextra -std=c99
lib_deps = EEPROM, Time
;test_build_project_src = true
test_build_src = yes
debug_tool = simavr
test_ignore = test_table3d_native

[ivankravets]

Duplicate of https://github.com/platformio/platformio-core/issues/3270

See solution in #3270