search

ARMmbed / htrun

MOVED: https://github.com/ARMmbed/mbed-os-tools (Flash, reset and run host supervised tests on mbed platforms)
8 stars 37 forks source link

-e *adds* to host test list when using --list, but *overrides* mbed host tests when actually running #217

Closed Alex-EEE closed 5 years ago

Alex-EEE commented 6 years ago

Repro steps:

Use mbed test with -e to add a folder for custom host tests, and run all unit tests. Any mbed unit test needing any of the "out of the box" host tests ( those that live in tools/host_tests ) will fail, saying it can't find tests like "RTCTest"

Although invoking mbedhtrun --list -e custom/host_tests will show the custom host tests getting added to the default ones:

+---------------+---------------------------------------------------------------------+---------------------------------------------------------------------------------------+
| name          | class                                                               | origin                                                                                |
+---------------+---------------------------------------------------------------------+---------------------------------------------------------------------------------------+
| default       | <class 'mbed_host_tests.host_tests.default_auto.DefaultAuto'>       | mbed-host-tests                                                                       |
| default_auto  | <class 'mbed_host_tests.host_tests.default_auto.DefaultAuto'>       | mbed-host-tests                                                                       |
| detect_auto   | <class 'mbed_host_tests.host_tests.detect_auto.DetectPlatformTest'> | mbed-host-tests                                                                       |
| dev_null_auto | <class 'mbed_host_tests.host_tests.dev_null_auto.DevNullTest'>      | mbed-host-tests                                                                       |
| echo          | <class 'mbed_host_tests.host_tests.echo.EchoTest'>                  | mbed-host-tests                                                                       |
| hello_auto    | <class 'mbed_host_tests.host_tests.hello_auto.HelloTest'>           | mbed-host-tests                                                                       |
| lwm2m         | <class 'lwm2m.Lwm2m'>                                               | C:\files\mbed\staging\framework\test-server-dependencies\greentea-host-tests\lwm2m.py |
| rtc_auto      | <class 'mbed_host_tests.host_tests.rtc_auto.RTCTest'>               | mbed-host-tests                                                                       |
| wait_us_auto  | <class 'mbed_host_tests.host_tests.wait_us_auto.WaitusTest'>        | mbed-host-tests                                                                       |
+---------------+---------------------------------------------------------------------+---------------------------------------------------------------------------------------+
theotherjimmy commented 6 years ago

Huh. Thanks for the report. I'll take a look. [Mirrored to Jira]

theotherjimmy commented 6 years ago

I'm pretty sure that listing will add all the default host tests, while running may skip that step.

My evidence is this line https://github.com/ARMmbed/htrun/blob/master/mbed_host_tests/__init__.py#L112

@bridadan @studavekar Would it be wise to remove this line, make -e accept many arguments, and default to ["./tests/host_tests"]? [Mirrored to Jira]

Alex-EEE commented 6 years ago

I like the many folders option, b/c I can easily see a desire to pull in host tests from multiple folders

BUT, I would always just add those tests to the default tests, not replace them. Most developers are probably like us: we run all of the "out of the box" mbed tests every night on our custom board, AND, we run the unit tests we've written for our custom modules. If you don't include the default tests, then some of those tests break.

There's really no downside to including the defaults. They're called by name, so if you don't need them, they just sit unused. [Mirrored to Jira]

theotherjimmy commented 6 years ago

Hi @Alex-EEE Thanks for getting back to me. The only downside I could think of is performance (ha!). [Mirrored to Jira]

theotherjimmy commented 6 years ago

Further, whatever we do should be consistent between --list and running. [Mirrored to Jira]

bridadan commented 6 years ago

I'm with @Alex-EEE, we should just add the new host tests vs replace them. And we should definitely be consistent between --list and running.

EDIT: Also, I like the idea of being able to supply -e multiple times. [Mirrored to Jira]

theotherjimmy commented 6 years ago

Alright. I'll implement this then. [Mirrored to Jira]

theotherjimmy commented 6 years ago

@Alex-EEE Maybe this is user error: try referring to the test using the name from the "name" column. Your "lwm2m" host_test happens to have a class name that's the same as the python filename without the ".py" suffix. [Mirrored to Jira]

theotherjimmy commented 6 years ago

BTW, the PR makes a few changes: no abspath in printing to match what's passed in, and multiple -e accepted. [Mirrored to Jira]

Alex-EEE commented 6 years ago

@theotherjimmy, do you mean referring to it inside the c code running on the DUT? This is what my reference to the host test looks like there: "GREENTEA_SETUP(120, "lwm2m"); " [Mirrored to Jira]

theotherjimmy commented 6 years ago

@Alex-EEE Yes. That matches the "name" column, which does not have a capitol "L". Perhaps you could refer to the RTCTest class as rtc_auto? [Mirrored to Jira]

Alex-EEE commented 6 years ago

@theotherjimmy I'm not using RTCTest, an mbed-os unit test uses that.

Just to clarify, I can get my greenTea test to pick up lwm2m when I run it by itself. It's just an either or situation right now...I can either get all the mbed-os tests to run, but not my custom test, or I can get my custom test to run, but then all the mbed-os tests that need host tests break [Mirrored to Jira]

theotherjimmy commented 6 years ago

That strange. I'll have to check into how lookup works... [Mirrored to Jira]

Alex-EEE commented 6 years ago

Well wait, I haven't tried your PR yet...this all goes back to original problem...b/c "running" and --list don't match, it creates problems. If you made running behave like --list, then I think everything should be golden. [Mirrored to Jira]

theotherjimmy commented 6 years ago

I don't think I changed anything in how running works, and the default tests are not in ./tests/host_tests. So I don't think my changes will help, yet. [Mirrored to Jira]

theotherjimmy commented 6 years ago

All GREENTEA_SETUP invocations in Mbed OS:

TESTS/mbed_drivers/c_strings/main.cpp:    GREENTEA_SETUP(5, "default_auto");
TESTS/mbed_drivers/dev_null/main.cpp:    GREENTEA_SETUP(2, "dev_null_auto");
TESTS/mbed_drivers/echo/main.cpp:    GREENTEA_SETUP(30, "device_echo");
TESTS/mbed_drivers/generic_tests/main.cpp:    GREENTEA_SETUP(20, "default_auto");
TESTS/mbed_drivers/rtc/main.cpp:    GREENTEA_SETUP(20, "default_auto");
TESTS/mbed_drivers/stl_features/main.cpp:    GREENTEA_SETUP(5, "default_auto");
TESTS/mbed_drivers/ticker/main.cpp:    GREENTEA_SETUP(test_timeout, "timing_drift_auto");
TESTS/mbed_drivers/lp_timeout/main.cpp:    GREENTEA_SETUP(240, "timing_drift_auto");
TESTS/mbed_drivers/mem_trace/main.cpp:    GREENTEA_SETUP(15, "default_auto");
TESTS/mbed_drivers/timeout/main.cpp:    GREENTEA_SETUP(240, "timing_drift_auto");
TESTS/mbed_drivers/race_test/main.cpp:    GREENTEA_SETUP(20, "default_auto");
TESTS/mbed_drivers/flashiap/main.cpp:    GREENTEA_SETUP(120, "default_auto");
TESTS/mbed_drivers/crc/main.cpp:    GREENTEA_SETUP(15, "default_auto");
TESTS/mbed_drivers/lp_ticker/main.cpp:    GREENTEA_SETUP(test_timeout, "timing_drift_auto");
TESTS/integration/basic/main.cpp:    GREENTEA_SETUP(15, "default_auto");
TESTS/mbed_drivers/lp_timer/main.cpp:    GREENTEA_SETUP(15, "default_auto");
TESTS/mbed_drivers/sleep_lock/main.cpp:    GREENTEA_SETUP(20, "default_auto");
TESTS/mbed_drivers/timerevent/main.cpp:    GREENTEA_SETUP(5, "default_auto");
TESTS/mbed_drivers/timer/main.cpp:    GREENTEA_SETUP(15, "default_auto");
TESTS/mbedmicro-mbed/attributes/main.cpp:    GREENTEA_SETUP(5, "default_auto");
TESTS/mbedmicro-mbed/call_before_main/main.cpp:    GREENTEA_SETUP(5, "default_auto");
TESTS/mbedmicro-mbed/cpp/main.cpp:    GREENTEA_SETUP(10, "default_auto");
TESTS/mbedmicro-mbed/div/main.cpp:    GREENTEA_SETUP(5, "default_auto");
TESTS/mbedmicro-mbed/static_assert/main.cpp:    GREENTEA_SETUP(5, "default_auto");
TESTS/mbedmicro-rtos-mbed/basic/main.cpp:    GREENTEA_SETUP(test_timeout, "timing_drift_auto");
TESTS/mbedmicro-rtos-mbed/mail/main.cpp:    GREENTEA_SETUP(10, "default_auto");
TESTS/mbedmicro-rtos-mbed/mutex/main.cpp:    GREENTEA_SETUP(10, "default_auto");
TESTS/mbedmicro-rtos-mbed/semaphore/main.cpp:    GREENTEA_SETUP(10, "default_auto");
TESTS/mbedmicro-rtos-mbed/queue/main.cpp:    GREENTEA_SETUP(5, "default_auto");
TESTS/mbedmicro-rtos-mbed/signals/main.cpp:    GREENTEA_SETUP(10, "default_auto");
TESTS/mbedmicro-rtos-mbed/malloc/main.cpp:    GREENTEA_SETUP(test_timeout, "timing_drift_auto");
TESTS/mbedmicro-rtos-mbed/threads/main.cpp:    GREENTEA_SETUP(20, "default_auto");
TESTS/mbedmicro-rtos-mbed/event_flags/main.cpp:    GREENTEA_SETUP(10, "default_auto");
TESTS/mbedmicro-rtos-mbed/MemoryPool/main.cpp:    GREENTEA_SETUP(20, "default_auto");
TESTS/events/queue/main.cpp:    GREENTEA_SETUP(20, "default_auto");
TESTS/mbedmicro-rtos-mbed/condition_variable/main.cpp:    GREENTEA_SETUP(10, "default_auto");
features/frameworks/greentea-client/README.md:    GREENTEA_SETUP(5, "default_auto");
features/frameworks/greentea-client/README.md:    GREENTEA_SETUP(15, "default_auto");
TESTS/mbedmicro-rtos-mbed/heap_and_stack/main.cpp:    GREENTEA_SETUP(test_timeout, "default_auto");
TESTS/mbedmicro-rtos-mbed/rtostimer/main.cpp:    GREENTEA_SETUP(10, "default_auto");
TESTS/events/timing/main.cpp:    GREENTEA_SETUP((number_of_cases + 1)*TEST_EVENTS_TIMING_TIME / 1000, "default_auto");
TESTS/mbedmicro-rtos-mbed/systimer/main.cpp:    GREENTEA_SETUP(5, "default_auto");
TESTS/mbedmicro-rtos-mbed/kernel_tick_count/main.cpp:    GREENTEA_SETUP(10, "timing_drift_auto");
features/frameworks/utest/README.md:    GREENTEA_SETUP(20, "default_auto");
TESTS/mbed_functional/callback/main.cpp:    GREENTEA_SETUP(10, "default_auto");
TESTS/mbed_functional/callback_big/main.cpp:    GREENTEA_SETUP(10, "default_auto");
TESTS/mbed_functional/callback_small/main.cpp:    GREENTEA_SETUP(10, "default_auto");
TESTS/mbed_functional/functionpointer/main.cpp:    GREENTEA_SETUP(10, "default_auto");
TESTS/mbed_hal/lp_ticker/main.cpp:    GREENTEA_SETUP(20, "default_auto");
TESTS/mbed_hal/rtc_time/main.cpp:    GREENTEA_SETUP(20, "default_auto");
TESTS/mbed_hal/sleep_manager/main.cpp:    GREENTEA_SETUP(20, "default_auto");
TESTS/mbed_hal/sleep_manager_racecondition/main.cpp:    GREENTEA_SETUP(30, "default_auto");
TESTS/mbed_hal/critical_section/main.cpp:    GREENTEA_SETUP(10, "timing_drift_auto");
TESTS/mbed_hal/ticker/main.cpp:    GREENTEA_SETUP(60, "default_auto");
TESTS/mbed_hal/rtc_time_conv/main.cpp:    GREENTEA_SETUP(300, "rtc_calc_auto");
TESTS/mbedtls/selftest/main.cpp:    GREENTEA_SETUP(120, "default_auto");
TESTS/mbed_hal/common_tickers/main.cpp:    GREENTEA_SETUP(80, "default_auto");
TESTS/mbedtls/multi/main.cpp:    GREENTEA_SETUP(10, "default_auto");
TESTS/mbed_hal/common_tickers_freq/main.cpp:    GREENTEA_SETUP(120, "timing_drift_auto");
TESTS/mbed_hal/crc/main.cpp:    GREENTEA_SETUP(30, "default_auto");
TESTS/mbed_hal/qspi/main.cpp:    GREENTEA_SETUP(180, "default_auto");
TESTS/mbed_hal/rtc/main.cpp:    GREENTEA_SETUP(60, "default_auto");
TESTS/mbed_hal/rtc_reset/main.cpp:    GREENTEA_SETUP(100, "rtc_reset");
TESTS/network/wifi/main.cpp:    GREENTEA_SETUP(240, "default_auto");
TESTS/mbed_hal/sleep/main.cpp:    GREENTEA_SETUP(60, "default_auto");
TESTS/network/interface/main.cpp:    GREENTEA_SETUP(480, "default_auto");
TESTS/mbed_hal/us_ticker/main.cpp:    GREENTEA_SETUP(20, "default_auto");
TESTS/mbed_hal/trng/main.cpp:    GREENTEA_SETUP(100, "trng_reset");
TESTS/network/emac/main.cpp:    GREENTEA_SETUP(1200, "default_auto");
TESTS/mbed_platform/FileHandle/main.cpp:    GREENTEA_SETUP(10, "default_auto");
TESTS/mbed_platform/SingletonPtr/main.cpp:    GREENTEA_SETUP(10, "default_auto");
TESTS/mbed_platform/Transaction/main.cpp:    GREENTEA_SETUP(10, "default_auto");
TESTS/mbed_platform/critical_section/main.cpp:    GREENTEA_SETUP(10, "default_auto");
TESTS/mbed_platform/stats_heap/main.cpp:    GREENTEA_SETUP(20, "default_auto");
TESTS/mbed_platform/system_reset/main.cpp:    GREENTEA_SETUP(2, "system_reset");
TESTS/lorawan/loraradio/main.cpp:    GREENTEA_SETUP(20, "default_auto");
TESTS/mbed_platform/stats_thread/main.cpp:    GREENTEA_SETUP(20, "default_auto");
TESTS/mbed_platform/SharedPtr/main.cpp:    GREENTEA_SETUP(10, "default_auto");
TESTS/mbed_platform/stats_cpu/main.cpp:    GREENTEA_SETUP(20, "default_auto");
TESTS/mbed_platform/error_handling/main.cpp:    GREENTEA_SETUP(120, "default_auto");
TESTS/mbed_platform/stats_sys/main.cpp:    GREENTEA_SETUP(20, "default_auto");
TESTS/mbed_platform/CircularBuffer/main.cpp:    GREENTEA_SETUP(15, "default_auto");
TESTS/netsocket/dns/main.cpp:    GREENTEA_SETUP(120, "default_auto");
TESTS/netsocket/tcp/main.cpp:    GREENTEA_SETUP(tcp_global::TESTS_TIMEOUT, "default_auto");
TESTS/netsocket/udp/main.cpp:    GREENTEA_SETUP(480, "default_auto");
TESTS/nfc/eeprom/main.cpp:    GREENTEA_SETUP(10, "default_auto");
features/frameworks/greentea-client/greentea-client/test_env.h:void GREENTEA_SETUP_UUID(const int timeout, const char *host_test_name, char *buffer, size_t size);
features/frameworks/greentea-client/greentea-client/test_env.h:void GREENTEA_SETUP(const int timeout, const char * host_test);
features/frameworks/greentea-client/source/greentea_test_env.cpp: *           GREENTEA_SETUP and GREENTEA_SETUP_UUID.
features/frameworks/greentea-client/source/greentea_test_env.cpp:void _GREENTEA_SETUP_COMMON(const int timeout, const char *host_test_name, char *buffer, size_t size) {
features/frameworks/greentea-client/source/greentea_test_env.cpp:extern "C" void GREENTEA_SETUP(const int timeout, const char *host_test_name) {
features/frameworks/greentea-client/source/greentea_test_env.cpp:    _GREENTEA_SETUP_COMMON(timeout, host_test_name, _value, GREENTEA_UUID_LENGTH);
features/frameworks/greentea-client/source/greentea_test_env.cpp: *           This function differs from GREENTEA_SETUP because it allows you to
features/frameworks/greentea-client/source/greentea_test_env.cpp:void GREENTEA_SETUP_UUID(const int timeout, const char *host_test_name, char *buffer, size_t size) {
features/frameworks/greentea-client/source/greentea_test_env.cpp:    _GREENTEA_SETUP_COMMON(timeout, host_test_name, buffer, size);
features/frameworks/utest/utest/utest_default_handlers.h:    ///    GREENTEA_SETUP(5, "default_auto");
TESTS/mbed_hal/flash/functional_tests/main.cpp:    GREENTEA_SETUP(20, "default_auto");
features/frameworks/utest/source/utest_greentea_handlers.cpp:    GREENTEA_SETUP(UTEST_DEFAULT_GREENTEA_TIMEOUT, UTEST_DEFAULT_HOST_TEST_NAME);
features/frameworks/utest/TESTS/readme/main_cpp_template:    GREENTEA_SETUP(20, "default_auto");
features/storage/TESTS/blockdevice/buffered_block_device/main.cpp:    GREENTEA_SETUP(30, "default_auto");
features/storage/TESTS/blockdevice/flashsim_block_device/main.cpp:    GREENTEA_SETUP(30, "default_auto");
features/storage/TESTS/filesystem/fat_filesystem/main.cpp:    GREENTEA_SETUP(10, "default_auto");
features/storage/TESTS/blockdevice/heap_block_device/main.cpp:    GREENTEA_SETUP(30, "default_auto");
features/storage/TESTS/filesystem/general_filesystem/main.cpp:    GREENTEA_SETUP(3000, "default_auto");
features/storage/TESTS/blockdevice/mbr_block_device/main.cpp:    GREENTEA_SETUP(10, "default_auto");
features/storage/TESTS/blockdevice/util_block_device/main.cpp:    GREENTEA_SETUP(10, "default_auto");
features/storage/TESTS/filesystem/multipart_fat_filesystem/main.cpp:    GREENTEA_SETUP(10, "default_auto");
features/device_key/TESTS/device_key/functionality/main.cpp:    GREENTEA_SETUP(45, "default_auto");
features/cellular/TESTS/api/cellular_device/main.cpp:    GREENTEA_SETUP(10 * 60, "default_auto");
features/cellular/TESTS/socket/udp/main.cpp:    GREENTEA_SETUP(10 * 60, "default_auto"); // network registration may take up to 180 seconds, DNS query a couple of minutes, etc.
features/cellular/TESTS/api/cellular_sms/main.cpp:    GREENTEA_SETUP(600, "default_auto");
features/cellular/TESTS/api/cellular_sim/main.cpp:    GREENTEA_SETUP(10 * 60, "default_auto");
features/cellular/TESTS/api/cellular_network/main.cpp:    GREENTEA_SETUP(10 * 60, "default_auto");
features/cellular/TESTS/api/cellular_power/main.cpp:    GREENTEA_SETUP(10 * 60, "default_auto");
features/cellular/TESTS/api/cellular_information/main.cpp:    GREENTEA_SETUP(10 * 60, "default_auto");
features/frameworks/utest/TESTS/unit_tests/basic_test/main.cpp:    GREENTEA_SETUP(20, "default_auto");
features/frameworks/utest/TESTS/unit_tests/case_setup_failure/main.cpp:    GREENTEA_SETUP(15, "default_auto");
features/frameworks/utest/TESTS/unit_tests/case_control_async/main.cpp:    GREENTEA_SETUP(15, "default_auto");
features/frameworks/utest/TESTS/unit_tests/case_control_repeat/main.cpp:    GREENTEA_SETUP(15, "default_auto");
features/frameworks/utest/TESTS/unit_tests/minimal_async_scheduler/main.cpp:    GREENTEA_SETUP(15, "default_auto");
features/frameworks/utest/TESTS/unit_tests/control_type/main.cpp:    GREENTEA_SETUP(15, "default_auto");
features/frameworks/utest/TESTS/unit_tests/minimal_scheduler/main.cpp:    GREENTEA_SETUP(15, "default_auto");
features/frameworks/utest/TESTS/unit_tests/test_assertion_failure_test_setup/main.cpp:    GREENTEA_SETUP(5, "default_auto");
features/frameworks/utest/TESTS/unit_tests/test_setup_case_selection_failure/main.cpp:    GREENTEA_SETUP(5, "default_auto");
features/frameworks/utest/TESTS/unit_tests/test_skip/main.cpp:    GREENTEA_SETUP(15, "default_auto");
features/frameworks/utest/TESTS/unit_tests/case_async_validate/main.cpp:    GREENTEA_SETUP(15, "default_auto");
features/frameworks/utest/TESTS/unit_tests/test_setup_failure/main.cpp:    GREENTEA_SETUP(5, "default_auto");
features/frameworks/utest/TESTS/unit_tests/case_teardown_failure/main.cpp:    GREENTEA_SETUP(15, "default_auto");
features/storage/FEATURE_STORAGE/TESTS/storage-volume-manager/basicAPI/basicAPI.cpp:    GREENTEA_SETUP(30, "default_auto");
features/frameworks/utest/TESTS/unit_tests/case_selection/main.cpp:    GREENTEA_SETUP(5, "default_auto");
features/storage/FEATURE_STORAGE/TESTS/cfstore/dump/dump.cpp:    GREENTEA_SETUP(400, "default_auto");
features/storage/FEATURE_STORAGE/TESTS/cfstore/close/close.cpp:    GREENTEA_SETUP(100, "default_auto");
features/storage/FEATURE_STORAGE/TESTS/cfstore/create/create.cpp:    GREENTEA_SETUP(CFSTORE_CREATE_GREENTEA_TIMEOUT_S, "default_auto");
features/storage/FEATURE_STORAGE/TESTS/flash_journal/basicAPI/basicAPI.cpp:    GREENTEA_SETUP(60, "default_auto");
features/storage/FEATURE_STORAGE/TESTS/cfstore/example2/example2.cpp:    GREENTEA_SETUP(400, "default_auto");
features/storage/FEATURE_STORAGE/TESTS/cfstore/example3/example3.cpp:    GREENTEA_SETUP(100, "default_auto");
features/storage/FEATURE_STORAGE/TESTS/cfstore/example1/example1.cpp:    GREENTEA_SETUP(100, "default_auto");
features/storage/FEATURE_STORAGE/TESTS/cfstore/example1/example1.cpp:    GREENTEA_SETUP(25, "default_auto");
features/storage/FEATURE_STORAGE/TESTS/cfstore/example4/example4.cpp:    GREENTEA_SETUP(400, "default_auto");
features/storage/FEATURE_STORAGE/TESTS/cfstore/example5/example5.cpp:    GREENTEA_SETUP(100, "default_auto");
features/storage/FEATURE_STORAGE/TESTS/cfstore/find2/find2.cpp:    GREENTEA_SETUP(CFSTORE_FIND2_GREENTEA_TIMEOUT_S, "default_auto");
features/storage/FEATURE_STORAGE/TESTS/cfstore/find/find.cpp:    GREENTEA_SETUP(400, "default_auto");
features/storage/FEATURE_STORAGE/TESTS/cfstore/flash_set/flash_set.cpp:    GREENTEA_SETUP(100, "default_auto");
features/storage/FEATURE_STORAGE/TESTS/cfstore/flash/flash.cpp:    GREENTEA_SETUP(100, "default_auto");
features/storage/FEATURE_STORAGE/TESTS/cfstore/add_del/add_del.cpp:    GREENTEA_SETUP(300, "default_auto");
features/storage/FEATURE_STORAGE/TESTS/cfstore/init/init.cpp:    GREENTEA_SETUP(100, "default_auto");
features/storage/FEATURE_STORAGE/TESTS/cfstore/flush/flush.cpp:    GREENTEA_SETUP(100, "default_auto");
features/storage/FEATURE_STORAGE/TESTS/cfstore/flush/flush.cpp:    GREENTEA_SETUP(CFSTORE_FLUSH_GREENTEA_TIMEOUT_S, "default_auto");
features/storage/FEATURE_STORAGE/TESTS/cfstore/misc/misc.cpp:    GREENTEA_SETUP(200, "default_auto");
features/storage/FEATURE_STORAGE/TESTS/cfstore/write/write.cpp:    GREENTEA_SETUP(200, "default_auto");
features/storage/FEATURE_STORAGE/TESTS/cfstore/read/read.cpp:    GREENTEA_SETUP(200, "default_auto");
features/storage/FEATURE_STORAGE/TESTS/cfstore/flush3/flush3.cpp:    GREENTEA_SETUP(CFSTORE_FLUSH3_GREENTEA_TIMEOUT_S, "default_auto");
features/storage/FEATURE_STORAGE/TESTS/cfstore/flush2/flush2.cpp:    GREENTEA_SETUP(100, "default_auto");
features/storage/nvstore/TESTS/nvstore/functionality/main.cpp:    GREENTEA_SETUP(120, "default_auto");
features/storage/filesystem/littlefs/TESTS/util/template_wrapper.fmt:    GREENTEA_SETUP(MBED_TEST_TIMEOUT, "default_auto");
features/storage/FEATURE_STORAGE/TESTS/cfstore/open/open.cpp:    GREENTEA_SETUP(CFSTORE_OPEN_GREENTEA_TIMEOUT_S, "default_auto");
targets/TARGET_Silicon_Labs/TARGET_EFM32/TESTS/spi/basic-spi/main.cpp:    GREENTEA_SETUP(25, "default_auto");
features/FEATURE_BLE/targets/TARGET_CORDIO/TESTS/cordio_hci/driver/main.cpp:    GREENTEA_SETUP(15, "default_auto");
features/storage/filesystem/littlefs/TESTS/filesystem_integration/format/main.cpp:    GREENTEA_SETUP(MBED_TEST_TIMEOUT, "default_auto");
features/storage/filesystem/littlefs/TESTS/filesystem_recovery/resilience/main.cpp:    GREENTEA_SETUP(MBED_TEST_TIMEOUT, "default_auto");
features/storage/filesystem/littlefs/TESTS/filesystem_recovery/resilience_functional/main.cpp:    GREENTEA_SETUP(MBED_TEST_TIMEOUT, "unexpected_reset");
features/storage/filesystem/littlefs/TESTS/filesystem_recovery/wear_leveling/main.cpp:    GREENTEA_SETUP(MBED_TEST_TIMEOUT, "default_auto");
features/storage/filesystem/littlefs/TESTS/filesystem/seek/main.cpp:    GREENTEA_SETUP(MBED_TEST_TIMEOUT, "default_auto");
features/storage/filesystem/littlefs/TESTS/filesystem_retarget/files/main.cpp:    GREENTEA_SETUP(MBED_TEST_TIMEOUT, "default_auto");
features/storage/filesystem/littlefs/TESTS/filesystem_retarget/seek/main.cpp:    GREENTEA_SETUP(MBED_TEST_TIMEOUT, "default_auto");
features/storage/filesystem/littlefs/TESTS/filesystem_retarget/interspersed/main.cpp:    GREENTEA_SETUP(MBED_TEST_TIMEOUT, "default_auto");
features/storage/filesystem/littlefs/TESTS/filesystem/files/main.cpp:    GREENTEA_SETUP(MBED_TEST_TIMEOUT, "default_auto");
features/netsocket/cellular/generic_modem_driver/TESTS/unit_tests/default/main.cpp:    GREENTEA_SETUP(600, "default_auto");
features/storage/filesystem/littlefs/TESTS/filesystem/dirs/main.cpp:    GREENTEA_SETUP(MBED_TEST_TIMEOUT, "default_auto");
features/FEATURE_BLE/targets/TARGET_CORDIO/TESTS/cordio_hci/transport/main.cpp:    GREENTEA_SETUP(15, "default_auto");
components/storage/blockdevice/COMPONENT_SD/TESTS/filesystem/dirs/main.cpp:    GREENTEA_SETUP(MBED_TEST_TIMEOUT, "default_auto");
components/storage/blockdevice/COMPONENT_SD/TESTS/filesystem/files/main.cpp:    GREENTEA_SETUP(MBED_TEST_TIMEOUT, "default_auto");
features/storage/filesystem/littlefs/TESTS/filesystem_retarget/dirs/main.cpp:    GREENTEA_SETUP(MBED_TEST_TIMEOUT, "default_auto");
components/storage/blockdevice/COMPONENT_SPIF/TESTS/block_device/spif/main.cpp:    GREENTEA_SETUP(60, "default_auto");
features/storage/filesystem/littlefs/TESTS/filesystem/interspersed/main.cpp:    GREENTEA_SETUP(MBED_TEST_TIMEOUT, "default_auto");
components/storage/blockdevice/COMPONENT_SD/TESTS/filesystem/fopen/fopen.cpp:    GREENTEA_SETUP(FSFAT_FOPEN_GREENTEA_TIMEOUT_S, "default_auto");
components/storage/blockdevice/COMPONENT_FLASHIAP/TESTS/filesystem/fopen/fopen.cpp:    GREENTEA_SETUP(FSLITTLE_FOPEN_GREENTEA_TIMEOUT_S, "default_auto");
components/storage/blockdevice/COMPONENT_SD/TESTS/filesystem/seek/main.cpp:    GREENTEA_SETUP(MBED_TEST_TIMEOUT, "default_auto");
components/storage/blockdevice/COMPONENT_SD/TESTS/filesystem/parallel/main.cpp:    GREENTEA_SETUP(MBED_TEST_TIMEOUT, "default_auto");

None invoke it with "RTCTest", and all use the "name" column name.

theotherjimmy commented 6 years ago

I think run and list use the same registry, the global mbed_host_tests.HOSTREGISTRY.

Alex-EEE commented 6 years ago

Ok, so RTCTest was just a guess, I couldn't remember from my output exactly which host tests it was looking for, just that it couldn't find them anymore.

Right now my observation is that if you supply -e, then --list will include all tests in mbed_host_tests.HOSTREGISTRY in addition to tests supplied with -e. But run will not...for running, it appears -e overrides mbed_host_tests.HOSTREGISTRY

If you can add a custom host test somehow in a way I haven't tried, and you can get greentea to pick it up, let me know what you did and by all means I'll do it that way. But every different way I've tried using -e, I lose the tests inside mbed_host_tests.HOSTREGISTRY

theotherjimmy commented 6 years ago

@Alex-EEE I can find nothing in the code that would make the new tests override old ones, nor could I find any use of the unregester_host_test function, the only way to remove host tests.

It looks like the only way that host tests are looked up is through get_host_test..

I think this is a windows-only issue: HOSTREGISTRY may be empty/cleared/inconsistent on the new Process's side of things. I ran into this sort of thing before (in particular https://github.com/ARMmbed/mbed-os/pull/7477). We should probably not have a global var anyway.

Alex-EEE commented 6 years ago

Ahhhh, the windows thing about HOSTREGISTRY makes a lot of sense. https://github.com/ARMmbed/htrun/blob/63eea4ee94a88f8e5c292588889fc243b5177aee/mbed_host_tests/__init__.py#L166 does appear to just add tests, and I remember when I saw that, I was confused why the original tests were going away

theotherjimmy commented 6 years ago

Well this means that the registry needs to not be a global, and that those "helper functions" need to go away.

adbridge commented 6 years ago

Internal Jira reference: https://jira.arm.com/browse/IOTCORE-493