search

ARMmbed / htrun

MOVED: https://github.com/ARMmbed/mbed-os-tools (Flash, reset and run host supervised tests on mbed platforms)
8 stars 37 forks source link

Add option to request allocation based on specified tags #163

Closed kegilbert closed 7 years ago

kegilbert commented 7 years ago

In the process of integrating additional tests to the CI we realized it would be helpful to be able to tag particular devices in RaaS that had unique or additional setups for tests that required them (for example a K64F with an ESP8266 or Ethernet).

Usage

Take the following tag as an example.

image

There is a separate PR for changes to greentea to pipe the new argument to this level. Given that greentea change and the above tag setup, we could run tests with the following setup.

Test Setup

mbedgt -g "K64F:raas_client:austin-ci-linux-001:8000" --test-spec=BUILD/tests/K64F/GCC_ARM/test_spec.json --report-html=K64F_GCC_ARM.html -e tests/host_tests/timing_drift_auto.py -v -V -n features-feature_lwip-tests-mbedmicro-net-tcp_hello_world --allocate-on-tag Ethernet

Multiple Tags

Comma separated lists can also be supplied if multiple tags are required (device resource must match all of the supplied tags, it cannot be used as an 'or' list).

.... --allocate-on-tag Ethernet,ESP8266,...
kegilbert commented 7 years ago

cc @studavekar

kegilbert commented 7 years ago

@jupe Updated, I'll squash when the review is finished.

jupe commented 7 years ago

By the way - this approach will use same tags as requirements for all tests that is going to be executed. Is that the way you really want ? It's going to be a bit inefficient (from utilization point of view) in case that requirement is not really needed for all cases. Eg if we are going to run 100 tests, but only first test require tag X we are still using same device for rest of the tests even there could be "more cheaper" device in the pool.. This will be problematic in case that we have very little amount of devices with tag X than others and it is needed also for other test jobs...

kegilbert commented 7 years ago

Sorry for the delay, the current plan is to generate a list of which tests require what tags based on the test binary dependencies and pass that along to htrun from the Greentea level. This would tune each greentea run to filter out device tags on a per-test/device level rather than passing it to an entire block of tests at the htrun layer.

theotherjimmy commented 7 years ago

@kegilbert Is the aforementioned plan implemented in this PR? Is this ready for review?

kegilbert commented 7 years ago

@theotherjimmy: @studavekar and I are still tinkering with the tag list generation implementation, updated what I had up but we still need to update a few things.

theotherjimmy commented 7 years ago

@kegilbert Squashed onto master.