Closed mike-scott closed 5 years ago
I could add @andyross here as he was looking into failures for real HW, but this is more of a configuration question: Why are we white-listing platforms for tests that require specific HW?
checking for HW that isn't installed on a plain nrf52840_pca10056.
(Emphasis mine)
There are limits to how fine-grained test suite configuration can become, otherwise the combinatorial explosion becomes unmanageable. It's always a trade-off. Especially so in the upstream project used by all products across all companies. Downstream validation projects specific to some products can be more fine-grained and it should be possible and easy for companies to fine tune test suite configurations to suite their particular needs.
samples/
are great but maybe the automation expectations for them are a bit lower than for tests/
. Otherwise they would/should be in tests/
. You can exclude samples
like this: sanitycheck -T $ZEPHYR_BASE/tests/
Keeping the above in mind I can see two possible (and fine-grained!) ways to solve this particular problem:
nrf52840_pca10056_with_led
and adjust the tests accordingly.sanitycheck [--exclude=led | --tag led]
@lemrey could you pick this up, and investigate?
@nashif is there an acceptable workaround using sanitycheck to skip unconnected devices when doing device testing?
I am removing myself, I won't be able to work on this anytime soon.
@nashif can you comment on this one please?
Describe the bug When running sanitycheck on nrf52840_pca10056, 2 items will fail unless you have a specific rig setup (these are both whitelisted for nrf52840_pca10056):
These samples are checking for HW that isn't installed on a plain nrf52840_pca10056.
To Reproduce Steps to reproduce the behavior:
sanitycheck --platform nrf52840_pca10056 --device-testing --verbose --inline-logs --enable-slow --device-serial /dev/ttyACM0
Expected behavior We should be able to run sanitycheck on real HW and have it pass.
Impact This means CI systems need to watch for and eliminate these very conditional tests.
Screenshots or console output
and
Environment (please complete the following information):