search

zephyrproject-rtos / zephyr

Primary Git Repository for the Zephyr Project. Zephyr is a new generation, scalable, optimized, secure RTOS for multiple hardware architectures.
https://docs.zephyrproject.org
Apache License 2.0
10.75k stars 6.56k forks source link

sanitycheck: nrf52840-pca10056 (dev kit) picks up sample/drivers items which will fail due to missing HW #17439

Closed mike-scott closed 5 years ago

mike-scott commented 5 years ago

Describe the bug When running sanitycheck on nrf52840_pca10056, 2 items will fail unless you have a specific rig setup (these are both whitelisted for nrf52840_pca10056):

These samples are checking for HW that isn't installed on a plain nrf52840_pca10056.

To Reproduce Steps to reproduce the behavior:

  1. sanitycheck --platform nrf52840_pca10056 --device-testing --verbose --inline-logs --enable-slow --device-serial /dev/ttyACM0
  2. See error

Expected behavior We should be able to run sanitycheck on real HW and have it pass.

Impact This means CI systems need to watch for and eliminate these very conditional tests.

Screenshots or console output

 14/237 nrf52840_pca10056         samples/drivers/ht16k33/sample.driver.ht16k33      FAILED: timeout (device)
../../../../../out/build/nrf52840_pca10056/samples/drivers/ht16k33/sample.driver.ht16k33/handler.log
***** Booting Zephyr OS build zephyr-v1.14.0-2209-g64d58901d3ed *****
[00:00:00.000,091] <err> i2c_nrfx_twi: Error 195952641 occurred for message 0
[00:00:00.000,122] <err> ht16k33: Enabling HT16K33 system oscillator failed (err -5)
[00:00:00.000,122] <err> gpio_ht16k33: HT16K33 parent device 'HT16K33' not found
[00:00:00.000,152] <err> gpio_ht16k33: HT16K33 parent device 'HT16K33' not found
[00:00:00.000,152] <err> gpio_ht16k33: HT16K33 parent device 'HT16K33' not found
[00:00:00.006,378] <err> main: LED device HT16K33 not found
***** Booting Zephyr OS build zephyr-v1.14.0-2209-g64d58901d3ed *****
[00:00:00.000,000] <err> i2c_nrfx_twi: Error 195952641 occurred for message 0
[00:00:00.000,000] <err> ht16k33: Enabling HT16K33 system oscillator failed (err -5)
[00:00:00.000,000] <err> gpio_ht16k33: HT16K33 parent device 'HT16K33' not found
[00:00:00.000,000] <err> gpio_ht16k33: HT16K33 parent device 'HT16K33' not found
[00:00:00.000,000] <err> gpio_ht16k33: HT16K33 parent device 'HT16K33' not found
[00:00:00.005,371] <err> main: LED device HT16K33 not found
../../../../../out/build/nrf52840_pca10056/samples/drivers/ht16k33/sample.driver.ht16k33/handler.log

and

 16/237 nrf52840_pca10056         samples/drivers/led_lp5562/sample.driver.led_lp5562 FAILED: timeout (device)
../../../../../out/build/nrf52840_pca10056/samples/drivers/led_lp5562/sample.driver.led_lp5562/handler.log
***** Booting Zephyr OS build zephyr-v1.14.0-2209-g64d58901d3ed *****
[00:00:00.000,000] <err> i2c_nrfx_twi: Error 195952641 occurred for message 0
[00:00:00.000,000] <err> lp5562: Enabling LP5562 LED chip failed.
[00:00:00.005,279] <err> main: LED device LP5562 not found
../../../../../out/build/nrf52840_pca10056/samples/drivers/led_lp5562/sample.driver.led_lp5562/handler.log

Environment (please complete the following information):

mike-scott commented 5 years ago

I could add @andyross here as he was looking into failures for real HW, but this is more of a configuration question: Why are we white-listing platforms for tests that require specific HW?

marc-hb commented 5 years ago

checking for HW that isn't installed on a plain nrf52840_pca10056.

(Emphasis mine)

There are limits to how fine-grained test suite configuration can become, otherwise the combinatorial explosion becomes unmanageable. It's always a trade-off. Especially so in the upstream project used by all products across all companies. Downstream validation projects specific to some products can be more fine-grained and it should be possible and easy for companies to fine tune test suite configurations to suite their particular needs.

samples/ are great but maybe the automation expectations for them are a bit lower than for tests/. Otherwise they would/should be in tests/. You can exclude samples like this: sanitycheck -T $ZEPHYR_BASE/tests/

Keeping the above in mind I can see two possible (and fine-grained!) ways to solve this particular problem:

ioannisg commented 5 years ago

@lemrey could you pick this up, and investigate?

carlescufi commented 5 years ago

@nashif is there an acceptable workaround using sanitycheck to skip unconnected devices when doing device testing?

lemrey commented 5 years ago

I am removing myself, I won't be able to work on this anytime soon.

carlescufi commented 5 years ago

@nashif can you comment on this one please?