Peripheral Test Unification

[hakehuang]

Introduction

A proposal to unify the test apporach to zephyr peripheral devices. By using Arduio device and Click board resources, which is off-shief support, to design our test cases.

Problem description

Currently, Zephyr Peripheral testing(GPIO/I2C/SPI) is based on loopback/self instance communication/ on board devices, which is board by board settled. this have below potential issue:

1. can not prove tests on real devices in mass usage. or it is hard to expand tests to new devices
2. test is not raliable, false positive will happen
3. Need extra test design efforts just for test purpose
4. it is hard to reproduce real divece issues, and real user case is not coverred by tests
5. self loopbacks can not cover compatible issues

Proposed change

1. keep on board internal loopback cases in P0 cases

2. allow new test harness/fixtures sets as below:

interface (ARD mean ardino interfaces) plus peripherals (GPIO_I2C/SPI/I3C) plush device name (e.g FRAM_32K)

e.g.

ARD_I2C_<device_name>
ARD_SPI_<device_name>
ARD_GPIO_<device_name>
CLICK_I2C_<device_name>
CLICK_SPI_<device_name>
CLICK_GPIO_<device_name>

uodate dts supporting with alias as below, inherient from https://github.com/zephyrproject-rtos/zephyr/issues/20340

zephyr,<test_interface>-<DEV>-<INSTANCE>

such as we will have

zephyr,onboard-led-0

and also have

zephyr,arduino-led-0

3. maintain a standard board list in zephyr peripher list as P1 cases, and allow community to add support for more device in P3 lists

Detailed RFC

consider to create a list from below market, or any community available and affordable shop Using Arduino compatible interfaces https://www.adafruit.com/search?q=fram Using Click compatible interfaces https://www.mikroe.com/shop

Dependencies

No.

Concerns and Unresolved Questions

need agreement on this peripheral case design approach. and testcases need to refactor to support multiply similar devices by device tree

Alternatives

Keep current way

[henrikbrixandersen]

• Phase out below cases: on board internal loopback cases.

I don't think loopback test should be phased out. They provide means for testing said peripheral bus controllers for people who do not have access to the relevant shield used for other test cases.

For CAN controllers, loopback testing is an integrated part of almost all CAN peripherals.

[nordicjm]

I don't see the value in replacing a simple easy to use test that anyone can conduct by using their board and some wires, with them suddenly needing a whole other device which has its own programming interface then having to wire the boards up. And an arduino is very limited, how is it going to e.g. to MIPI? Now suddenly it's an ardunino and a £12,000 logic analyser that would be needed?

[teburd]

Loop back testing is valuable when possible for sure. I don't think we should get rid of it.

But it doesn't always work out that its possible (i2c). It also doesn't test out the full peripheral usage of talking to a device like an end user of Zephyr will be doing on their own custom boards. Meaning chip select, pin configuration for drive mode, pull up/down, etc are effectively untested.

I just found a bug the other day in testing i2c on just 3 boards in a pinctrl configuration. I very much doubt there's no more bugs to be found.

SPI and UART surely are fine with loopback testing as a minimum, but with SPI I'd note that for example on the sam e70 spi controller CS would misbehave when using gpio CS. That would be unnoticed with simple loopback testing. An external device to communicate to is needed.

Confused by the programming interface and 12k logic analyzer comment here.

[nordicjm]

Confused by the programming interface and 12k logic analyzer comment here.

These tests run through twister, so now you need to control another board which is not the board under test which means you need to flash some firmware for it using some interface and somehow also control it. The 12k comment is about an arduino being a very limited piece of hardware, how is it going to test e.g. i3c when it does not have i3c support, so now you're going to need yet another piece of hardware that isn't an arduino to do anything i3c testing, and the list goes on...

[hakehuang]

For CAN controllers, loopback testing is an integrated part of almost all CAN peripherals.

Can is not in this proposal yet, so Can loopback is kept.

[hakehuang]

I don't think loopback test should be phased out. They provide means for testing said peripheral bus controllers for people who do not have access to the relevant shield used for other test cases.

it is why I recommand that we use some affordable and off-shief boards.

[hakehuang]

And an arduino is very limited, how is it going to e.g. to MIPI? Now suddenly it's an ardunino and a £12,000 logic analyser that would be needed?

MIPI is not in this proposal, now limited to SPI/I2C/GPIO/I3C

[hakehuang]

These tests run through twister, so now you need to control another board which is not the board under test which means you need to flash some firmware for it using some interface and somehow also control it. The 12k comment is about an arduino being a very limited piece of hardware, how is it going to test e.g. i3c when it does not have i3c support, so now you're going to need yet another piece of hardware that isn't an arduino to do anything i3c testing, and the list goes on...

pleaes check below web site for devices to recommand Using Arduino compatible interfaces https://www.adafruit.com/search?q=fram Using Click compatible interfaces https://www.mikroe.com/shop

and I3C can be use as I2C as well, it is not good enough, but I3C is mainly used for camera. but we have user cases to use i3c bus as I2c.

[nordicjm]

These tests run through twister, so now you need to control another board which is not the board under test which means you need to flash some firmware for it using some interface and somehow also control it. The 12k comment is about an arduino being a very limited piece of hardware, how is it going to test e.g. i3c when it does not have i3c support, so now you're going to need yet another piece of hardware that isn't an arduino to do anything i3c testing, and the list goes on...

pleaes check below web site for devices to recommand Using Arduino compatible interfaces https://www.adafruit.com/search?q=fram Using Click compatible interfaces https://www.mikroe.com/shop

Right I think I've misunderstood you, by ardiuno I thought you meant actual arduino which has a firmware emulating e.g. i2c communication, not peripherals with these interfaces which are arduino sheilds, can ignore my comment from above. Though I agree with Brix in that I am against removing/preventing loopback testing, but adding testing with actual peripherals is not a bad thing

[hakehuang]

Loop back testing is valuable when possible for sure. I don't think we should get rid of it.

Yes, it can be kept for old platforms. Just want to recommand to use real devices for new platforms

[hakehuang]

Though I agree with Brix in that I am against removing/preventing loopback testing, but adding testing with actual peripherals is not a bad thing

Let me update the proposal, we still support loopback as P0 cases.

[teburd]

These tests run through twister, so now you need to control another board which is not the board under test which means you need to flash some firmware for it using some interface and somehow also control it. The 12k comment is about an arduino being a very limited piece of hardware, how is it going to test e.g. i3c when it does not have i3c support, so now you're going to need yet another piece of hardware that isn't an arduino to do anything i3c testing, and the list goes on...

pleaes check below web site for devices to recommand Using Arduino compatible interfaces https://www.adafruit.com/search?q=fram Using Click compatible interfaces https://www.mikroe.com/shop

Right I think I've misunderstood you, by ardiuno I thought you meant peripherals with these interfaces which are arduino sheilds, not an actual arduino which has a firmware emulating e.g. i2c communication, can ignore my comment from above. Though I agree with Brix in that I am against removing/preventing loopback testing, but adding testing with actual peripherals is not a bad thing

Glad we're all on the same page now :+1:

[str4t0m]

It could also make sense to use a customized CI shield, that ticks all our requirements in a single board. Something like https://github.com/teslabs/zephyr-ci-shield from @gmarull.

[teburd]

I'd note a second benefit of clicks is that... there's many many sensor boards, which would allow for potentially setting up a hw pool of sensor testing, which I have as sort of a back burner goal I'd like to work towards.

[jfischer-no]

There was also this https://github.com/zephyrproject-rtos/zephyr/issues/20642

[hakehuang]

It could also make sense to use a customized CI shield, that ticks all our requirements in a single board. Something like https://github.com/teslabs/zephyr-ci-shield from @gmarull.

the problem is how community can get this board? it is better to be off-shielf, otherwise the cycle time maybe too long.

[jfischer-no]

It could also make sense to use a customized CI shield, that ticks all our requirements in a single board. Something like https://github.com/teslabs/zephyr-ci-shield from @gmarull.

the problem is how community can get this board? it is better to be off-shielf, otherwise the cycle time maybe too long.

https://www.zephyrproject.org/store/ :smile:

[hakehuang]

https://www.zephyrproject.org/store/ 😄

I did not see the board, is there a direct link?

[jfischer-no]

https://www.zephyrproject.org/store/ 😄

I did not see the board, is there a direct link?

That was a joke. But it might be a solution.