Zephyr testing via emulators

[sjg20]

Introduction

Disclaimer: I am a beginner with Zephyr so please don't be offended if I have the wrong end of the stick in some areas.

Add a new emulator framework which allows integrated testing of a tree of interconnected drivers and the real-world logic that makes use of them. The initial use case is an Embedded Controller (e.g. as used on x86 laptops).

Problem description

Zephyr has a large array of tests. At present testing in Zephyr seems quite fragmented and much of it relies on real hardware (required setup) or Qemu (slow, limited functionality, harder to debug than on a host).

Zephyr has a native_posix board which is targeted at running Zephyr on a Linux host. It supports quite a lot of peripheral drivers (https://docs.zephyrproject.org/latest/boards/posix/native_posix/doc/index.html#peripherals). In terms of purpose, these are a bit of a mix. The Bluetooth driver connects to hardware on the host whereas the real-time-clock (RTC) driver simulates time itself (although it does start with the host time). For testing, it is helpful to isolate the test application from the host system running it. This would allows Bluetooth traffic to be simulated, for example. An NRF52-simulator provides something like this, but does not run on native_posix.

Most tests exist to cover a particular feature in isolation. For example, tests/drivers/i2c/i2c_api tests the I2C API if the board supports it, tests/drivers/flash_simulator tests the flash simulator, tests/drivers/gpio/gpio_basic_api tests the GPIO API, etc. These tests typically require a particular fixture (e.g. real hardware with things attached). The tests cover particular drivers to some extent, but without the hardware or qemu it isn't possible to test those drivers.

It is hard to test higher-level code that uses those drivers, such as something that receives data over a serial device and writes it to an EEPROM device. It is hard to test drivers that depend on others, such as an EEPROM driver that uses an I2C driver.

Configuration for tests is handled via Kconfig and devicetree fragments but the test code itself does not appear in the devicetree. Thus it is harder to discover and change the connections between device drivers and their emulators.

All of these points make it hard to simulate a full system on a host.

Proposed change

Add emulation drivers for all buses (e.g. I2C, SPI) which allows peripheral drivers (such as eeprom_at2x.c) to be used on a host and connected to a simulator for that peripheral. This allows testing of many peripheral drivers. Add emulation drivers for other subsystems that allow for control over the subsystem's state. For example, a GPIO driver (see https://github.com/zephyrproject-rtos/zephyr/pull/26484) allows an emulated GPIO to be toggled and have the system react Consider adding emulation for memory-mapped peripherals. This has less value in general (just use qemu!) but it can be helpful for things like PCI where we want to test the mapping and access code. Allow test stimulus to come from code compiled into the test (for easy development and debugging) or from outside (for better integration with larger test fixtures). Use devicetree to connect up the drivers and emulators, so it is easy to example and reconfigure a system, e..g to view or update I2C addresses and bus numbers. Continue to use native_posix for this, or if necessary create a new 'sandbox' target which runs on Linux but is purely for running emulation drivers

With the above it should be possible to implement an entire Embedded Controller with most of its drivers, excluding the chip-specific ones. Thus if you have a set of reliable drivers for a microcontroller (verified by device-specific tests that run on real hardware or qemu) it is likely that the Embedded Controller will run correctly on that hardware.

Detailed RFC

The diagram below shows application code / high-level tests at the top. This is the ultimate application we want to run.

Below that are peripheral drivers, such as the AT24 EEPROM driver. We can test peripheral drivers using an emulation driver connected via a native_posix I2C controller/emulator which passes I2C traffic from the AT24 driver to the AT24 simulator.

Separately, we can test the STM32 and NXP I2C drivers on real hardware using API tests. These require some sort of device attached to the bus, but with this, we can validate much of the driver functionality.

Putting the two together, we can test the application and peripheral code entirely on native_posix. Since we know that the I2C driver on the real hardware works, we should expect the application and peripheral drivers to work on the real hardware also.

Using the above framework we can test an entire application (e.g. Embedded Controller) on native_posix using emulators for all non-chip drivers:

The 'real' code is shown in green. The Zephyr emulation-framework code is shown in yellow. The blue boxes are the extra code we have to write to emulate the peripherals.

With this approach we can: Write individual tests for each driver (green), covering all failure modes, error conditions, etc. Ensure 100% test coverage for drivers (green) Write tests for combinations of drivers, such as GPIOs provided by an I2C GPIO expander driver talking over an I2C bus, with the GPIOs controlling a charger. All of this can work in the emulated environment or on real hardware. Write a complex application that ties together all of these pieces and runs on native_posix. We can develop on a host, use source-level debugging, etc. Transfer the application to any board which provides the required features (e.g. I2C, enough GPIOs), by adding Kconfig and devicetree fragments.

Proposed change (Detailed)

No more detail here.

An initial stab at this was created at https://github.com/zephyrproject-rtos/zephyr/pull/27300 but then modified considerably.

Dependencies

This may affect the existing uses of native_posix if they rely on real hardware being connected in some places. We may need to have two separate drivers in some cases, one for use with emulation and one for talking directly to the host hardware.

Concerns and Unresolved Questions

It is unclear how to handle cover coverage.

Alternatives

The main alternative is the status quo, as described in the introduction.

[aescolar]

Hi @sjg20 A couple of comments:

the real-time-clock (RTC) driver simulates time itself (although it does start with the host time).

The RTC driver actually is coupled to the host in real time mode (it can also emulate clock drift), or it can just be decoupled.

An NRF52-simulator provides something like this, but does not run on native_posix.

The nrf52_bsim uses the POSIX architecture, just like native_posix. So it runs as a native executable too, but with no link to host HW; only simulated HW on simulated time (with timed behaviour in the simulated HW)

Ensure 100% test coverage for drivers (green)

How are you envisioning emulating peripherals timing?

It is unclear how to handle cover coverage.

You meant code coverage? (if so that works today out of the box in POSIX arch targets: https://docs.zephyrproject.org/latest/guides/coverage.html#coverage-reports-using-the-posix-architecture)

[pabigot]

Thanks; this clarifies and corrects my understanding of the goals of #26003 and #27300. In particular I was not aware of how tightly the concept remains bound to native_posix (#25308). Why is this? Do you expect it to require capabilities that could only be provided in native posix, e.g. dynamically linking code or reading state or scripts or other data to support the emulation/simulation?

I acknowledge that the hardware emulation is most likely to be run either on native_posix or qemu, but it doesn't initially appear to have any dependency on environment.

Configuration for tests is handled via Kconfig and devicetree fragments but the test code itself does not appear in the devicetree. Thus it is harder to discover and change the connections between device drivers and their emulators.

I think this identifies a misunderstanding that explains some of the miscommunication in the other PRs. In fact, Zephyr uses devicetree only to describe hardware. The selection of which drivers are enabled, and options that control driver-implemented behavior, is (supposed to be) done through ~devicetree~ KConfig. This is a critical and useful feature as it means an application that needs (for example) a Sensirion SHT21 can re-use the hardware description while substituting a completely different driver that is more suited for its needs.

The sole connection between the devicetree and the driver is that if a driver is enabled via Kconfig it will use the compatible properties of devicetree nodes to identify the hardware for which it provides instances. This is demonstrated in the current tests/drivers/eeprom code where tests on hardware enable (e.g.) CONFIG_I2C=y and CONFIG_AT24=y which together enable CONFIG_AT2X=y which selects eeprom_at2x.c, while on native_posix the board definition provides CONFIG_EEPROM_SIMULATOR=y which instead selects eeprom_simulator.c. Thus on native_posix the AT2X driver is not what's being tested (making a test on native_posix rather pointless, IMO). Otherwise the devicetree EEPROM nodes for native_posix and a board/shield with an actual EEPROM could be identical (both could specify atmel,at24, though in fact currently native_posix uses zephyr,sim-eeprom).

I have a very strong interest in this emulation capability, though not for simulating an entire system. The ability to control the responses produced by I2C leader transactions, and/or GPIO behavior, allows full coverage testing of hardware drivers by forcing the driver to follow control paths that are very difficult to produce using real hardware (e.g. behavior when the temperature drops below 0 Cel or above 100 Cel). Given that, it becomes (theoretically) easy to run a standard hardware test on native_posix with a trivial overlay that replaces the board-specific compatible for I2C and GPIO with the emulation ones, and adds an external module that provides the control that uses the GPIO and device emulator API.

I outlined my proposed solution design in https://github.com/zephyrproject-rtos/zephyr/pull/27300#issuecomment-669212227 which I won't repeat here, except to highlight that it does not require significant devicetree changes relative to real hardware (just overriding the compatible for a couple nodes), and it also assumes that device emulation is likely to be highly tailored to a specific test.

The emulator for a properly-functioning sensor used to simulate a complete system could be quite different.

[sjg20]

Hi @sjg20 A couple of comments:

the real-time-clock (RTC) driver simulates time itself (although it does start with the host time).

The RTC driver actually is coupled to the host in real time mode (it can also emulate clock drift), or it can just be decoupled.

Sounds ideal.

An NRF52-simulator provides something like this, but does not run on native_posix.

The nrf52_bsim uses the POSIX architecture, just like native_posix. So it runs as a native executable too, but with no link to host HW; only simulated HW on simulated time (with timed behaviour in the simulated HW)

OK, that's good.

Ensure 100% test coverage for drivers (green)

How are you envisioning emulating peripherals timing?

Not seriously. That gets into the realm of simulation. You could tell the emulator to reply after a certain time I suppose. I think it has some benefits (e.g. to check that code is robust with different timing and corner cases), but it does complicate things. Perhaps this is something to consider if we see particular bugs that are timing-related.

It is unclear how to handle cover coverage.

You meant code coverage? (if so that works today out of the box in POSIX arch targets: https://docs.zephyrproject.org/latest/guides/coverage.html#coverage-reports-using-the-posix-architecture)

Yes code coverage. I mean that we want to verify code coverage for the code being tested, but not for the emulators, etc. Perhaps that is already handled as it should be just a simple filter based on filename. I haven't actually looked at it.

[sjg20]

Thanks; this clarifies and corrects my understanding of the goals of #26003 and #27300. In particular I was not aware of how tightly the concept remains bound to native_posix (#25308). Why is this? Do you expect it to require capabilities that could only be provided in native posix, e.g. dynamically linking code or reading state or scripts or other data to support the emulation/simulation?

No I'm not saying it can't run on real hardware, but I don't see any benefit to that. If you were running on real hardware, you might just use the real drivers on that board.

I acknowledge that the hardware emulation is most likely to be run either on native_posix or qemu, but it doesn't initially appear to have any dependency on environment.

Yes I can't think of any either.

Configuration for tests is handled via Kconfig and devicetree fragments but the test code itself does not appear in the devicetree. Thus it is harder to discover and change the connections between device drivers and their emulators.

I think this identifies a misunderstanding that explains some of the miscommunication in the other PRs. In fact, Zephyr uses devicetree only to describe hardware. The selection of which drivers are enabled, and options that control driver-implemented behavior, is (supposed to be) done through ~devicetree~ KConfig. This is a critical and useful feature as it means an application that needs (for example) a Sensirion SHT21 can re-use the hardware description while substituting a completely different driver that is more suited for its needs.

The sole connection between the devicetree and the driver is that if a driver is enabled via Kconfig it will use the compatible properties of devicetree nodes to identify the hardware for which it provides instances. This is demonstrated in the current tests/drivers/eeprom code where tests on hardware enable (e.g.) CONFIG_I2C=y and CONFIG_AT24=y which together enable CONFIG_AT2X=y which selects eeprom_at2x.c, while on native_posix the board definition provides CONFIG_EEPROM_SIMULATOR=y which instead selects eeprom_simulator.c. Thus on native_posix the AT2X driver is not what's being tested (making a test on native_posix rather pointless, IMO). Otherwise the devicetree EEPROM nodes for native_posix and a board/shield with an actual EEPROM could be identical (both could specify atmel,at24, though in fact currently native_posix uses zephyr,sim-eeprom).

With my pull request, native_posix does actually use the AT24 driver, unmodified. I feel that an EEPROM emulator is useful to emulate an EEPROM IP block in a microcontroller where you don't care how it works. But for an I2C EEPROM, we can use the actual driver and a paired emulator.

[..]

I outlined my proposed solution design in #27300 (comment) which I won't repeat here, except to highlight that it does not require significant devicetree changes relative to real hardware (just overriding the compatible for a couple nodes), and it also assumes that device emulation is likely to be highly tailored to a specific test.

Yes and I updated the pull request to work that way, so far as I can tell.

The emulator for a properly-functioning sensor used to simulate a complete system could be quite different.

Do you mean it would have a different implementation of the 'back door' functions? Perhaps we should call these emulator controls?

[cfriedt]

With the gpio, i2c, and spi emulators I put together, I was mainly interested in using TDD to validate the functionality of the Greybus subsystem (in a fork) and was able to do so fairly successfully, including how it behaves with the network layers. I did something similar years ago for Linux and was able to recycle some work.

Aside from ease of initial bring up of the subsystem, I agree that the major benefit is simulating not only the "happy path", but also corner cases that are very difficult to encounter in reality.

Note, just because code is 100% covered does not mean that every possible execution condition is simulated. At least it gets us closer than otherwise possible to the ideal.

A further benefit is to use the simulated bus driver as a model for all other hardware drivers; at the very least, even if behaviour varies among hardware vendors, we want to coerce it to behave according to the model (or change the model to be more inclusive).

Another interesting possibility that this introduces, is the ability to simulate what might happen if hardware fails. E.g. if some bits in a sensor are shorted and e.g. a temperature or analog reading is consistently wrong. In some cases, that can cause catastrophic failures (think auto acceleration, gyros in aircraft, etc). It could be beneficial if there was ever a reason to pursue safety critical certification.

I think my desired result for any group of such drivers aligns with everyone's here. @pabigot has provided valuable feedback on what the expectations are for gpio-sim, so I'll be making those changes when I have spare cycles (likely after Linux Plumber's). I can also make some headway on getting my SPI simulator up to snuff at that point.

[aescolar]

How are you envisioning emulating peripherals timing?

Not seriously. That gets into the realm of simulation. You could tell the emulator to reply after a certain time I suppose. I think it has some benefits (e.g. to check that code is robust with different timing and corner cases), but it does complicate things. Perhaps this is something to consider if we see particular bugs that are timing-related.

Fair enough. I guess the main use case I was thinking about were devices that take some time to do something or be ready. For example a device where you send a command, and need to wait X µs until you can read out a result/status, or send the next one.

Yes code coverage. I mean that we want to verify code coverage for the code being tested, but not for the emulators, etc. Perhaps that is already handled as it should be just a simple filter based on filename. I haven't actually looked at it.

Yep, that should be covered already, the reports are per file, and include branch coverage. You can check with sanitycheck --coverage -p native_posix -T sometestpath/

[pabigot]

Fair enough. I guess the main use case I was thinking about were devices that take some time to do something or be ready. For example a device where you send a command, and need to wait X µs until you can read out a result/status, or send the next one.

I would do that by having the emulator in that test act as a spy and check whether transaction timing is acceptable.

The need to do that sort of thing is why I'm more interested in supporting focused, custom emulators than expecting to have a single emulator handle all cases. I think the latter is useful for integration tests, but less for functional or unit tests.

[andrewboie]

For driver subsystem APIs that have common code independent of specific drivers, something like this is valuable.

I am specifically thinking of the verification functions for driver API system calls. We would then be able to validate the correctness of these functions in emulators and not require having the relevant HW lying around to test it.

But this doesn't stop at user mode verification, a quick grep shows several subsystems have common code which could be exercised in this way:

$ find | grep common.c$
./i2c/i2c_common.c
./pcie/endpoint/pcie_ep_common.c
./adc/adc_common.c
./can/can_common.c
./video/video_common.c
./lora/hal_common.c
./i2s/i2s_common.c

As well as all the inline functions in the subsystem headers.

[andrewboie]

One thing I think worth pointing out, as I understand the focus has been on doing this under native_posix. At the moment, native_posix doesn't support user mode. So if it only worked there, we wouldn't be able to test driver subsystem syscalls with this.

There's an open ticket for this https://github.com/zephyrproject-rtos/zephyr/issues/20083 although so far nobody's had time to look into it further.

Barring implementing that, would it be possible to also ensure that we could use these simulators in QEMU platforms, such as qemu_x86 or mps2_an385? I would expect that since these simulators are purely software constructions, they ought to pretty much work anywhere, with perhaps a DTS overlay for the simulator emulated characteristics.

[sjg20]

@andrewboie Yes I think so. Of course every platform is different but ideally if we could use the same DT overlay for them it would make things easier.

But if there are tests we can't do with native_posix we should probably fix that.

[sjg20]

This is merged, so I'm closing this issue.

[sjg20]

Merged: https://github.com/zephyrproject-rtos/zephyr/pull/27300