System devicetree support

[mbolivar-nordic]

Postponed

The initial support for #51833 is expected to come without system devicetree. We would like to wait until a system devicetree specification v1.0 is available before merging support for it into Zephyr, and think we can make meaningful progress with standard devicetree as long as the other aspects of 51833 are implemented.

Parent issue

This issue is part of a larger issue: https://github.com/zephyrproject-rtos/zephyr/issues/51833

Is your enhancement proposal related to a problem?

Zephyr's support for AMP SoCs... works, but has several well-known usability problems, and breaks down in some cases.

Some of these problems are due to workarounds for limitations in the devicetree specification itself. This issue is about allowing users to resolve the devicetree-specific portions of these problems by adoption of the system devicetree concept described further below. The proposal, if adopted, shall be opt-in and backwards compatible for vendors and maintainers that choose to ignore it.

Details on the problem

In particular, it is complicated and error-prone for users to configure shared resources used by the different CPU clusters in an AMP SoC in devicetree. This is because the same or similar devicetree overlays must be written for each cluster in the SoC describing the shared resource configuration, and applied individually to application build systems for each cluster. This makes it error prone to handle basic use cases like overriding memory partitions (for use cases related to bootloaders, file systems, etc.) and setting application-specific peripheral ownership (for use cases like assigning secure-only peripherals to Armv8-M trusted firmware applications).

There are also problems which occur when doing board porting. Porters have to define multiple different "BOARD" names and associated BOARD.dts files, one per CPU cluster, within the same board directory. This is awkward, but worse, it breaks down completely when the different CPU clusters in the SoC have different architectures. There is currently no way to support this in Zephyr except for resorting to hacks like defining two or more different board directories, one per CPU architecture used in the SoC, adding further needless complexity to the board maintainer's task of setting default resource configurations in devicetree.

The reason for these problems starts with the base devicetree .dtsi file for the "SoC": this is actually (and usually) just a devicetree for one particular core cluster within the SoC. In the case of Armv8-M SoCs with TrustZone support, it can be a devicetree for one security level within one core cluster. These "SoC" devicetrees are then included separately by "board" devicetrees, one per target cluster (or cluster/security level).

The nRF5340 is a prototypical example: to support a board with this SoC, you need 3 BOARD names, e.g.:

• nrf5340dk_nrf5340_cpuapp (application core, secure)
• nrf5340dk_nrf5340_cpuapp_ns (application core, non-secure)
• nrf5340dk_nrf5340_cpunet (network core)

These in turn include (in some cases, only slightly) different "SoC" .dtsi files defining the hardware visible to a particular core (or core + security level):

• nrf5340_cpuapp_qkaa.dtsi (application core "SoC", secure)
• nrf5340_cpuappns_qkaa.dtsi (corresponding non-secure "SoC")
• nrf5340_cpunet_qkaa.dtsi (network core "SoC")

We adopted this in zephyr as a workaround for what devicetree can do, but it's not ideal for (hopefully) obvious reasons, and is confusing to new users.

Use cases like overriding memory in devicetree for shared use between different subsets of the available CPU clusters means setting up multiple different devicetree overlay files with substantially the same content, and passing these into the build system via DTC_OVERLAY_FILE or boards/ directories for each BOARD in the relevant applications. In addition to being awkward, this is error prone, because the individual files can go out of sync.

Describe the solution you'd like

I would like Zephyr's devicetree tooling to support system devicetree and use it to enable solutions to the above problems.

System devicetree basics

The upstream devicetree organization is hosting the system devicetree specification here:

https://github.com/devicetree-org/lopper/tree/master/specification

System devicetree is an extension to the basic devicetree semantics that allows users to declare multiple CPU clusters in an AMP SoC, along with their respective peripherals, memories, etc., all in one file. The system devicetree file format is still DTS syntax; only the semantics are an extension. It is possible to define multi-core SMP CPUs within this single file alongside independent CPU clusters.

By adopting this specification, we can write a single "system devicetree" for an SoC that really describes all the hardware in it, and configure shared resources at the system level in this syntax, at the SoC, board, and application levels. This will in turn enable build system enhancements that avoid the confusion, awkwardness, and error-proneness described above.

Architecture overview

The final system devicetree is intended to be postprocessed into one or more "regular" devicetrees that are each compatible with existing users of the DT specification, including Linux and Zephyr. The basic picture looks like this:

In the diagram, the "regular" devicetrees at the bottom can be processed using zephyr's existing python-devicetree package.

The generation process itself will be handled by further extensions to the python-devicetree package, along with an additional script that uses the package at CMake time to perform the conversion from system DT to regular DT(s). The process will look like this:

Above, a system devicetree include (sysdtsi) file for the SoC is provided by the SoC maintainer. This is then included in a system devicetree (sysdts) for the board within a board directory. That in turn is postprocessed along with any overlays by the new python tooling into regular devicetrees (dts) consumable by existing zephyr build system tooling, device drivers, etc. The result is an opt-in, backwards compatible extension to existing features that addresses our problems. Some example use cases addressable at different places are shown in the lists adjacent to each file.

The current end goal is that this process will use sysbuild to process the system devicetree along with any overlays. The individual zephyr build systems managed by sysbuild are shown as shaded boxes surrounding each "Generated regular DT" in the diagram. If we see fit to allow using the system devicetree within the Zephyr build system for addressing particular use cases, though, that remains an option.

Details regarding python tooling

The authors of the system devicetree specification also provide a tool called lopper which can transform system devicetrees to regular devicetrees.

I evaluated this tool and concluded that it would be simpler and better for Zephyr to extend its existing devicetree tooling instead of adopting it for the following reasons:

• lopper is geared towards the FDT format in ways that make it hard to use for us. To adopt lopper, we would have to go through a process of extending its internal APIs to suit our needs by exposing information we already have "for free" in the python-devicetree representation to be able to use them to transform SDT -> DT, which is extra unnecessary effort
• the features lopper provides are much more general than I anticipate we will need within Zephyr, so the extra dependency and level of indirection (and attendant learning curve for users and SoC maintainers) doesn't seem worth it to me right now.
• there doesn't seem to be much point to me at present in adopting lopper's Python APIs for manipulating devicetrees, as they are effectively Python wrappers for our own existing Python APIs when using python-devicetree as a backend. I would rather avoid having to go through an upstream review and merge process every time we want to make an extension to our DT APIs. By keeping everything in python-devicetree but adopting the system DT semantics, we benefit from the work they have done while preserving our ability to quickly add specific API features as needed.

If we ever want to reverse this decision and adopt lopper, it should be possible to do this without breakage in a backwards-compatible way.

Describe alternatives you've considered

1. Continue the status quo: in Nordic's opinion the existing tools have been stretched to their limit, and we would really like to drive towards a solution alongside the entire Zephyr community
2. Adopt system DT as a specification along with lopper as a tool: see "details regarding python tooling" above

Specification level issues

This section contains a list of follow-up issues in the system devicetree specification after I converted it to .rst and restructured it to match the regular DTSpec style in a series of commits.

Resolving these blocks 100% complete support for system DT in our python-devicetree package. We will likely be able to get an MVP done without having to resolve all of them by just restricting our implementation to support a subset of the spec at first, however. We can proceed incrementally from there by adding more support as semantics are more clearly specified.

• https://github.com/devicetree-org/lopper/issues/171
• https://github.com/devicetree-org/lopper/issues/170
• https://github.com/devicetree-org/lopper/issues/169
• https://github.com/devicetree-org/lopper/issues/168
• https://github.com/devicetree-org/lopper/issues/167
• https://github.com/devicetree-org/lopper/issues/166
• https://github.com/devicetree-org/lopper/issues/165
• https://github.com/devicetree-org/lopper/issues/164
• https://github.com/devicetree-org/lopper/issues/163
• https://github.com/devicetree-org/lopper/issues/162
• https://github.com/devicetree-org/lopper/issues/161

Change history

• 2023-03-02: add list of follow-up issues blocking completion of system DT support in our python-devicetree package
• 2023-02-20: adjust python tooling language after external review
• 2022-10-31: initial proposal

[mbolivar-nordic]

Here are the slides for a presentation I plan to give at the next architecture working group on this subject: (EDIT: uploaded a second version with a few more slides)

20221101-upstream-sysdt-v2.pdf

[galak]

I think we should think about having "system devicetree" for ALL devicetrees in Zephyr regardless of multi-core or not. The main aspect I think for the single CPU is using 'cpu clusters' instead of 'cpus'.

[galak]

I think it would be interesting

I think we should think about having "system devicetree" for ALL devicetrees in Zephyr regardless of multi-core or not. The main aspect I think for the single CPU is using 'cpu clusters' instead of 'cpus'.

@henrikbrixandersen made the comment about compatibility of devicetrees with Linux.

[galak]

Would also be interesting to describe single core ARM v8m-TZ devices with system devicetree as a use case. So the tooling we introduce into zephyr could handle taking a system devicetree for a single core device and producing the "correct" dts based on being to run under TZ or not.

[mbolivar-nordic]

Those should all be doable, for sure. You can keep /cpus and simple buses in system DT as well, so I believe we can treat regular devicetrees as special cases of system devicetrees if that's what people prefer.

[mbolivar-nordic]

Added a list of "Specification level issues" in the description that are hurdles to completing our sysdtlib implementation in python-devicetree.