Modularisation and Restructuring of Documentation

[nashif]

Is your enhancement proposal related to a problem? Please describe. The Zephyr documentation currently is composed of different sources all tied together through one single and long build process that collects data from the different sources, coverts the information into the final format and links everything into what we get on docs.zephyrproject.org.

The whole process is very long and on some machines can take up to 15-20 minutes, basically building everything we have, including:

• doxygen documentation converted from XML to RST using breathe
• Kconfig reference from Kconfig files converted into RST using custom scripts
• Devicetree bindings reference from binding files converted into RST using custom scripts
• board documentation in RST collected from the source tree using custom extensions
• sample/test documentation in RST collected from the source tree using custom extensions
• and finally the documentation under doc/ which is structured in a way to allow integrating the above sources

Right now it is either you build the whole thing or nothing. Some steps were taken to reduce the time by skipping the generation of lengthy reference pages (Kconfig), but things are still very slow.

Another limitation is related to the fact that we are not able to publish or build a subset of the documentation for different use cases. For example, to build a printable reference of Zephyr in PDF we would have to include everything that we have on the online version and there is no way to control how things are structured and what information end up in such a reference. If you are using the Zephyr documentation to support a product, you might be interested in including only the sections of the documentation relevant to your product, supported subsystems and hardware and not documentation of all samples and boards that come with Zephyr.

For Zephyr itself, there is a desire to produce documentation for only a subset of the features and be selective about what ends up in the final document, this requires a well structured documentation layout in general and a well defined and controlled content to allow for extraction of information without breaking cross-references and linkage.

Describe the solution you'd like

• At the first level, we need to think about how to restructure the documentation in the most optimal way to support the above, basically
  • adapt and optimize the tree structure,
  • group sections if applicable,
  • establish a standard for how sections are added and where they are added, and
  • standardize index pages and how we show table of content in sub-sections
• Provide a mechanism to allow customization of the docs based on the output and intended use. There is a difference between a catch-all online reference and a programming and developer reference. For example we could establish a set of "chapters" in an imaginary book:
  • Introduction, What is Zephyr, etc.
  • Getting Started: Environment setup, etc.
  • Developer Guide: Development with Zephyr, Debugging, etc.
  • Architecture of Zephyr, Device Model, Architecture support, Configuration system, etc.
  • Subsystem and Core OS documentation
  • API Reference
  • Board Reference
  • Sample Reference
  • Configuration option reference
  • etc.

Through configuration options it should then be possible to select any of the above to create the desired output, but it should also be possible to filter content of each chapter if it makes sense, for example, limit the board reference to a certain vendor, or filter out samples that are not relevant to the mode of output.

The above could be done in one single documentation instance, but could also be done in multiple instance that are linked together using the current doc build system. This would also improve the build speed and eventually would allow building documentation parallel (?)

[nashif]

@kestewart FYI

[nashif]

Throwing this in to get started and kick off the shuffling, covers a subset only, so feel free to add anything you think is missing, whether in chapter form or sub sections...

• Overview / Introduction
• Developing with Zephyr ○ Getting Started ○ Developer Guides ○ Contribution Guidelines ○ West ○ Modules
• Build and Configuration System ○ Build System ○ Configuration System
• Kernel ○ Kernel Services ○ User mode ○ Data structures ○ Time Utilities ○ Utilities ○ Timing Functions
• Hardware Support ○ Device Model ○ Device Tree ○ Pinctrl ○ Peripherals
• OS Services ○ Async Notifications ○ Resource Management ○ File Systems ○ Formatted Output ○ Logging ○ Debugging ○ Shell ○ Power Management ○ Storage ○ Settings ○ Crypto ○ Tracing
• Connectivity ○ Networking ○ Bluetooth

[gmarull]

I've been doing some quick hacks to the toctree and I think a layout similar to the one shown in the picture would make things easier fort both newcomers and experienced developers:

As you may notice, there are many missing parts, but I hope it shows the idea:

• Create multiple toctrees, each being kind of orthogonal to others
• Getting started should cover: setup a dev environment, how to create an application (based on example-application?), general concepts (explained quickly), how to use common IDEs, etc. Getting started pages should be short and straightforward, referencing more advanced guides where necessary.
• "Topics" would expose at top-level areas such as sensors, power management, connectivity (/bluetooth), fieldbus (/canbus, /modbus...), Kernel, etc. These areas would include inside: guides, API, examples, etc. The motivation: if I'm interested in a particular topic, e.g. sensors, I should have everything under topics/sensor: the API reference, how do I create a driver, examples, etc.

This approach is somehow inspired by https://docs.godotengine.org/en/latest/index.html (not new, I like these docs :smile:)

[nashif]

• "Topics" would expose at top-level areas such as sensors, power management, connectivity (/bluetooth), fieldbus (/canbus, /modbus...), Kernel, etc. These areas would include inside: guides, API, examples, etc. The motivation: if I'm interested in a particular topic, e.g. sensors, I should have everything under topics/sensor: the API reference, how do I create a driver, examples, etc.

"Topics" is very general, what is the criteria for something to land under Topics? Based on the example you provide this could grow into a large number of topics that are in fact sub-topics, i.e. I do not see why pinctrl or fieldbus needs to be a top level topic and not part of something more general (i.e. hardware support or something else) with those being sub-topics instead.

• Create multiple toctrees, each being kind of orthogonal to others

Can you elaborate on this a bit more?

[gmarull]

• "Topics" would expose at top-level areas such as sensors, power management, connectivity (/bluetooth), fieldbus (/canbus, /modbus...), Kernel, etc. These areas would include inside: guides, API, examples, etc. The motivation: if I'm interested in a particular topic, e.g. sensors, I should have everything under topics/sensor: the API reference, how do I create a driver, examples, etc.

"Topics" is very general, what is the criteria for something to land under Topics? Based on the example you provide this could grow into a large number of topics that are in fact sub-topics, i.e. I do not see why pinctrl or fieldbus needs to be a top level topic and not part of something more general (i.e. hardware support or something else) with those being sub-topics instead.

I didn't have better ideas, but I agree Topics is not a great name. What is a topic or a sub-topic is really subjective, so there's no single answer here. I think the idea would be to expose at top-level common or popular areas, e.g. sensors, Kernel or Devicetree may deserve a top-level entry whereas Fieldbus could be under protocols, industrial, or communications areas. I was thinking on a range of ~20-25 top-level topics.

• Create multiple toctrees, each being kind of orthogonal to others

Can you elaborate on this a bit more?

If you look at the screenshot you have in purple "GENERAL", "TOPICS", etc. These are the different toctree titles, IMO they create a clear content split. We now have 2: "CONTENTS" (everything) and "REFERENCE" (API docs, Kconfig, DT bindings).

[nashif]

If you look at the screenshot you have in purple "GENERAL", "TOPICS", etc. These are the different toctree titles, IMO they create a clear content split. We now have 2: "CONTENTS" (everything) and "REFERENCE" (API docs, Kconfig, DT bindings).

ok, thanks. this looks good and we should consider it IMO

[carlescufi]

API meeting:

• Verify via GH Action if Doxygen groups are not pulled into RST
• New APIs must require context in RST and not in Doxygen. Doxygen groups shall only contain define/macro/function documentation and not generic descriptions
• Kconfig will be rehashed completely in #41753
• Devicetree bindings stay as they are for now
• Agreement on grouping by subject and not by documentation type (guide/reference)

[mbolivar-nordic]

Here is my suggested top level tree. It includes a few changes from @nashif 's. For convenience, here is a diff from his proposal to mine: https://gist.github.com/mbolivar-nordic/55081101a53641cfed2555d01220ff63

• Overview / Introduction

• Developing with Zephyr ○ Getting Started ○ Tutorials

  • Software Configuration: Kconfig
  • Hardware Configuration: Devicetree ○ West ○ Modules

• Build and Configuration System ○ Build System ○ Kconfig ○ Devicetree

• Kernel ○ API status [stability, etc] ○ Kernel Services ○ User mode ○ Data structures ○ Time Utilities ○ Utilities ○ Timing Functions ○ Device Model

• OS Services ○ API status [stability, etc] ○ Async Notifications ○ Resource Management ○ File Systems ○ Formatted Output ○ Logging ○ Debugging ○ Shell ○ Power Management ○ Storage ○ Settings ○ Crypto ○ Tracing

• Hardware Support ○ SoCs [new, list of SoCs with forward links to drivers pages for SoC peripherals] ○ Boards [changed, each board page now links to SoC page and describes extra HW features] ○ Drivers [new, list of drivers by API, with DT compatibles, main Kconfig options, & descriptions]

• APIs ○ API status [stability, etc] ○ [ other APIs mentioned in "API Overview" not already covered elsewhere ]

• Connectivity ○ Networking ○ Bluetooth ○ ...

• Development and Governance ○ TSC Project Roles ○ Release Process ○ Feature Tracking ○ Code Flow and Branches

• Contributing to Zephyr ○ Contribution Guidelines ○ Sending Pull Requests ○ Reviewing Pull Requests ○ Modifying Contributions Made by Other Developers ○ Filing Issues ○ Writing Documentation

[carlescufi]

@mbolivar-nordic, I do like your proposal, but I have a few questions:

1. Where is the Application Development section?
2. Why that "Tutorials" section with Kconfig and DT when we already have entries for those under "Build and Configuration System"?
3. Why a separate "APIs" top-level entry?

[gmarull]

Another tentative toctree. Not all sections may be correct, the general idea is to expose more content (make root toctree longer). Many content is hard to discover now because it is deeply buried in the toctree. It's hard to do since I'm still discovering new content...

INTRODUCTION
------------

* About Zephyr
* Getting Started
  - GSG
  - Beyond GSG
  - Basic samples
* Supported Boards
  - ARM
    - ...

REFERENCE
---------

/* A section with reference guides (including APIs, related samples in each section) */

* Build System
* Devicetree
* Kconfig
* Kernel
* Connectivity
  - Networking
  - Bluetooth
* Peripheral drivers
  - ADC
    - API description, usecases, implementation guidelines, etc.
    - Supported hardware (include table with DT compatibles?)
    - Samples
  - GPIO
  - ...
* OS Services
  - File systems
  - Logging
  - Tracing
  - ...
* Testing
...

DEVELOPMENT
-----------

/* A section related to day-to-day development */

* Applications
  - Example Application (?)
* IDE Support
  - VSCode
  - PlatformIO
  - ...
* Coding Standards
  - Styleguide
  - Rules
* Documentation
  - Howto
  - Guidelines (rst/Doxygen)
  ...
* Tools
  - Coccinelle
  - clang-format
  ...

COMMUNITY
---------

* Contributing
* Communication channels
* Organization
  - TSC Project Roles
  - Release Process
  ...
* Example projects
  - Projects using Zephyr (?)

OTHERS
------

/* Useful shortcuts, used as reference */

* API Pages (Doxygen)
* Kconfig Options
* Devicetree Bindings
* Terminology
  - Glossary
  - Environment Variables

Other points I'd like to discuss:

• What happens with pages that just pull API (no value). Examples:
  • https://docs.zephyrproject.org/latest/reference/crypto/index.html
  • https://docs.zephyrproject.org/latest/reference/display/index.html
• Supported hardware is loosely described in boards now. We should improve this. Alternative experiment here: https://docs.zephyrproject.org/latest/boards/arm/gd32f450i_eval/doc/index.html#supported-features
• Most boards repeat the same "programming and debugging" piece. https://docs.zephyrproject.org/latest/boards/arm/gd32f450i_eval/doc/index.html#programming-and-debugging.
• Bring tutorials in. It would be nice to include content like https://blog.golioth.io/how-to-parse-json-data-in-zephyr/ in the docs.

[mbolivar-nordic]

• Where is the Application Development section?

Good question. I guess it's missing from @nashif's proposal as well, so I am sure he will have his own answer for this question. Probably we should add it back, although looking at it again I think some of the content would be better off elsewhere in the docs.

• Why that "Tutorials" section with Kconfig and DT when we already have entries for those under "Build and Configuration System"?

This is part of a long term plan I have that relies on cross-platform QEMU support as part of @stephanosio 's new SDKs for macOS and Windows having all the host tools.

Basically I think we should add quick tutorials that cover simple modifications to hello_world that treat the basic things.

So for Kconfig, the tutorial would look like this:

• creating an application-specific boolean kconfig option
• testing the ifdef in a C file, seeing it's off
• setting it to y in prj.conf
• repeating the ifdef test in the C file, seeing it's on
• setting it back to n in menuconfig or guiconfig
• repeating the ifdef test, seeing it's off

Then you tell people that drivers and subsystems each have kconfig options they can turn on, and other values like integers and strings that they can use to set similar macros. This would give people better intuition for Kconfig if they don't have a Linux background than our current documentation IMO.

Then do something similar for devicetree, showing people how to

• get their devicetree in the build directory
• get a simple node identifier in C
• print a simple property value in C
• write a simple overlay that sets up something in /zephyr,user
• print that value from C

That in turn enables other documentation improvements like:

• rewriting the driver model documentation in a way that assumes this knowledge, encapsulating a lot of the ideas I went over in my ZDS 2021 talk on the driver model and its relationship with Kconfig/DT in a clean way
• writing device driver indexes that list the main options to turn on the driver along with the devicetree compatibles consumed by the driver

• Why a separate "APIs" top-level entry?

Where else are we going to document the [ other APIs mentioned in "API Overview" not already covered elsewhere ]?

[mbolivar-nordic]

• Supported hardware is loosely described in boards now. We should improve this. Alternative experiment here: https://docs.zephyrproject.org/latest/boards/arm/gd32f450i_eval/doc/index.html#supported-features

I am thinking we need:

1. a machine-readable index of SoCs (map from SoC names to metadata, like what DTSI files are associated with the SoC)
2. a machine-readable index of device drivers that implement particular APIs (map from APIs to lists of drivers; each driver in the list is a map from a name to driver metadata; driver metadata is things like the Kconfig option to enable the driver and any devicetree compatibles it enables

Then we can add some sphinx scripting to make the documentation generated from this info in a nice way.

I've been sketching around and have a proposal like this for SoCs:

───────┬───────────────────────────────────────────────
       │ File: soc/arm/nordic_nrf/socs.yaml
───────┼───────────────────────────────────────────────
   1   │ # supported nRF SoCs; repeat for other vendors
   2   │ 
   3   │ nRF51822:
   4   │   dtsis:
   5   │     - nordic/nrf51822_qfaa.dtsi
   6   │     - nordic/nrf51822_qfac.dtsi
   7   │ nRF52805:
   8   │   dtsis: [nordic/nrf52805_caaa.dtsi]
   9   │ nRF52810:
  10   │   dtsis: [nordic/nrf52810_qfaa.dtsi]
  11   │ nRF52811:
  12   │   dtsis: [nordic/nrf52811_qfaa.dtsi]
  13   │ nRF52820:
  14   │   dtsis: [nordic/nrf52820_qdaa.dtsi]
  15   │ nRF52832:
  16   │   dtsis: [nordic/nrf52832_qfaa.dtsi]
  17   │ nRF52833:
  18   │   dtsis: [nordic/nrf52833_qiaa.dtsi]
  19   │ nRF52840:
  20   │   dtsis: [nordic/nrf52840_qiaa.dtsi]
  21   │ nRF5340:
  22   │   dtsis:
  23   │     - nordic/nrf5340_cpuappns_qkaa.dtsi
  24   │     - nordic/nrf5340_cpuapp_qkaa.dtsi
  25   │     - nordic/nrf5340_cpunet_qkaa.dtsi
  26   │ nRF9160:
  27   │   dtsis:
  28   │     - nordic/nrf9160ns_sica.dtsi
  29   │     - nordic/nrf9160_sica.dtsi
───────┴───────────────────────────────────────────────

We could do similarly for drivers. Thoughts?

[nashif]

Closing, as the bulk of the proposed changes has been done and is merged.