search

zephyrproject-rtos / zephyr

Primary Git Repository for the Zephyr Project. Zephyr is a new generation, scalable, optimized, secure RTOS for multiple hardware architectures.
https://docs.zephyrproject.org
Apache License 2.0
10.93k stars 6.65k forks source link

Safety development concept (for IEC61508 certification) #57516

Open romkell opened 1 year ago

romkell commented 1 year ago

Introduction

The quality and finally safety (IEC61508 SIL3 certification) and security development of Zephyr OS requires

This RFC defines a rough concept how the safety committee intends to setup, both, tools and processes to do safety development.

Serveral areas, in example requirements management, are not restricted to safety and shall be setup at the best of our current knowledge to suite for the full extend of the ecosystem.

Nevertheless, in order to be able to finish with the safety certifcation task, it must be accepted that the focus of the safety committee and WG and therefore this RFC is safety.

Please feel invited

Problem description

An overall concept how to do safety is not (fully) defined, described and known to the safety WG and TSC. It is necessary to achieve a common understanding.

Proposed change

Create an overall concept by defining processes and a tool environment that allows to get 'main' in a 'safety certifiable ready' state of which an auditable can be branched with basically no gaps to reach certification. Doing safety on auditable and forward merge all code changes, all tests, requirements etc. has shown to not work.

We start with an rough concept and iteratively detail and implement the different areas.

Detailed RFC

Proposed change (Detailed)

Overview

image

Requirements Management

For safety (IEC61508) the following specifications need to be written:

Requirements specifications

Requirements management(SRS, SAIS, STS) is done using

Storage location is zephyrproject-rtos/reqmgmt repo (https://github.com/zephyrproject-rtos/reqmgmt).

Test specifications

Remark: possibly replace term 'module' as in 61508 by 'component' as in 50128 or 'unit' as in unit test for SMDS and SMTS, since easily confused with git (sub-)module.

Definition: Module as in 61508 is in most cases referred to as a implementation / header file pair in a C/C++ context (my_module.c/.h).

Test specifications are written as doxygen comments to the unit-, integration- and software tests into the zephyrproject-rtos/zephyr repo (https://github.com/zephyrproject-rtos/zephyr). The linkage to a Doorstop requirements (SRS, SAIS, SMDS) is done by doxygen tag referencing the requirement by its ID.

Requirements process

Adding requirements or test specifications uses the same workflow as any code change does:

The standard github features (PR, diff, review) are used

Requirements wording

Requirememts shall be

and follow the 'Easy Approach to Requirements Syntax' (EARS) principle

Traceability

Traceability is created by a dediated 'tracedoc' tool. The tool takes the

Remark: Intel has a prove-of-concept implemented.

By defining an open and simple input interface (csv) into tracedoc, requirements tools are not limited to Doorstop and the requirements do not necessarily have to come from the same source (repo, system etc.)

Future: tracedoc could be extended to process multiple csv input files from different requirements tools and scan over multiple repos. We start with the simple 1:1 case.

Linkages

The following link roles will be used:

Safety Scope

Why a safety scope? The Zephyr OS is by far too big, that we can achieve a safety certification of the full code base. Hence we need to restrict it to a smaller scope and provide measures to distinguish non-safety from safety (for us and for a user).

The safety scope as defined will be marked using two mechanisms:

  1. The source files in the safety scope will carry an SPDX tag, similar to the SPDX license tag, indicating the safety relevance/scope

    • In that way safety relevant code is clear identified and immediately visible to developers and maintainers
    • Special (stricter) guidelines may apply to source code within the safety scope than to the rest of the code (e.g.: use of dynamic memory)
    • The SPDX tool can be used to verify that a built binary only comprises of sources out of the safety scope and no other (similar to that a binary only comprises of sources under a certain license model (e.g. no GPL)

  2. The maintainer.yml will define folders and file belonging to the safety scope

    • In that manner PRs affecting the safety scope can be auto-assigned to dedicated safety maintainers
    • Possibly to an include syntax also an exclude syntax need to be defined and implemented (/folder/A but not /folder/A/fileB.c)

Using both, SPDX and maintainer.yml, will be used to validate the safety scope against each other for plausibility / congruence (tool: scope validator, added to CI)

Coding Guidelines

The coding guidelines will be enforced as far as possible using a static code analyzer. Guidelines which cannot be enforce by tooling, will be verified by code review prior to merging the github PR.

Static Code Analyzer (SCA)

Testing

Testing will be done according to IEC61508 on the three levels:

Testing folder structure

The /tests folder (as already existing) will resemble (except for the /tests folder itself) the repos folder structure and place the tests under dedicated sub-folders.

An example to depict the idea (the shown existing tests might have been placed in the wrong category/location, creating the shown folder structure will require a transition phase from the existing structure to the shown)

Dependencies

Introducing a system for quality to achieve a safety certification will introduce changes in various areas. It has to be an iterative step by step process in which the direction very likely will have to be corrected now and then.

Concerns and Unresolved Questions

Is part of this RFC's discussion to adjust the concept.

Alternatives

Is part of this RFC's discussion to adjust the concept.

gregshue commented 1 year ago

Nice! Thank you!

  1. What level of security must be in place to make these safety claims? This will help identify which security functionality must be included in the safety scope.

  2. Some users will also need to extend or develop/verify safety-certifiable code in other repositories (e.g., libc, HAL, mcuboot, downstream forks of Zephyr). Please be explicit about how this process extends across multiple repositories, especially for each of the specified workspace topologies (T1, T2, T3).

  3. End users will need to trace Zephyr requirements to their higher-level system requirements. Please describe how that can be done while still using Zephyr's safety-certifiable SHA.

gregshue commented 1 year ago

šŸ¤” If "module" refers to an individual .c/.h pair, then we need to expect multiple SRS/SAIS recursions between the requirements on the executable and the requirements on an individual file. The number of recursions may change as the architecture evolves.

nicpappler commented 1 year ago

That's a lot to discuss in a single RFC - how about we split this into several sub RFCs?

nicpappler commented 1 year ago

@gregshue - regarding the security and safety claims - as already discussed, there are no security claims attached to the safety claims. Both safety and security need a certain level of systematic capability. We aim for the systematic capability of SIL 3 (IEC 61508:2010), that also is enough for security, but does not make specific claims regarding security features, mechanisms, etc.

romkell commented 1 year ago

That's a lot to discuss in a single RFC - how about we split this into several sub RFCs?

I would leave the rough concept in here as a central point and to have an overview which we loose when splitting - and the overview was one goal. But we can create topic RFC such as "Safety development concept - RQM" etc. linked to here and discuss details there.

gregshue commented 1 year ago

@gregshue - regarding the security and safety claims - as already discussed, there are no security claims attached to the safety claims. Both safety and security need a certain level of systematic capability. We aim for the systematic capability of SIL 3 (IEC 61508:2010), that also is enough for security, but does not make specific claims regarding security features, mechanisms, etc.

I repeated the question here to capture the topic within this issue. The question originated from the following statement in Embedded Software Development for Safety-Critical Systems, Second Edition (Chris Hobbs, CRC Press, 2020), p 10,

Safety is interlinked with security - the behavior of an insecure device is effectively unpredictable once a hacker has found a vulnerability.

This may be another subject appropriate for a topic RFC.

RobertoBagnara commented 1 year ago

Regarding the choice of Doorstop, I think it is not a good one, for the following reasons.

Doorstop is arguably unmaintained ā€“ its creator only sends Python support patches from time to time. For any advanced patches from other people, there are long unfinished threads hanging around with no progress being made.

About "Doorstop uses single yaml files per requirement", this is not so good granularity, if one considers that everything in the industry is kept in documents (requirement specifications, test specifications, verification control document, etc.). StrictDoc intentionally packages requirements in .sdoc document files because people create and read the whole documents and requirement sections. Maintaining separate files without first-class document support is cumbersome. Take the "level:" field for example https://github.com/doorstop-dev/doorstop/blob/develop/reqs/REQ002.yml#LL4 ā€“ it is so inconvenient to manage it on the single req's YAML file level.

Merge conflict argument works a little better with Doorstop, but I could argue that if there is a conflict, this already means that the engineers working on requirements went out of sync. To get a merge conflict in an .sdoc file, you really have to work on the same or closely neighboring requirements or someone has to start moving sections around, which already indicates that people work without any synchronization.

Additionally, StrictDoc has made and is making a lot of progress recently: see the GitHub project and compare it with the one for DoorStop. In particular, StrictDoc has at least an 80%-working bridge to ReqIF export and import. I personally used it to successfully import Polarion-generated requirements in ReqIF format. Doorstop doesn't have ReqIF support.

RobertoBagnara commented 1 year ago

The following sentence misses one point: "Even if the open source SCA does not have the same quality as a commercial grade solutions, it will be suitable to detect and this way fix 80 - 90% of the violations in a fast development cycle." It is not just that cppcheck has lots of false negatives, it also has lots of false positives. The danger is that developers that have not been properly trained on the coding guidelines will not be able to recognize the false positives and will believe they have to to something about them. In my experience, this always causes a decrease in code quality.

gregshue commented 1 year ago

We need a requirements management system keeping with the composability and extensibility of Zephyr.

  1. It needs to support a single "document" (e.g., StRS, SyRS, SyAID, SRS, SAID, etc.) that has requirements spread across multiple repositories in the workspace.
  2. It needs to support "variation management" - tracking moderately different requirement sets for closely related products.

I have attempted to use Doorstop and found it at least unable to link to requirements in other repositories. I didn't look into variation management or aggregating requirements from multiple repositories.

@RobertoBagnara How well does StrictDoc support those needs?

sebhub commented 1 year ago

Zephyr is not the first project for which a pre-qualification according to safety related standards is available. We did something similar for the RTEMS real-time operating system. The pre-qualification for RTEMS was performed using software development standards from the European Cooperation for Space Standardization: ECSS-E-ST-40C and ECSS-Q-ST-80C Rev.1. We took also other standards such as IEC 61508, ISO 26262, DO-178, DO-330, DO-333, and Galileo Software Standard (GSWS) into account. The work started in 2020 and the first results are available on a ESA website: https://rtems-qual.io.esa.int/. The technical note QT-109 contains some background information. The latest documentation of the RTEMS pre-qualification process is available here: https://docs.rtems.org/branches/master/eng/req/index.html. A key observation is that the standards require a high degree of traceability from requirements to source code and tests and vice versa.

Interestingly, the proposal here has some overlaps with what we did. I can highly recommend the EARS syntax for requirements.

We also use Doxygen to make the link from the specification to the source code and vice versa. In a first pass, Doxygen is invoked to produce a tagfile. This tagfile is used to link requirements to source code. Then a file is generated which contains Doxygen comments for source code elements such as functions. The generated comments augment the existing Doxygen documentation and provide links to the specification.

For the requirements (or specification in general) we also started with Doorstop. However, while working with it, it turned out that some customization was required: https://docs.rtems.org/branches/master/eng/req/tooling.html. The RTEMS specification contains not just requirements, so we added a hierarchical type system for specification items: https://docs.rtems.org/branches/master/eng/req/items.html. These types are also specified by specification items: https://github.com/RTEMS/rtems-central/tree/master/spec/spec. It can be easily customized to meet project-specific requirements.

Real-time operating systems such as Zephyr or RTEMS are just a collection of state machines and data structures. For RTEMS the majority of the function specification is done by the specification of state machines. We use a specialized specification item for this: https://docs.rtems.org/branches/master/eng/req/howto.html#action-requirements. It contains the specification along with the corresponding validation test code. It needs a bit of learning to be able to write these files (example: https://github.com/RTEMS/rtems-central/blob/master/spec/c/req/posix-memalign.yml), however, it turned out to be very useful for the functional specification of RTEMS. We reach 100% branch and code coverage for the pre-qualified feature set of RTEMS through validation tests. Since we enumerate all state transitions, this yields more or less MC/DC coverage automatically.

Performance requirements are stated in terms of runtime measurement requests. The measurements are done using the RTEMS Test Framework. This test framework would also work with Zephyr.

We use Python modules for the tooling (BSD-2-Clause license): https://github.com/RTEMS/rtems-central/tree/master/rtemsspec.

Just let me know if you are interested in this work. Maybe it can be generalized so that it can be used also for Zephyr.

romkell commented 1 year ago

@sebhub

Zephyr is not the first project for which a pre-qualification according to safety related standards is available. We did something similar for the RTEMS real-time operating system.

Many thanks for reaching out to us. We are aware of RTEMS in general, basically due to your evaluation of Doorstop and your decision to implement your own RQM tool for various reasons. At least I have not looked into your detailed implementation and requirements of RTEMS, but appreciate your input.

The Zephyr OS code is existing and rewriting everything to yield MC/DC coverage will I assume effort-wise not work at least in the first place. It also is not demanded by IEC61508 SIL3, which we aim for. Let me have a look at all the links you provided.

romkell commented 1 year ago

@RobertoBagnara

It is not just that cppcheck has lots of false negatives, it also has lots of false positives. The danger is that developers that have not been properly trained on the coding guidelines will not be able to recognize the false positives and will believe they have to to something about them. In my experience, this always causes a decrease in code quality.

The final check is the SCA in the CI that matters. How do you suggest to provide fast SCA checks on a local developer machine with short cycle times with free access to the community? Or how do you suggest the community shall do SCA runs? Just via CI?

RobertoBagnara commented 1 year ago

We need a requirements management system keeping with the composability and extensibility of Zephyr.

1. It needs to support a single "document" (e.g., StRS, SyRS, SyAID, SRS, SAID, etc.) that has requirements spread across multiple repositories in the workspace.

2. It needs to support "variation management" - tracking moderately different requirement sets for closely related products.

I have attempted to use Doorstop and found it at least unable to link to requirements in other repositories. I didn't look into variation management or aggregating requirements from multiple repositories.

@RobertoBagnara How well does StrictDoc support those needs?

TL;DR: Currently, StrictDoc does not support this, but it could. In my experience, developers are very keen on receiving feedback and making StrictDoc more useful. Rather, what is needed is a more detailed specification of how the feature-based requirements can be filtered.

Longer version:

The filtering has to happen based on some criteria. Specific feature-based requirements are either:

Once the convention is established, one can think of implementing a query mechanism that takes a given document, filters in/out given requirements and then the resulting document is a needed subset. We did implement a similar mechanism in a day.

This feature definitely requires a good specification and a good implementation, so that it is not hard-coded to only Zephyr community as a single user.

RobertoBagnara commented 1 year ago

@RobertoBagnara

It is not just that cppcheck has lots of false negatives, it also has lots of false positives. The danger is that developers that have not been properly trained on the coding guidelines will not be able to recognize the false positives and will believe they have to to something about them. In my experience, this always causes a decrease in code quality.

The final check is the SCA in the CI that matters.

If developers plagued by false positives worsen code quality, the final check will come too late.

How do you suggest to provide fast SCA checks on a local developer machine with short cycle times with free access to the community? Or how do you suggest the community shall do SCA runs? Just via CI?

As far as I know, no free tool has half-decent support of the MISRA coding standards. Said that, yes, a CI infrastructure allowing developers to analyze their own branches/forks/whatever could do the job.

gregshue commented 1 year ago

The filtering has to happen based on some criteria.

In the EARS syntax optional feature requirements are captured with the "Where \<feature is included>," fragment. This condition can be expressed as either a series of Where conditions (implicitly all must be true), or as a single condition with explicit logical "and" conjunctions.

sebhub commented 1 year ago

We need a requirements management system keeping with the composability and extensibility of Zephyr.

1. It needs to support a single "document" (e.g., StRS, SyRS, SyAID, SRS, SAID, etc.) that has requirements spread across multiple repositories in the workspace.

The RTEMS specification graph can be built using any number of directories, for example:

The tools visit a configurable list of base directories and search recursively for specification items. The items are added to an item cache which associates an UID with exactly one item. The UID relates to the path of the file relative to the base directory. Specification item are stored in YAML or JSON files.

2. It needs to support "variation management" - tracking moderately different requirement sets for closely related products.

For RTEMS we solved this by adding an enabled-by attribute to each specification item. For example, some requirements are restricted to SMP configurations: https://github.com/RTEMS/rtems-central/blob/master/spec/rtems/scheduler/req/get-processor-smp-only.yml. Other requirements depend on a particular SoC: https://github.com/RTEMS/rtems-central/blob/master/spec/bsp/sparc/leon3/req/errata-gr712rc-05.yml. These enabled-by attributes can be viewed as Where clauses in EARS.

gregshue commented 1 year ago

@sebhub Thank you for the pointers. It looks to be very relevant content.

The RTEMS specification graph can be built using any number of directories, ...

From the RTEMS UID definitions it looks like RTEMS requires a (all?) specification(s) to be within a consistent, fixed directory tree. The Zephyr build system learns about the Zephyr module pathnames through the ZEPHYR_MODULES and ZEPHYR_EXTRA_MODULES variables. Any idea about the complexity of inserting a module reference parameter into the UIDs?

stanislaw commented 1 year ago

It needs to support a single "document" (e.g., StRS, SyRS, SyAID, SRS, SAID, etc.) that has requirements spread across multiple repositories in the workspace.

It needs to support "variation management" - tracking moderately different requirement sets for closely related products.

@RobertoBagnara How well does StrictDoc support those needs?

Hello everyone,

I am the developer of StrictDoc. I became aware of this thread and thought I could answer some of the questions about StrictDoc directly.

StrictDoc is in active development, and we are always happy to get feedback from the users and implement all improvements suggestions that match our roadmap.

I started building StrictDoc in 2019 to implement some features that I was missing in Doorstop and the main use case I had in mind originally was the production of requirements documentation for spacecraft avionics projects according to the ECSS standards. The ECSS-E-ST-40C suggests that a documentation package for software projects consists of documents like:

StrictDoc is based on a Document concept that consists of sections/chapters or requirements, and sections can consists of child sections or requirements. Besides including requirements, sections can have blocks of free text. Free text blocks provide all necessary glue to help a user to understand the requirements. A document can also contain only sections and free text. Example: An ECSS SW User Manual document normally does not have requirements but only free text structured with sections (chapters).

An example of a typical SDoc file

StrictDoc deliberately does not cut the requirements to be one requirement per one file because, to my original and still existing view, that breaks the idea of a coherent requirements specification document that can be read and understood as a whole.

After some time, I also implemented a Python library for ReqIF format, and it was very easy to map the concept of StrictDoc's Document to ReqIF's Specification and the concept of StrictDoc's Section to ReqIF's Spec Hierarchy. The library is not coupled to StrictDoc and at some point I extracted it to be a standalone building block: https://github.com/strictdoc-project/reqif.

It needs to support a single "document" (e.g., StRS, SyRS, SyAID, SRS, SAID, etc.) that has requirements spread across multiple repositories in the workspace.

StrictDoc supports including Sections from other files. It should be possible to build a StrictDoc document from a number of contributing sections stored in separate files. StrictDoc does not support including single requirements but that should be doable. In that case, I would like to see a strong rationale/use case for going back to Doorstop's "single req - single file" approach.

StrictDoc is not connected to a single repository concept in any way. You can run StrictDoc against a given folder and it will recursively find all *.sdoc files and collect them into a single documentation tree. Nested folders and submodules are possible. The same should apply to finding the document fragments but I admit I haven't tested this multi-repo case with section fragments yet.

@gregshue, could you elaborate a little more on which repositories do you have in mind and how are you planning to maintain a single document that is spread across several repositories? I can guess that from these child repositories, you will not be cherry-picking single requirements to your parent document but rather you will be including more coherent and self-contained sections with requirements and free text, is it so?

It needs to support "variation management" - tracking moderately different requirement sets for closely related products.

Formulated in this way, it is hard to give a general answer if StrictDoc is capable of doing it but I would say we can certainly talk about this in more detail: how do these variations look like and how StrictDoc could implement this. One way of doing it is certainly by using a system of specialized fields like @RobertoBagnara suggests in his comment. A document can be a result of a query applied to the specification tree, filtering only the needed features.

With regard to this second question, I would also like to add some background information to what @sebhub provided in his comment.

@sebhub and I were looking into using Doorstop and were contributing to it around the same time in 2019-2020. There is a very interesting conslusion by https://github.com/doorstop-dev/doorstop/issues/471 that can shed light into why RTEMS qualification initiative went to building the toolchain in the way that it is implemented now.

Compared to StrictDoc's focus on the single-tree projects where in the end there is one final product, RTEMS qualification effort had to be accomplished for a number of selected hardware platforms which assumed high configurability of the BSP parameters and build settings. The variation management requirement was there from the very beginning and the qualification engineers selected very early a subset of RTEMS of what would become a "profile for space applications". This led to a "bottom-up" approach where the specification items were not centered around the concept of documents but around the already existing source files, directories and functional blocks of the RTEMS operating system. The resulting RTEMS Qualification Data Package is generated from these smaller specification items, and the final documents look like what ECSS standards expects them to look like. A part of this toolchain is that the spec item files (like this one from @sebhub's comment) support a number of special fields that emerged according to the needs of the qualification effort and there is definitely no code in Doorstop or StrictDoc that would be able to implement any of these out of the box.

In the end, StrictDoc as a top-down system can produce Documents from smaller fragments, and the RTEMS toolchain can produce Documents from composable specification items. What matters for RTEMS is that its toolchain is tailored to the RTEMS QDP needs very closely. To achieve something similar with StrictDoc, custom code like that of RTEMS qualification toolchain will have to be written.

Having developed StrictDoc to what it is now and having observed the RTEMS qualification progress as an outsider (The European space industry is largely an RTEMS user), I can say that the bottom-up qualification approach of RTEMS is certainly worth a very close look if Zephyr wants to do a similar qualification exercise. At the same time, I would be happy to support the initiative from StrictDoc tool side if the working group decided to include it somewhere in the Zephyr toolchain.

To conclude, here are a few general bullet points:

Thank you for your attention, and I hope that my input adds value to the discussion.

danwos commented 1 year ago

Hi all,

thanks @stanislaw for mentioning Sphinx-Needs and for the great, detailed answer.

Let me try to introduce Sphinx-Needs, which in my understanding can also be used to solve your documentation requirements for safety and security docs.

First of all, I'm the current maintainer and founder of Sphinx-Needs, which was created ~6 years ago based on the needs of safety-relevant documentation at an Automotive OEM company (following ISO26262).

What is it?

In my understanding, Sphinx-Needs is more of an ALM toolbox for docs-as-code-based documentation, which contains safety-relevant documentation but is also used for architecture docs, test run docs or simple FAQs.

And as each team is following slightly different processes, with different tool and CI setups, Sphinx-Needs is highly customizable to fulfill nearly every dream regarding customization, data validation, and final representation.

Sphinx-Needs is an extension for the documentation generator Sphinx. It uses and supports natively all Sphinx data formats and features. So defining a requirement in rst or md syntax, and adding tables, images, math formulas, and own written solutions, is given by default. This also means Sphinx-Needs objects (like requirements) can be defined everywhere in the documentation: all in one file, one per file, and between other paragraphs. It's up to your own rules how the structuring should look like.

You are also free to decide, what kind of objects you need. Is it a customer requirement, or more an epic with a linked user story? And all of these decisions can be validated by Python based statements (needs-warnings) or even more complex by pydantic models via Sphinx-Modeling, which is an extension on top of Sphinx-Needs.

Another one is Sphinx-Test-Report to get test results into your docs and automatically link them with test cases. The API and data model are completely open, so you are free to inject our own customizations.

With needservices you can integrate external systems, for instance, to fetch GitHub issues and link them to requirements/specs. Or you use the Variant Handling to maintain different options for different targets/customers for the same requirement. And dynamic functions allows the automatic calculation of option values, like status or safety statements based on the linked elements.

How does it look like?

An code example using rst. Starting with basic Sphinx data and then using Sphinx-Needs in the directives .. req and .. spec.

Component A
===========
This site describes our *Component A*. 
For other component please take a look on :ref:`component_overview`.

Requirements
------------

.. req:: My first requirement
   :id: req_001
   :tags: main_example

Specifications
--------------

.. spec:: A specification
   :id: spec_001
   :links: req_001

   A first **specification** with some Sphinx features.

   Here is a list of needed parameters:

   .. csv-table:: Component A parameters
      :file: component_a/param.csv
      :header: "ID", "Name "Type", "Default"
      :widths: 10, 40, 25,  25

A typical need look like this (not related to the above example): image

Where is it used?

Sphinx-Needs is used in some bigger commercial Automotive ECU projects for SW development. The biggest project I'm aware of is ~2.000 SW developers with a single documentation containing up to 130.000 Sphinx-Needs objects in ~22.000 rst/md files (Disclaimer: I and my team are providing commercial support for this).

Some projects are trying to introduce Sphinx-Needs for all steps in the V-Model: image Starting from below (SW dev part) and injecting more and more process steps from the left and right side (SW Arch, SW Testing, System Arch, ...). It's an ongoing process to make Sphinx-Needs available to support also the special requirements on the upper process steps.

What else? There is an online presentation available, which shows most features on some slides: Documentation without frustration presentation. Sphinx-Needs specific part starts here.

Sorry, that I wasn't able to get into your special project requirements and provide already some solutions.

But in my understanding, StrictDoc and Sphinx-Needs are both capable to deal with your requirements. And if not, we hopefully have found some new contributors ;)

gregshue commented 1 year ago

@stanislaw Good questions. I don't have a solution in mind, but I have a pretty clear idea of the problem.

Let's look at this from the perspective of a manufacturer/product developer:

So ...

could you elaborate a little more on which repositories do you have in mind

I'm thinking of any/all of them. We need a requirements solution that scales to:

  1. up to all levels of requirements decomposition;
  2. across all potential repository distributions (e.g., zephyrproject-rtos, downstream OSS, commercial, private);
  3. down to being in the middle of a larger solution.

how are you planning to maintain a single document that is spread across several repositories?

I don't have a proposal. I'm looking for suggestions too.

I can guess that from these child repositories

We cannot restrict these to being child repositories.

you will not be cherry-picking single requirements to your parent document but rather you will be including more coherent and self-contained sections with requirements and free text, is it so?

The solution needs to scale across both usages.

In the end, StrictDoc as a top-down system can produce Documents from smaller fragments, and the RTEMS toolchain can produce Documents from composable specification items.

Fundamentally, a Zephyr application is a selection/configuration of composable subsystems & drivers (components) controlled via Kconfig & Devicetree. It is natural to model the RM for interfaces & components in the same way.

danwos commented 1 year ago

@gregshue thanks for the detailed problem description. Sphinx-Needs together with Sphinx-Collections have some concepts to deal with this kind of decentralized requirement distribution.

First of all, Sphinx-Needs has an exchange format, called needs.json. This can be used to import objects from other projects and have a representation of them in the new project. Or the objects from needs.json are just referenced, so that you can link objects to them, but the external objects have no representation in the new project. Links in this kind of referencing are later real HTML-links to the correct location of the referenced need in the external project.

A simple workflow looks like: image

The above mechanism is good to share objects between Sphinx projects. But it does not solve generic integration problems, where you have for instance standalone projects for components and need an integrated, overall system documentation, which shall include all the content (text, images, tables, ...) from the component projects. For this, Sphinx-Collections was created, which integrates files from different projects into an overall project. Sources can be repositories, local folders, functions, and data provided by self-written functions. Sphinx tags can be used to easily switch between different integration configurations. A build call could look like: sphinx-build -t system_view -b html input/ output/

Some further links:

I hope this already helps to get a feeling about some mechanisms of object-handling between different docs-as-code projects.

sebhub commented 1 year ago

@sebhub Thank you for the pointers. It looks to be very relevant content.

The RTEMS specification graph can be built using any number of directories, ...

From the RTEMS UID definitions it looks like RTEMS requires a (all?) specification(s) to be within a consistent, fixed directory tree. The Zephyr build system learns about the Zephyr module pathnames through the ZEPHYR_MODULES and ZEPHYR_EXTRA_MODULES variables. Any idea about the complexity of inserting a module reference parameter into the UIDs?

Each specification item (for example a requirement) needs a unique identifier (UID). For the RTEMS specification, the UID is derived from the file system path to the file which stores the specification item (in YAML or JSON format). The base directory and the file format extension is removed from the path to form the UID. For example spec/req/root.yml is loaded using the base directory spec which results in the UID /req/root. The specification items are loaded from the file system by recursively searching for files of a given format in a configurable list of base directories. The RTEMS specification and tooling is located in the rtems-central repository and the RTEMS sources added through a Git submodule.

The links in the specification item are unidirectional and have a role. This can be used to make the specification composable. For example, the requirement refinement relationship is stored in the specializing item and links to the more general requirement. So, the higher level parts of the specification do not need to know the specializations. You could for example have a higher level requirement like:

The system shall be extendable by modules.

Then in a specific module you may have this:

The network stack module shall extend the system to provide TCP/IP networking services.

When you build the documents for a given configuration, the links are traversed to compose documents specific to this configuration.

Another feature which helps to make the specification modular are proxy items. They can be used by higher level specification parts to refer to items which may be implemented in an architecture-specific or device-specific way. For example, the Gaisler IP library is available for SPARC and RISC-V systems. The generic part uses a proxy item /dev/grlib/if/store-32 to refer to some method which stores a 32-bit value to a register. This method is provided by an architecture-specific implementation /bsp/sparc/if/grlib-load-32. For a particular configuration, the proxy items are resolved to a specific item.

Another feature is the hierarchical glossary of terms. A glossary of terms is quite important for larger projects. However, some terms may be context-specific. An example are state machines specific to a certain functionality such as timers. For this you can create a hierarchy of glossaries:

The terms defined by the Timer Manager glossary are used in the state machine specifications, for example /rtems/timer/req/reset.

sebhub commented 1 year ago

@stanislaw thanks for your good overview. Yes, the RTEMS specification approach is very bottom up. It was a reverse engineering activity and not the development of a new product. For general requirements engineering I would not store individual requirements in individual files. This is just way to cumbersome and does not help you to discover a good set of high level requirements, user stories, etc. I think these two approaches could be easily combined. In the end, the specification is just a directed graph where each node is identified by an UID. I also belief that each projects needs a bit of specific tooling to be productive. It is important to have simple and open data formats which can be effectively used with Git (like reST, StrictDoc, YAML).

tobiaskaestner commented 1 year ago

IMHO, this issue is a big leap into the right direction for allowing product creators in safety-critical domains adopt zephyr for their use-cases. That said I felt a bit lost in the discussions as it seems that ideas at least around process and actual tools are mixed up. Not saying they do not relate to one another, obviously they do. Still, disentangling the threads might help to focus the discussion.

To sum up before going to call for some tools and rush into action some thought experiments how the process will play out over some time may help not falling into avoidable pitfalls

gregshue commented 1 year ago

some thought experiments how the process will play out over some time may help

Agreed, which is why I the first thing I contributed to #57703 are some needs from downstream users, particularly from product manufacturers. Please feel free to add others that you are aware of.

tobiaskaestner commented 1 year ago

@sebhub I eventually managed to take a deeper look into what you guys are doing at RTEMS and it's really impressing.

Personally, I think it would make a lot of sense if FOSS projects targeting the safety-critical domain would eventually align on a 'standard' approach towards requirements engineering and how to provide a safety case for the FOSS under consideration. Reason being that many notified bodies (those agencies eventually responsible to asses the safety case) will probably have had none to very little exposure to FOSS yet. And the less familiar things look the more scrutiny auditors will put forth and hence whatever could be presented as a 'standard' way may help (even more so if someone would actually testify that the process is sound and apt for FOSS projects)

The RTEMS process and the content you have already produced using it could well serve as a reference for other projects to follow.

I wonder how much experience you have had with the process when it comes to adding updates to design elements that had been captured previously

zephyrbot commented 9 months ago

Hi @simhein,

This issue, marked as an RFC, was opened a while ago and did not get any traction. It was just assigned to you based on the labels. If you don't consider yourself the right person to address this issue, please re-assing it to the right person.

Please take a moment to review if the issue is still relevant to the project. If it is, please provide feedback and direction on how to move forward. If it is not, has already been addressed, is a duplicate, or is no longer relevant, please close it with a short comment explaining the reason.

@romkell you are also encouraged to help moving this issue forward by providing additional information and confirming this request/issue is still relevant to you.

Thanks!