Adopting "product line" terminology

[gregshue]

Introduction

Zephyr Project is already delivering much more than "an RTOS". In fact, it is already delivering:

• a set of software-intensive systems (samples, tests, example-application)
• sharing a common, managed set of features (protocol stacks, MCU boot, drivers, etc)
• that satisfy the specific needs of a particular market segment (resource-constrained IoT devices)
• and that are developed from a common set of core assets (subsystems, drivers, modules) in a prescribed way (Kconfigs, DTS, overlays)

This matches the Software Engineering Institute's definition of a "software product line". Since the early 2000s, this has been recognized as a best practice for supporting reuse across similar products, and training materials have been freely available. This practice seems to be cost-effective when a manufacturer needs to simultaneously deliver 3-4 similar products. This threshold is very reachable by IoT device manufacturers.

In ISO terminology this is referred to simply as a "product line", which prefixes several related terms relevant to what the Zephyr Project is delivering and what the end users are developing.

This proposal is to adopt the "product line" terminology so that:

• we are clearly communicating what the Zephyr Project is providing to the greater community
• we express our solution in ways to more easily recognize applicable "best practices"
• we are demonstrating continued commitment to our value statement

Problem description

Currently we have no clear way of completely describing what Zephyr is, and we have inconsistent visions of what it should be. This severely hinders our ability to measure and achieve "best in class", needed to fulfill our Mission Statement.

Proposed change

This proposal is to adopt the "product line" terminology so that:

• we are clearly communicating what the Zephyr Project is providing to the greater community
• we express our solution in ways to more easily recognize applicable "best practices"
• we are demonstrating continued commitment to our value statement

Detailed RFC

This is just a terminology/documentation change, so the immediate functional impact should be non-existent. However, it should clarify best practices for how to pull the entire ecosystem into a more consistent, composable, extensible solution.

We can represent current Zephyr organization and activities with the following diagram:

Here are some notable quotes from the SEI training material Software Product Lines, Course Introduction:

the key attributes [of an SPL] are:

1. Use of a core asset base ... {Architecture}
2. in the production ... {Production Plan}
3. of a set of related products. {Scope Definition, based on a Business Case}

Software Product Lines are NOT

• Fortuitous small-grained reuse
• Single-system development with reuse
• Just component-based or service-based development
• Just a configuration architecture
• Releases and versions of single products
• Just a set of technical standards

Important Terms:

• Core asset: A reusable artifact or resource that is used in the production of more than one product in a software product line. A core asset may be an architecture, a software component, a requirements statement or specification, a document, a plan, a test case, a process description, or any other useful element of a software production process.
  • Core asset base: The complete set of core assets associated with a given software product line.
  • Domain: An area of knowledge or activity characterized by a set of concepts and terminology understood by practitioners in that area.
  • Product: Deployed software or software-intensive system.
  • Reuse: Using an item more than once.
  • Software product line practice: The systematic use of core assets to assemble, instantiate, or generate the multiple products that constitute a software product line.
  • Strategic reuse: Planned, systematic reuse that implements tightly connected business and technical strategies.

When we compare these terms to Zephyr we see a strong overlap:

• Core Asset: the zephyr/ repository provides many of these.
• Core asset base: the zephyr/ repository provides a subset of these for downstream reuse.
• Domain: the zephyr/ domain is 32b/64b MCU devices, particularly those needing an RTOS and those targeting secure, safety-critical, and/or IoT usage.
• Product: Zephyr does not provide a "product" because it does not provide hardware.
• Reuse: Many of Zephyr's subsystems and the kernel are designed and implemented for direct reuse.
• Software product line practice: If we relax the definition of "product" to not need deployment, each sample is a "product" and each Zephyr release demonstrates following a software product line practice.
• Strategic reuse: In the case of Zephyr Project, the business and technical strategies are set by the Zephyr Governing Board and the Zephyr Technical Steering Committee, respectively. The Zephyr initiatives, decisions, and investments demonstrate planned, systematic reuse.

So, Zephyr is almost an SPL ...

The SEI research and training material was from the late 1990s and early 2000s. Since then the practice has evolved to be known as Software and Systems Product Line (SSPL) practices. The ISO/IEC 2655x series of standards for software and systems product line engineering and management. A further specialization of this practice was standardized in ISO/IEC 26580:2021 Software and systems engineering — Methods and tools for the feature-based approach to software and systems product line engineering. The associated terminology has similarly evolved. Much of this comes from formally separating the Domain-related work from the Application-related work.

Here is a diagram using the Software Product Line Model and terminology from ISO/IEC 2655x, ISO/IEC 26580:

The only difference is the names.

ISO/IEC 26580 introduced a new term that aligns very well with what Zephyr provides:

product line platform. (1) product line architecture, a configuration management plan, and domain assets, enabling application engineering to effectively reuse and produce a set of derivative products (ISO/IEC 26550:2015 Software and systems engineering--Reference model for product line engineering and management, 3.18)

So ... Zephyr seems to be accurately described as a product line platform.

Proposed change (Detailed)

This proposal is to adopt the "product line" terminology so that:

• we are clearly communicating what the Zephyr Project is providing to the greater community
• we express our solution in ways to more easily recognize applicable "best practices"
• we are demonstrating continued commitment to our value statement

There are several relevant terms worth adopting (definitions from SEVOCAB):

• product line platform == zephyr repository (manifest + build/doc system + "module") + modules:
  product line architecture, a configuration management plan, and domain assets, enabling application engineering to effectively reuse and produce a set of derivative products

• domain engineering == contributions to zephyr and zephyr modules: reuse-based approach to defining the scope (i.e., domain definition), specifying the structure (i.e., domain architecture), and building the assets for a class of systems, subsystems, or applications

• domain architecture == Zephyr ecosystem APIs + KConfigs + guides: core architecture that captures the high-level design of a software and systems product line including the architectural structure and texture (e.g. common rules and constraints) that constrains all member products within a software and systems product line

• domain assets == subsystems/drivers: output of domain engineering life cycle processes that can be reused in producing products during application engineering

• domain scoping == part of the current responsibilities of the Zephyr TSC: subprocess for identifying and bounding the functional domains that are important to an envisioned product line and provide sufficient reuse potential to justify the product line creation

• commonality == Zephyr framework ++: set of functional and non-functional characteristics that is shared by all applications belonging to the product line

• variability mechanism == Kconfig + DTS + ...: variability implementation methods in a product line for supporting assembly of domain assets

• application engineering == end user integrations with Zephyr ecosystem: life cycle consisting of a set of processes in which the application assets and member products of the product line are implemented and managed by reusing domain assets in conformance to the domain architecture and by binding the variability of the platform

Dependencies

This will not immediately impact any designs or implementations. This should help clarify the evolutionary direction of the ecosystem as a whole, and identify further best practices to be applied.

Concerns and Unresolved Questions

None known.

Alternatives

This is proposing aligning on standard terminology, so no alternatives were considered.

[carlescufi]

Architecture WG:

• @henrikbrixandersen states that, in his experience, he has never seen these concepts used in embedded software development, and they might be overkill in our context
• @kartben thinks that the technical documentation might not be the right place to document this, because of its verbosity. Instead this could live outside the technical doc, in a white paper for example
• @gregshue explains that the documentation changes he proposes already map the IEC standard to Zephyr
• @nashif objects to the way we are mapping an ISO standard to Zephyr, the way the terminology is being applied to Zephyr
• @nashif also objects to the relevance that the changes are giving to Zephyr-specific tooling and functionality like modules
• @gregshue will add to the RFC the mapping between the standards and Zephyr itself
• @jfischer-no suggest moving it to its own page instead of being part of the introduction.
• @stephanosio, like @kartben, suggests a white paper: "Zephyr from the ISO/IEC 26580 perspective"
• @fabiobaltieri asks why is Zephyr not an RTOS, @gregshue mentions that it can be a platform for bare-metal development
• @rodrigopex suggest reusing parts of the PR text while reworking others
• @henrikbrixandersen states that the PR removes clarity instead of bringing it. @carlescufi agrees
• @nashif argues that the fact to the standard is not publicly available, @carlescufi agrees
• @nashif and @henrikbrixandersen state that the Problem Description is not really describing a problem. @carlescufi agrees
• @gregshue states that Zephyr maps well to ISO/IEC 26550. @cfriedt mentions whether we need certifications for it, @gregshue states that no certifications are required, the standard is useful for its concepts and terms

[gregshue]

[the concepts] might be overkill in our context

That is a healthy concern. Many usages of Software and Systems Product Lines fit squarely in our context. Consider some of the Hall of Fame usages recognized at the Software and Systems Product Line Conference.

The SPLC is:

an international collaboration that has resulted from the merger of two former events: SPLC (which started in 2000 in the USA) and Product Family Engineering (PFE, which started in 1996 in Europe). The SPLC steering committee leads three SPLCs: SPLC-Europe (formerly PFE), sponsored by ESI; SPLC-Americas, sponsored by the SEI; and SPLC-Asia, sponsored by the Korea Software Institute.

The freely available Software Engineering Institue training material specifically mentions very successful results in commercial usage of SPL (including the following directly targeted by Zephyr Project):

• mobile phones
• avionics systems
• pagers (Motorola)
• engine control systems (Cummins, Bosch)
• mass storage devices (LSI Logic)
• printers (HP)
• consumer electronic products
• medical devices

objects to the relevance that the changes are giving to Zephyr-specific tooling and functionality like modules

The impact of Zephyr specific tooling, processes are core to downstream users being able to reuse external code (e.g., HALs) and to segregating their proprietary content from the flow of upstream changes. These are important enablers and should be documented and managed as such. The module mechanism is a huge benefit for extensibility and composability, both key capabilities for downstream adoption and necessary for a successful SPL.

a white paper: "Zephyr from the ISO/IEC 26580 perspective"

Good suggestion, though it would be ISO/IEC 26550. ;-) The relevant content was already presented at ZDS 2021 in the Device Management min-session.

the standard is not publicly available

The SPL concepts are publicly available in the SEI SPL free training materials, and the standard terms are publicly available. This RFC is not proposing we claim compliance with the standard, but just standard terms. (The terms have evolved a bit over the last 20+ years.) If preferred we can use the terms from the SEI training material.

[gregshue]

Description has been extended with diagrams and content in response to the above Architecture WG feedback.

[gregshue]

The Architecture WG discussion raised significant concern about quoting the technical language used in definitions from the standards. Zephyr is targeting users around the world that may be developing safety-critical and secure connected products. Where has Zephyr documented the range of technical competencies expected of its target users and its target contributors?

[gregshue]

In response to https://github.com/zephyrproject-rtos/zephyr/pull/57527#issuecomment-1541671860:

This proposal is not to align with a standard. It is just to adopt a standard term to accurately describe what Zephyr is. The concepts come from the freely available SEI training materials, which contain many hundreds (if not thousands) of slides in PDFs covering concepts and practices. The smallest SPL introduction deck is 24 slides. The relevant content is not hidden behind a paywall.

If this had no strategic or architectural direction impact then I completely agree that changing terminology is noise. Adopting this will actually help us fulfill the ZP mission statement of delivering "best-in-class" RTOS. Note that manufacturers are evaluating Zephyr against commercial options which already support these practices for safety-critical and secure device development.

Adopting SPL/SSPL/Feature-based PEL terminology will (among other things):

• reframe our dialogs to being focused on the product developer
• further guide architectural decisions toward how patterns scale and components work together
• prioritize reuse and extensibility
• help us not break what has already been enabled

From the SEI Software Product Lines, Course Introduction deck:

Organizations use product line practices to

• achieve large-scale productivity gains
• improve time to market
• maintain market presence
• sustain unprecedented growth
• achieve greater market agility
• compensate for an inability to hire
• enable mass customization
• get control of diverse product configurations
• improve product quality
• increase customer satisfaction
• increase predictability of cost, schedule, and quality

More Examples of Product Line Benefits - 1

• Hewlett Packard (HP) - printer systems
  • 2-7x cycle-time improvement (some 10x)
  • sample project
  • shipped 5x number of products
  • that were 4x as complex
  • and had 3x the number of features
  • with 4x products shipped/person
  • The products from HP’s Owen Firmware Cooperative were produced using 1/4 of the staff, in 1/3 of the time, and with 1/25 the number of bugs

Summary: Organizational Benefits

• Improved productivity
  • by as much as 10x
• Increased quality
  • by as much as 10x
• Decreased cost
  • by as much as 60%
• Decreased labor needs
  • by as much as 87%
• Decreased time to market (to field, to launch...)
  • by as much as 98%
• Ability to move into new markets
  • in months, not years

Many of these benefits will come to ZP as well - with alignment and commitment to these practices.

[tejlmand]

If this had no strategic or architectural direction impact then I completely agree that changing terminology is noise.

So what you're saying is that adopting your proposal will have strategic and architectural impact. But that is not what you argued here: https://github.com/zephyrproject-rtos/zephyr/pull/57527#issuecomment-1538934186

All I am doing is using standard terminology to describe what Zephyr already is and has been for a few years. AFAICT, nothing here changes how Zephyr is already used, how it is developed, or what it fundamentally is.

So which is it ..... no impact and thus just noise or a change to how Zephyr is developed and what it is ?

[gregshue]

So what you're saying is that adopting your proposal will have strategic and architectural impact.

Yes. It will be a stronger commitment to sustain what has already been established and evolve it towards even more reuse and extensibility. This is consistent with the direction things have been going and match the objectives of many portions of Zephyr, but gives more priority to maintaining and continuing to evolve other parts with those characteristics and strategically design them to work together. Hence, no change in direction.

no impact and thus just noise ...

It will introduce a change in thinking about how proposals fit in the aggregate ecosystem, which may change how to evaluate proposals in architecture and process. Does this change how Zephyr components are implemented? No. Does it change what components are implemented? Possibly.

and what it is?

Fundamentally it is design and implement for strategic large-scale reuse. In other words, adopting a "reuse-and-extend" paradigm instead of a "leverage-and-potential-reuse" paradigm.

The easiest way to start understanding the topic is to read all 24 slides in SEI's Introduction to Software Product Lines material. Here is their keystone definition to get started:

What Is a Software Product Line? A software product line is a set of software-intensive systems sharing a common, managed set of features that satisfy the specific needs of a particular market segment or mission and that are developed from a common set of core assets in a prescribed way.

[tejlmand]

It will introduce a change in thinking about how proposals fit in the aggregate ecosystem, which may change how to evaluate proposals in architecture and process.

so if it changes thinking then it also impact how things are developed, so your earlier comment was not correct then: https://github.com/zephyrproject-rtos/zephyr/pull/57527#issuecomment-1538934186

[gregshue]

This adoption of terminology impacts things no more or less than each of the following:

1. Introduction of unenforced coding guidelines
2. Introduction of a schema-validated module.yml
3. Introducing of an unenforced desired level of code coverage.

Each of these things clarified how the project will achieve attributes already implicit in the project's mission statement - without requiring a change in behavior or current implementation. Clarification of concepts impacts development at a high level (e.g., systems-level thinking, requirement elicitation, SDLC and architectural patterns) while enforcement impacts development at the lower levels (e.g., subsystem-level thinking, specifications, processes and design/implementation/verification cases).

Adoption of product line terminology is simply a case of more formally expressing what is informally present and aligned with the implications of the project's mission and vision statements. It brings clarity (impacts high level), but does not bring any enforcement (in itself does not impact the low level). The amount of impact depends on which level you have in view.

[gregshue]

@tejlmand It's time to stop this diversion about whether or not I contradicted myself.

This discussion needs to be about:

1. How well do you think the SPL concepts and practices representing the current state of Zephyr?
2. How well do you think the SPL concepts and practices fit meeting the project's Mission and Vision statements?

Which requires sharing:

3. How far have you learned about SPL from the SEI free material or the free LF course Software Engineering Basics for Embedded Systems?

[rodrigopex]

The following text I have picked from Wikipedia:

_"Software product lines (SPLs), or software product line development, refers to software engineering methods, tools and techniques for creating a collection of similar software systems from a shared set of software assets using a common means of production."_

@gregshue SPL (Software Product Line) is a software engineering study area. We can drive an SPL using Zephyr. For example, a company with several devices can use Zephyr and its tools to make software variations to achieve the whole device fleet's needs with less effort. Are we on the same page?

Questions for you:

1 - Do you intend to make this community call Zephyr a "Software Product Line" that provides an RTOS, among other things? 2 - Are you trying to bring the benefits of SPL to Zephyr by considering the technique?

IMHO, you won't change the perception (of years of work) of the whole community based on a PR. So, you may need to focus on another direction. By using SPL techniques, we can improve Zephyr for sure. So, I suggest you provide some Zephyr improvement based on the SPL with a concrete PR instead of trying to make the community adopt the term.

[gregshue]

@rodrigopex Yes, we are on the same page.

is a software engineering study area

... that has matured enough that standards have been created to guide the usage.

1 - Do you intend to make this community call Zephyr a "Software Product Line" that provides an RTOS, among other things?

Not quite. Zephyr is not delivering products, so in itself it is not a Software Product Line. As you pointed out Zephyr can be used by a company to drive that company's SPL. Thus, Zephyr fits ISO/IEC 26580's definition of a "product line platform". I intend this community to better understand what we already have and how it fulfills end user needs by thinking of and calling Zephyr a "product line platform".

2 - Are you trying to bring the benefits of SPL to Zephyr by considering the technique?

Yes ... AND I am trying to keep the Zephyr Project from making (more) architectural decisions that break the downstream users that are already using it in this way.

The community has struggled for years trying to describe Zephyr, so their perception is already different than the terms they are familiar with. I am pointing out that, when viewing Zephyr from the perspective of some manufacturers, Zephyr is seems to align with a term from an industry standard. This PR finally triggered the discussion I tried to kick off with my presentation at ZDS 2021.

I suggest you provide some Zephyr improvement based on the SPL with a concrete PR

I have multiple times, and they get seem to get rejected because of values and priorities are not aligned with SPL practices. SPL is not a set of low-level techniques (e.g., MISRA C rules). It is a set of principles and practices for organizational management, technical leadership, and technical contributors to follow that produce the previously mentioned large-scale reusability, et. al.

Currently the Zephyr Project is not far off of SPL practices, but it has some critical gaps. The Zephyr Project inevitably will transition to requirements-driven development for the auditable portion of the code. At that point it will need to capture or make an assumption of end-user needs to produce requirements for that part of the codebase. At that point the needs for large-scale reuse, extensibility, and composability (all key to SPL) will be undeniable and unavoidable. We can delay the topic until then if you prefer. Meanwhile we have a good window for the audience to learn about it in some detail.

[rodrigopex]

Only a suggestion. As a first step, you could quickly review the topic to share with the contributors. After that, starting to write some best practices blog posts (or similar). And so on. We need (including me, of course) to be familiar with the topic to discuss more about it. SPL will help Zephyr on several fronts. So let's prepare everyone here for the undeniable and unavoidable time of need. I guess going over the course is too heavy and time-consuming for the majority here. So small steps are very welcome. What do you think? @gregshue

[nashif]

now that we have discussed this in the arch WG and we finally have comments in an RFC, want to put this on table for the TSC to review and decide how we want to proceed with this and if the project shall adopt this terminology.

[gregshue]

@rodrigopex Good suggestions.

The SEI courses on SPL are substantial to say the least. I have seen one of their many SPL courses take 11 hrs of YouTube. I have not taken their courses ($K), but I have frequently used the slide decks as reference material.

There are already a number of free resources available:

1. Watching the 30m ZDS 2021 presentation Field Report: Setting up a Software Product Line (SPL) Architecture which provides an overview and application of these concepts and practices on top of Zephyr.
2. Going over some of SEI's free materials:
   • Introduction to Software Product Lines 24 slide deck is the easiest way to get a brief introduction.
   • Software product Lines Essentials is a bit meater at 85 slides.
   • and for the very ambitious: Software Product Lines Course Introduction at a hefty 513 slides!
3. Taking the free self-paced Linux Foundation course Software Engineering Basics for Embedded Systems - LFD116 estimated (for those with CS backgrounds) to take 4-6 hours to complete. The last section introduces SPL using SEI material as well as the ISO/IEC 2655x and ISO/IEC 26580 standards.

I expect that is enough to get people started. Blogs and a whitepaper are also reasonable.

Given the Zephyr Project is sponsored by the Linux Foundation and the LF commissioned LFD116, at some point the ZP TSC should have a prepared statement related to SPL. This conversation should help us figure that out. ;-)

[nashif]

TSC: no motion raised in the TSC to adopt this terminology.

[nashif]

Closing as this was not pushed forward to be adoped by the project.