search

zephyrproject-rtos / zephyr

Primary Git Repository for the Zephyr Project. Zephyr is a new generation, scalable, optimized, secure RTOS for multiple hardware architectures.
https://docs.zephyrproject.org
Apache License 2.0
9.91k stars 6.1k forks source link

Example downstream manifest+module repo in zephyrproject-rtos #28438

Closed carlescufi closed 3 years ago

carlescufi commented 3 years ago

Create a new repository in the https://github.com/zephyrproject-rtos organizationt that can serve both as reference and as a starting point for users wanting to create their own manifest+module repo (later this can be extended with additional sample module repos).

https://github.com/zephyrproject-rtos/zephyr-downstream-application which would have:

We could then move https://github.com/zephyrproject-rtos/zephyr/tree/master/samples/application_development/out_of_tree_driver/hello_world_module/zephyr to the new repository.

carlescufi commented 3 years ago

This will require integration with our CI system to make sure we don't break downstream modules and applications, either as a new GitHub Workflow or in BuildKite.

mbolivar-nordic commented 3 years ago

self-assigned at least for any west instrumentation changes needed to get this working in CI on each PR

gregshue commented 3 years ago

For clarification,is this an example of what could be done, what is expected to be done, or what will be supported for the mid- to long- term?

mbolivar-nordic commented 3 years ago

A driver configurable with Kconfig

... and dts, I assume?

tejlmand commented 3 years ago

For clarification,is this an example of what could be done, what is expected to be done, or what will be supported for the mid- to long- term?

imo this will be a repo that can be used to ensure that Zephyr module features for out-of-tree are working can be tested by CI so that it is ensured that it work as expected. At the same time, it can serve as a simple reference point for downstream users.

That way we ensure:

gregshue commented 3 years ago

@tejlmand, I agree with the need to support CI verification of Zephyr module features. I think this needs to be more than a "simple reference point" of how those features can be used. I want to know how the module is expected to be setup - in practice, a boilerplate. Extending my downstream modules as Zephyr module features get added is fine (and expected), but restructuring it to take advantage of a new feature is highly undesirable.

Since we need to support extending (potentially forks of) existing external repositories to be zephyr modules (e.g. github.com/project-chip/connectedhomeip), I propose:

gregshue commented 3 years ago

A couple more:

gregshue commented 3 years ago

Please remember that the board "unit_testing" requires different build logic in the test files, so the module interface functionality verification needs to include an example of that.

mbolivar-nordic commented 3 years ago

I think this needs to be more than a "simple reference point" of how those features can be used. I want to know how the module is expected to be setup - in practice, a boilerplate.

@gregshue I'm not sure I get you on this one. I don't think we have expectations of our users in the form of "if you don't structure your module in this way, you're doing it wrong" in this case. I am assuming @tejlmand means "simple reference point" as "here is a way to do it that we think works well and can commit to maintaining". (Please LMK if I misunderstood that.)

So a simple reference point is a boilerplate for a "typical" application with an in-tree SoC, but out of tree board, application, and driver for an IC on the board that doesn't have an upstream driver.

Is it possible we are in violent agreement on this?

mbolivar-nordic commented 3 years ago

@gregshue my take on the other bullets:

All(?) the zephyr unique files be kept under the necessary zephyr sub-directory, following the zephyr repo directory structure (e.g. manifests, those discovered by sanitycheck/CI, hello_ sample, west extension scripts, Kconfig files).

-1 from me.

Flat is better than nested in this case, in my opinion. Users can always nest if their use case is suited for that.

Shared namespaces (e.g. include pathnames, CPP symbols, Kconfig vars, etc) be prepended with the module repository org/name, where dashes are replaced with underscores. (It's really too long, but we need to avoid namespace conflicts.)

I think it could be nicer to 'steal' some short names that are unlikely to conflict. I expect most users will change these anyway, to suit their own project naming conventions.

Karim Yaghmour does this in his Embedded Android book's chapters on adding board support for that operating system. He names the board the "coyote pad", and defines namespaces and files based on that. Microsoft does something similar with the fictional "Contoso" corporation in some of their publications. I personally find this practice easy to understand. It's just "foo" in another form.

Some specific ideas:

Test case names would use these as well.

It would be straightforward to tell the story of the development of the new Plank device and its killer application in the documentation. This can also say that internally, the Plank includes an SoC already supported by Zephyr and the VND corporation's chippy sensor, which is not supported upstream. Etc.

Automatic formatting rules are provided for all files beneath the zephyr sub-directory structure (e.g. .clang-format). This will simplify file comparisons against the boilerplate.

Here be dragons; this is far from a settled question in the project, even just for line length.

I suggest that instead, we just make sure the linters that run on code in main zephyr repository also pass on this new module. We can discuss formatters if and when Zephyr itself starts to use them.

A repo-level .gitignore be commented with why each setting is necessary, since some may conflict with pre-existing build tools.

+1

Names/prefixes of tags and the main development branch and tags are recommended to be the manifest repository name. Since this is a manifest+module repository the main development branch should be renamed to the name of this repository. (This means there will not be a branch named "master", and a "zephyr" branch likely will not be meaningful or useful.)

While I am personally in favor of avoiding a main branch named "master", I think this is a project-wide question (which I believe the TSC is already discussing) and this module's main branch name should follow what the zephyr repository uses, to be changed if and when zephyr itself does.

A module version and manifest version schemes, policies, and query/discovery mechanisms should be set up, since the zephyr version is no longer the manifest version.

I don't know that I agree with a module-wide version right now; can you explain why that is worth having? I agree that a version numbering mechanism for killer-app makes good sense, though.

gregshue commented 3 years ago

@mbolivar-nordic , I had not equated "simple reference point" with "boilerplate". With your clarification I think we are in agreement. Thanks for sharing your understanding. (BTW, I never assumed approaches other than this boilerplate were "wrong", just that the downstream maintainer assumes responsibility for designing/implementing/verifying support as module interface features get extended. :-))

Regarding your comments above:

If we don't set up a pattern to manage namespaces, et. al. at scale and beyond the scope of GitHub, then we have not addressed the crux of the problem. Similarly, if we haven't provided an example of how to extend existing repositories to be modules, then we also have not addressed the crux of the problem.

In my experience reuse-and-extend is a much more powerful enabler and paradigm than leverage-and-potential-reuse. I expect users to not rename anything from zephyrproject-rtos. It is much more efficient to live within the existing naming and reuse all applicable analysis and code provenance than to re-apply unnecessary changes over the long support/maintenance lifetime of IoT products (est. 10+ yrs). For boards, samples, drivers, applications (samples/testcases), and device tree settings, there is no need to "rename" anything. ("Renaming" implies removing the original.) Zephyr has already been designed to support extensions for each of these. You can easily copy from an existing implementation of any of these to create an additional one (whether in-tree or out-of-tree), and leaving the original intact makes it easier to track updates and apply relevant fixes.

Here be dragons; this is far from a settled question in the project, even just for line length.

By providing the same .clang-format file as exists in Zephyr, users can choose to enable, change, or disable automatic formatting as they like. It's no different than currently in the zephyr repo. :-)

While I am personally in favor of avoiding a main branch named "master", I think this is a project-wide question (which I believe the TSC is already discussing) and this module's main branch name should follow what the zephyr repository uses, to be changed if and when zephyr itself does.

I have been participating in Process Working Group meetings and monitoring TSC discussions for several weeks, so am aware of the topic. The recommendation for the main branch name to be 'zephyr' came from the PWG, and it makes sense that all existing modules (which happen to be upstream of zephyr) have a main/integration branch name of zephyr. This issue is now introducing downstream modules, which necessarily provide their own manifest. Naming the downstream module's main branch as 'zephyr' doesn't set up a scalable pattern. But recommending that main integration branch names follow the manifest repo name does scale within at least GitHub, and fits with the existing TSC preference. Part of the reason for expressing it here is to help the TSC evaluate their current preference. :-)

I don't know that I agree with a module-wide version right now; can you explain why that is worth having? I agree that a version numbering mechanism for killer-app makes good sense, though

As I see it, one of the huge benefits is Zephyr is a Software Product Line. Zephyr Project is set up for reuse-and-extension, and some of that reuse will come from open source repos outside of zephyrproject-rtos (e.g. Project CHIP). In an SPL, a FW image is simply a set of build configuration settings and a few image-unique values. The vast majority of logic is or will be shared with at least one other image. Building out of the same source branch reduces maintenance across the whole SPL. Consider the case where a user wants to build dozens of different FW images for the same class of from the same branch of code. Or they need to support the same implementation over SOCs from a variety of vendors.

What does that have to do with module version numbers? Anyone wanting to integrate a downstream module into their product will want to use a released version for code stability, issue tracking, and eventually partial analysis for security certification submissions. In this case we are setting up a boilerplate for a downstream module, so we should provide a versioning mechanism tied into the Zephyr ecosystem.

Cheers!

gregshue commented 3 years ago

@mbolivar @tejlmand @carlescufi After looking into how the Zephyr build system currently extracts version information, it seems like the following is all that is needed to tie downstream manifest modules into the Zephyr build system version extraction mechanisms:

These will reveal places where we have assumed ZEPHYR_BASE is also the manifest repository (which is no longer true), and drive distinction back into the zephyr repository.

gregshue commented 3 years ago

Considering module names (and associated namespaces), it would be beneficial if all zephyrproject-rtos downstream modules were easily identified within the name. Perhaps github.com/zephyrproject-rtos/**oot_***, with this specific repository being named oot_boilerplate or oot_template?

gregshue commented 3 years ago

@tejlmand We will also need to verify the solution also supports the Forest topology, where the manifest repo contains no business logic. I expect this will require a separate OOT manifest(s)-only repository with:

Perhaps this additional repository should be named oot_manifest_only?

tejlmand commented 3 years ago

@tejlmand We will also need to verify the solution also supports the Forest topology, where the manifest repo contains no business logic. I expect this will require a separate OOT manifest(s)-only repository with:

Note, the main point of testing in this case is to ensure that Zephyr modules works. West and west workspace topology should be tested by west, and not Zephyr.

This is done here: https://github.com/zephyrproject-rtos/west/tree/master/tests

gregshue commented 3 years ago

@tejlmand The Zephyr build system also needs to be verified to work in different topologies, notably when zephyr is not the manifest repository. This is true whether the manifest repository has any prj.conf or is configured from an alternate manifest file. Since Zephyr explicitly supports the Forest topology separate from the Star topology (w/o Zephyr as manifest), an example of a manifest-only repository seems appropriate. This is especially important if the overall FW version or BUILD_VERSION should map to a distinct manifest.

tejlmand commented 3 years ago

The Zephyr build system also needs to be verified to work in different topologies, notably when zephyr is not the manifest repository.

@gregshue you seem to miss the point that west is an optional tool. https://docs.zephyrproject.org/latest/guides/modules.html#without-west

What this issue should focus on is that the Zephyr module infrastructure works correctly.

That is, ensuring ZEPHYR_MODULES works. One way to obtain a list of modules, is by using west list, and therefore it makes sense to combine this into a repo that can also work as a simple reference point, as that will serve two purposes:

Doing testing of other typologies is therefore outside the scope of this.

tejlmand commented 3 years ago

In this case we are setting up a boilerplate for a downstream module, so we should provide a versioning mechanism tied into the Zephyr ecosystem.

Zephyr is already calling into the module's CMake build system. That build system can decide how it wants to handle versioning. It could decide to use a VERSION: https://github.com/nrfconnect/sdk-nrf/blob/master/VERSION

or it could decide to fetch the version info using git (if available), e.g.: git describe --tags

If we start imposing a version field in the module.yml, then we force users to update that field. As long as Zephyr itself doesn't use the field, I see no reason to introduce this, and definitely not as part of this task. (Feel free to open a new enhancement suggestion for this/)

Considering module names (and associated namespaces), it would be beneficial if all zephyrproject-rtos downstream modules were easily identified within the name. Perhaps github.com/zephyrproject-rtos/oot_*, with this specific repository being named oot_boilerplate or oot_template?

I don't think this is the right issue for such discussion.

These will reveal places where we have assumed ZEPHYR_BASE is also the manifest repository

west is an optional tool, so Zephyr knows nothing about being a manifest repo.

Shared namespaces (e.g. include pathnames, CPP symbols, Kconfig vars, etc) be prepended with the module repository org/name, where dashes are replaced with underscores. (It's really too long, but we need to avoid namespace conflicts.)

I think it could be nicer to 'steal' some short names that are unlikely to conflict. I expect most users will change these anyway, to suit their own project naming conventions.

agree with @mbolivar-nordic here. Especially as usersin some cases might want to uses same names as another module in order to replace/overrule that module. As example, they want to use the same set of MBEDTLS Kconfig names as Zephyr does, as this allows them to override default menus/default settings.

gregshue commented 3 years ago

@tejlmand , Thank you for your feedback. I absolutely agree that ZephyrProject specifies that west is optional.

What this issue should focus on is that the Zephyr module infrastructure works correctly.

I think you have recognized a general need that is tangential to this ticket. This ticket is about creating an example downstream module with simple business logic and covers the breadth of points of extensions and is a suitable pattern for extending existing 3rd-party open source repositories (e.g. https://github.com/project-chip/connectedhomeip , https://github.com/bacnet-stack/bacnet-stack, https://github.com/micro-ROS/micro_ros_zephyr_module).

As such, this ticket will set precedence (and likely direction) for the following:

Why consider versioning? Because PROJECT_VERSION* does not refer to the image built from an out-of-tree prj.conf.

gregshue commented 3 years ago

I have created a proposed implementation for this example at https://github.com/gregshue/oot_module_example. It does not have a board, driver, or dts implementation, but provides the structure and documentation to hold them. I would submit this as a PR but the Zephyr Project repository has not yet been created.

I will post an RFC for this proposal below.

@carlescufi @tejlmand Is there anything else needed to move this along?

gregshue commented 3 years ago

Introduction

Establish an out-of-tree module example (boilerplate) repository demonstrating how to extend open-source repositories to also be Zephyr out-of-tree modules.

Problem description

Users will be creating and integrating open source and private out-of-tree extensions to the Zephyr ecosystem. To be successful the Zephyr support for out-of-tree integration needs to be more formally documented, continuously verified, and one or more examples provided and maintained. No example currently exists.

Proposed change

Create a feature-rich example/boilerplate repository for out-of-tree modules which provides the following in a natural application and extension of existing Zephyr patterns:

Detailed RFC

Existing external repositories have structures and build systems that are probably not aligned with the Zephyr build system. These repositories must provide a top-level zephyr subdirectory to be recognized by the Zephyr build system as a module. This is a natural place to keep Zephyr integration files isolated from the rest of the repository and for users to start debugging system integration. This subdirectory introduces a level of indirection that affects the following parts of the current Zephyr repository patterns:

There are also a few changes needed as Zephyr is no longer the manifest repository. These include:

All of the above proposals are for patterns that apply to all current and future Zephyr Project originated OOT modules, and are recommendations for OOT modules originated elsewhere. The specific values for the first example also need to be addressed.

Proposal: This specific example repository is named 
`oot_module_example` and uses the abbreviation `OMX` 
(Out-of-tree Module eXample) or `omx` in the above namespaces.

Dependencies

All of the above has been developed and verified in a non-Zephyr Project repository using west 0.8.01a and dependent on Zephyr v2.3.0. No changes were necessary, but build system support broke in Zephyr v2.4.0.

The support of out-of-tree modules impacts every team to varying degrees. A few teams must maintain parallel implementations in each repository (e.g. Documentation generation, build files, release notes, CI processes).

Concerns and Unresolved Questions

There are a few open issues that eventually need to be address:

Alternatives

The option of addressing only this one example repository was considered, but the first example will set de-facto standards and precedence. Ignoring scalability of the example was considered too large of a risk to the project.

Options with fewer parallels were considered but that increased the chance of inconsistencies across OOT modules, making the integration effort more difficult.

gregshue commented 3 years ago

I noticed two edits are needed in the RFC:

Proposal: module.yml defaults map to the Zephyr directory tree
placed at the location of the module.yml file.

For backwards compatibility module.yml defaults need to map to the directory above the one containing the module.yml file.

Concerns and Unresolved Questions

Add the following concern:

gregshue commented 3 years ago

@carlescufi @tejlmand This issue needs a new repository within zephyrproject-rtos. The proposed name is oot_module_example.

carlescufi commented 3 years ago

@gregshue @tejlmand @mbolivar-nordic @nashif @MaureenHelm @galak

Let's settle on the name first, I am not very happy with oot_module_example.

Here's a few proposals:

Essentially, do we want to call this an application or a module? or something in-between? And also, is this a sample or an example?

I personally favor sample-oot-project because:

gregshue commented 3 years ago

@carlescufi @tejlmand @mbolivar-nordic @nashif @MaureenHelm @mglb

Based on the description of this issue and the need to verify all out-of-tree mechanisms, this repository will eventually contain all the following as a clone-able and copy-able/rename-able boilerplate for user repositories:

It is never expected that this repo represents the only way to support modules. Rather, it provides a maintained template that may be copied onto an existing repository to extend it into a Zephyr module, and the changes tracked by users as Zephyr evolves.

Having worked through the preliminary versions (privately and publicly), this is no more or less of an application than zephyr is (since every sample and test is itself an application according to Zephyr documentation). We are really dealing with an (OOT) extension plugging into the Zephyr build system. This seems closest to how Zephyr uses the term module, and it is well beyond the scope of how Zephyr uses the term application.

The term project is heavily overloaded already and already causing challenges in CMAKE variable naming. If we introduce project for a repository, then what will developers conclude when they hear the phrase "Zephyr Project"?

Without knowing the difference between a sample vs an example, I would prefer separate terms at the repository name vs a folder name. (It will already be bad enough when we consider the zephyr repository being detected by the Zephyr build system as a module itself. This would end up with the mounted file zephyr/zephyr/module.yaml.)

Edited addition: Repository/module names will be abbreviated (really need <= 4 chars). These will need to appear in the various namespaces. Beware of offensive abbreviations. Consider things like allowing 'x' as a letter abbreviation for eXample.

gregshue commented 3 years ago

Please consider this also. At least one existing Zephyr module (Civetweb) is purely at the application layer, uses a communication protocol generally unsuitable to resource constrained devices (HTTP/S), and probably has Kconfig variables independent of the rest of the Zephyr tree. If it were to evolve into an OOT manifest+module (downstream of the zephyr repository instead of upstream), what would be the desired repository name with zephyrproject-rtos?

gregshue commented 3 years ago

Hmm ... from here I found the following distinction between example vs sample:

2.One difference between the two concepts is in the purpose of their use. An example’s purpose is to illustrate or reinforce something. Examples are made to be followed and replicated. On the other hand, a sample is made to describe the quality and nature of something.

It seems this repository is scoped (and desired) to be an example, not just a sample.

mbolivar-nordic commented 3 years ago

@gregshue I read your comments and it seems like you don't like the name suggested by @carlescufi , but I don't see a concrete suggestion for a replacement. Did I miss one?

gregshue commented 3 years ago

@mbolivar I had already suggested oot_module_example, which @carlescufi didn't like.

Since I had wrestled with this naming pattern issue before, I just wanted to pass along what I had already discovered. :-) What are most important to me is:

For clarity reasons I had avoided project, application, external, sample. and repo. It turns out that example fits the repository purpose and intended usage better than sample.

At this point we're really figuring out a repository naming scheme that scales well and makes sense. :-)

mbolivar-nordic commented 3 years ago

OK, thanks :). I get it now. So summarizing where we are at on the naming:

I agree with @carlescufi that this is more than just a 'module'. Actually I find "OOT module" a bit redundant. A module, pretty much by definition, is out of tree.

I agree with @gregshue that 'project' is a bit of an overloaded term in general, but disagree in this context. In west terms, 'project' is a well-defined concept that captures what this issue describes making, and I don't have a more specific term in mind.

I don't have strong feelings about 'sample' vs 'example', but I lean softly towards 'sample' just because it keeps consistency with 'zephyr/samples', which for better or worse is what our example applications are inside.

So all in all I agree with @carlescufi 's proposal.

gregshue commented 3 years ago

I'm surprised to hear this described as more than just a 'module'. What does this provide that is beyond the existing defined scope of an out-of-tree 'module'?

Though the west definition of 'project' maps cleanly to this, neither the CMake definition, Zephyr build variables PROJECT, nor the usage of "Zephyr Project" do. All of them are well defined concepts that need to be clearly managed and configured within this module scripts and files.

I wasn't aware of the difference between 'sample' vs 'example' until prompted by @carlescufi 's question. I know samples get referenced in documentation and maintained as part of the CI system, but samples are intentionally simplistic and short. These are necessarily much more complete to act as a boilerplate or templates of best practice coding patterns.

@mbolivar One point of clarification: Zephyr already distinguishes between OOT and in-tree modules by where the glue code is located and managed. An OOT module has the Kconfig & glue code outside the zephyr repository. Either kind of module is always a repository external to zephyr even if it is hosted at zephyrproject-rtos.

mbolivar-nordic commented 3 years ago

Greg Shue notifications@github.com writes:

I'm surprised to hear this described as more than just a 'module'. What does this provide that is beyond the existing defined scope of an out-of-tree 'module'?

It's not externally maintained, and it's a west manifest repository.

Though the west definition of 'project' maps cleanly to this, neither the CMake definition, Zephyr build variables PROJECT, nor the usage of "Zephyr Project" do.

This is restating the premise "the term 'project' is overloaded", which is already granted.

All of them are well defined concepts that need to be clearly managed and configured within this module scripts and files.

This doesn't seem to follow from the premise; does it?

What about including the word 'project' in the name of this git repository prevents managing and configuring those things?

I wasn't aware of the difference between 'sample' vs 'example' until prompted by @carlescufi 's question. I know samples get referenced in documentation and maintained as part of the CI system, but samples are intentionally simplistic and short. These are necessarily much more complete to act as a boilerplate or templates of best practice coding patterns.

I don't have a dog in this fight. Either is fine with me, slight preference for consistency, like I said.

@mbolivar One point of clarification: Zephyr already distinguishes between OOT and in-tree modules by where the glue code is located and managed. An OOT module has the Kconfig & glue code outside the zephyr repository. Either kind of module is always a repository external to zephyr even if it is hosted at zephyrproject-rtos.

To be honest, I don't agree, and I don't think any of the definitions of "Zephyr module" in the documentation agree either.

"In-tree module" is an oxymoron in this context.

gregshue commented 3 years ago

@mbolivar-nordic , perhaps I misunderstood the definition of a Zephyr 'module'. Here are the things that led to my impression:

There were other documentation statements that contradicted these. Given the existence of the OOT samples, references to modules defining out-of-tree boards, and the explicit directions on how to add modules of my own, I assumed the contradictory statements are simply out of date.

Terms and definitions are important for clear communication. What did I miss or misunderstand?

gregshue commented 3 years ago

@mbolivar-nordic , Thank you for your comments. I should have said an "in-tree module" is one where the Kconfig for any of the module variables are within the zephyr repository (e.g. zephyr/modules/Kconfig.civetweb). Though "in-tree module" is not used in the documentation it is the natural opposite of out-of-tree (a term used several places).

tejlmand commented 3 years ago

have been a bit busy, but let me give my comments.

Actually I find "OOT module" a bit redundant. A module, pretty much by definition, is out of tree.

I completely agree. I would like to avoid this term.

There were other documentation statements that contradicted these. Given the existence of the OOT samples, references to modules defining out-of-tree boards, and the explicit directions on how to add modules of my own, I assumed the contradictory statements are simply out of date.

yes, I guess this stems from the days where Zephyr was a single repo, and as such everything else was out-of-tree. With the introduction, and especially the root extension, then I think we should try to cleanup the usage of out-of-tree. Maybe use Zephyr module defined boards, and similar.

Terms and definitions are important for clear communication. What did I miss or misunderstand?

Completely agree. Sometimes features are introduced, but old terms stays around, even if they starts to be a bit imprecise. So we should clear up those places.

As has also been discussed #28583, I believe it is better to refer to Zephyr module, and then reserve out-of-tree to something that is completely outside the Zephyr workspace (and maybe things not a Zephyr module).

For anything related to a Zephyr module, let's use the Zephyr module term

so my proposal on this:

  • @carlescufi proposes sample-oot-project

  • @gregshue proposes oot_module_example

would be:

as it is an example of how a Zephyr module can be created. If there will be more example repos, then they can be named accordingly, like zephyr-module-<topic>-example

but I would not prefix other Zephyr modules with this. That is, hal_nordic, hal_stm32, etc should not be prefixed with zephyr-module-, because the fact that those repos are modules is not important. What is important is their purpose, which is to provide hal code.

The purpose of: zephyr-module-example is to showcase how a Zephyr module can be constructed, hence having zephyr-module in the name adds value to the understanding of what this repository does.

gregshue commented 3 years ago

I can live with zephyr-module-example.

I suppose zephyr-manifest-example would be the name of a non-module repo that contained the west manifest file(s), top-level documentation, and project files (.prjconf, etc) that simply integrated all the Zephyr Project-hosted modules? (I expect we will need this to verify that zephyr-module-example can be integrated by an external manifest/project.)

tejlmand commented 3 years ago

I expect we will need this to verify that zephyr-module-example can be integrated by an external manifest/project.

That's a good question. Seen from educational point of view, then it would probably be easier to explain to user / easier to understand:

but from build integration point of view, it technically doesn't matter if the Zephyr module is also a manifest repository. Cause the build has no knowledge of the concept of a manifest file, it just get a list of projects to examine.

gregshue commented 3 years ago

@carlescufi @mbolivar-nordic , Any input?

gregshue commented 3 years ago

I think we will end up needing one repository of each of the following scopes in order to have appropriate test coverage. We should end up with a naming pattern that fits these scopes.

  1. A non-module manifest repository containing manifest file(s), top-level documentation content and generation scripts, version information, release notes. Possible names:

    • zephyr-example-manifest

  2. A module only for projects integrating other modules downstream of zephyr, in addition to all the content of (1). This would not contain any top-level CMakeLists.txt or Kconfig files, no include/ or subsys/ folders, and only tests for verifying multi-module integrations. Possible names;

    • zephyr-example-module-integrations

  3. A module that is an example of all the extension capabilities of a Zephyr module. This would contain all the files and content of (2) and also contain include/, subsys/, unit and subsystem tests, and boards. Possible names:

    • zephyr-example-module

Thoughts?

gregshue commented 3 years ago

@tejlmand @carlescufi I can't tell if discussion has stalled or come to a natural conclusion. What else is needed to create the repositories and have them ready for PRs?

cfriedt commented 3 years ago

Is anyone else is working on this? I would be happy to take a crack at it - need to do the same anyway for finishing the Greybus for Zephyr module within the next couple of days.

gregshue commented 3 years ago

At this point it is stalled waiting for alignment on the repository naming pattern, then for the empty repositories to get created to enable a PR with my draft content.

@cfriedt, I just looked at the Greybus for Zephyr staging branch. Before you release this to production you should look at https://github.com/gregshue/oot_module_example.

cfriedt commented 3 years ago

@cfriedt, I just looked at the Greybus for Zephyr staging branch. Before you release this to production you should look at https://github.com/gregshue/oot_module_example.

Ah, interesting - I'll take a look at that. I just hacked something together last night independently of the Greybus repo. It looks similar-ish to yours with the exception that you have everything within the zephyr directory.

https://github.com/cfriedt/zephyr-example-module

I'll take a look at your repo as well.

gregshue commented 3 years ago

@cfriedt Everything Zephyr-unique is kept beneath the zephyr/ subdirectory to allow extending any existing repository to also be a Zephyr module simply by copying this in and filling in the glue. In order to avoid a top-level west.yml I had to use west version >= 0.8a1. I expect my CMakeLists.txt files could be improved. :-)

I applied a non-documented version of this in the zephyr branch of https://github.com/bacnet-stack/bacnet-stack and migrated all the unit tests to ztest. sanitycheck nicely ran with code coverage metrics.

Thanks for sharing, and let me know how to improve mine.

carlescufi commented 3 years ago

Proposal for CI:

  1. The west.yml in the external repo points to a release tag of zephyr, and the repo itself is only tested against that last release, on-PR to the external repo. Who would be responsible for updating the external repo to the latest release when an -rc1` is tagged?
  2. We integrate building the external repo in our on-PR CI in the main zephyr tree, by adding something similar to: 
    
zephyr-example-module/west.yml:
projects:
- name: zephyr
  remote: zephyrproject-rtos
  revision: master
  import: true
self:
 path: example-repo

west init -m https://github.com/zephyrproject-rtos/zephyr-example-module west update west build app


Since we do **not** want the external repo to be part of upstream zephyr's `west.yml` (that would be "cheating"), we need to figure out a strategy of keeping the revision of the last known working external repo revision somewhere, or just testing the external repo `master` branch against zephyr's `master` branch.
Note: the workflow for contributors will be more complicated if the external repo build breaks.
galak commented 3 years ago

My suggestion for a name would be either example-application or example-project

galak commented 3 years ago

dev-review (Dec 17 2020):

gregshue commented 3 years ago

@galak Right now the Zephyr directory tree structure has conf.prj files (arguably "applications") under tests and samples. Would it make sense to add a parallel projects tree for non-test, non-sample conf.prj et. al., to live?

gregshue commented 3 years ago

Is it appropriate for this repository to also be a regressive test case to verify that the west -m <> --mr <> and --mf <> options continue to work?

nashif commented 3 years ago

Just for illustration of the various use cases:

1) the upstream model:

image

2) a project (product tree) that is a zephyr module as well:

image

3) a project (product tree) that is not a zephyr module:

image