Each driver in individual repos and submodule into this project

[rashedtalukder]

I would like to see all the drivers (all entries in components dir) exist in their own repositories and get submoduled into here.

Request description

There are scenarios where using set( EXTRA_COMPONENT_DIRS is not practical (e.g. using it nested in another component library) and due to the structure of this project, this is a cumbersome process. Tracking any changes to the individual drivers that are being used becomes difficult as well. Additionally, as the number of supported drivers grow, this can make the project more difficult for the maintainers. Breaking it apart would make maintenance easier in the long-term. I suggest using this repository as a cohesive example repo, with the individual drivers existing in their own separate repositories and getting submoduled in to here.

This gives the ability to directly submodule in the specific component into the components directory and have a cleaner project and library level CMakeLists.txt.

[trombik]

(e.g. using it nested in another component library)

would you elaborate your use-case? how you do it and why it needs the proposed change?

this can make the project more difficult for the maintainers

who do you mean the maintainers? probably as I maintain just a few drivers, i don't think it's difficult.

as a maintainer of CI, i have a question:

how do you test in CI when you change something in i2cdev or esp_idf_lib_helpers? i can think of possible solutions, but none looks simple.

[rashedtalukder]

I think it would take a while to explain the "how," but this blog does a great job at being pretty unbiased: https://medium.com/@porteneuve/mastering-git-submodules-34c65e940407

Git submodules are used in the ESP-IDF as well: https://github.com/espressif/esp-idf/blob/master/.gitmodules

So my use-case might be this...I'm creating a BSP that needs only some of the libraries for my device hardware. I want to provide this BSP so that anyone can download it and put it into their project. However, the current way to do it, I would need to put it in something like /components/my-bsp/dependencies/esp-idf-lib/, which is a relatively large download and there are many libraries that will never be linked since they aren't needed.

The life/maintenance-cycle of each driver is now decoupled from the others.

You can run all the unit tests in the individual driver repos, then the integration tests in this repo.

[trombik]

I think it would take a while to explain the "how," but this blog does a great job at being pretty unbiased:

we know how submodule works.

You can run all the unit tests in the individual driver repos, then the integration tests in this repo.

no, you cannot. both of the components i mentioned are depended by other drivers. when you change either of them, you would like to make sure the change does not break other drivers.

[trombik]

again, please describe your use-case.

[rashedtalukder]

My use-case is simple. I only want a slice, not the whole cake. I only want a few of the libraries, not the entire bundle.

On Mon, Nov 8, 2021 at 8:57 AM Tomoyuki Sakurai @.***> wrote:

again, please describe your use-case.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/UncleRus/esp-idf-lib/issues/257#issuecomment-963363650, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACGKTJFQBUZ36SZVK6CEXCLUK76PZANCNFSM5HSZOMNA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[trombik]

if you don't disclose your actual use case, this would be the last answer from me.

I only want a slice, not the whole cake

a short answer: you cannot.

some drivers requires others. they now assume that requirements are there. if you want to change that, each driver needs dependency resolver. In CI, reverse dependency resolver is required. your proposal requires other significant changes as well; development workflows, release process, and CI. as a maintainer of CI, that sounds too much.

having been a sysadmin, i became an ESPer. here is my advice from my sixth sense.

• if downloading is the problem, do not clone the whole repository. use --depth 1
• use cache in CI of your product
• don't bundle the whole repository. automate to cut a slice you want

[rashedtalukder]

I fully agree with you that it will be a significant amount of work to do this for your workflows. I can't state our use-case directly at the moment, but it closely resembles Rop's example that he created here: https://github.com/ropg/m5core2_esp-idf_demo

You can see how there are dependencies between the components (e.g. i2c_manager, m5core2_axp192, and the axp192 libraries). Anyway, I would rather use submodules rather than create my own distribution for proper attribution to the source and ensure that community contributions go back to the original project, not any fork. Supporting the developer community and working together on existing efforts is important to me.

[tuupola]

Use case is developers often do not want to include unrelated code in their projects. I think STMicroelectronics MEMS drivers project is doing what @rashedtalukder described.

Individual drivers could be submodules of the main repo ie. this repository. This way the CI workflow does not require any changes but it would enable developers to cherry pick which drivers they want to install.

Downside is there will be extra work maintaining the main repository because submodule commit hashes need to be updated every time an individual driver has a new release.

[rashedtalukder]

That's actually a fantastic example of a structure that's very easy to consume/comprehend. You are correct that it's initial work to restructure this repository and then to keep the submodule pointers pointing to the correct hash, but I think the value is very much there.

Additionally, some of the licenses of the libraries included here are GPL, which for some organizations will cause issues if that license is uploaded to a centralized VCS and the license is detected in automated scans.

[trombik]

create my own distribution for proper attribution to the source

this cannot be a justification as each source code file has a proper license header. if not, it's a bug.

Use case is developers often do not want to include unrelated code

submodules are created NOT because they are unrelated but because development can be individually made. in addition. as I mentioned, some code are related.

Additionally, some of the licenses of the libraries included here are GPL

we don't have GPL code.

Downside is there will be extra work maintaining the main repository because submodule commit hashes need to be updated every time an individual driver has a new release.

this is the reason why i wrote "none looks simple". by using the proposed repository management, we will have many sub repositories, and each has its own CI and release process. the parent repository will have an integration CI process. development will require additional steps to follow. users will have to manage dependency. what is the benefit? for whom? apparently, the PR author will benefit from it. I have not heard of any complaints from other users so far.

let's go back to the claimed problems:

set( EXTRA_COMPONENT_DIRS is not practical

I haven't seen any example or proof.

Tracking any changes to the individual drivers that are being used becomes difficult

most of the drivers are very stable.

as the number of supported drivers grow, this can make the project more difficult for the maintainers.

ask @UncleRus. he does almost everything.

if you have teams of developers, multiple projects, and a release team, submodule makes sense. we don't have that. it is @UncleRus that does most of the hard work. I'm not a good coder, so my role is to help him. the proposal will not. if you disagree, please make a proposal for changes to development, CI, and releasing, and explain how simple it is.

this is my opinion, not a final verdict.

[UncleRus]

I have already pondered this question many times, but have not come to clear conclusions. The disadvantages of the mono-repository are obvious, @rashedtalukder listed them correctly (IMHO the main disadvantage is the inclusion of esp-idf-lib in another library as a submodule). However, I am still stopped by the additional hassle associated with it.

If you consider submodules the right way, then help us:

• correctly separate the components (may be leaving some core components like i2cdev in the root repository?)
• describe possible use cases of esp-idf-lib, which is divided into submodules for both developers and users
• develop an appropriate CI

I do not plan to do this all alone in the near future, but I will gladly and gratefully accept the help of the community.

[UncleRus]

@rashedtalukder

Additionally, some of the licenses of the libraries included here are GPL, which for some organizations will cause issues if that license is uploaded to a centralized VCS and the license is detected in automated scans.

Please indicate the components containing the GPL code.

[rashedtalukder]

Please indicate the components containing the GPL code.

I mis-wrote. I thought there were GPL libraries included here, but I might have been misremembering.

[rashedtalukder]

I think the place to start is to define what this project is, and is not. I think that's your decision, @UncleRus.

Full disclosure, I haven't spent a lot of time (or almost any) thinking about it for this repository. If this was a project that showcased usage of the libraries as a number of demos, you could keep i2cdev as a separate repository with it's own lifecycle. You could have a light test mock setup for any devices that used i2c (or other deps). This mock should pass the same tests as i2cdev, and i2cdev should pass tests on physical hardware. Now, with that out of the way, the test themselves could bring in the dependencies just for the sake of the tests using CMAKE FetchContent or even External Project in the test CMakeLists.txt. This way the driver libraries stay independent from each of the other libraries, except for testing, and you potentially never need to run them on physical hardware.

[trombik]

the issue boils down to supporting IDF Component Manager, and integrating it into the development process.