Allow modification of managed component directories, skip hash sum checking (PACMAN-687)

[higaski]

Related area

missing feature regarding codegen

Is your feature request related to a problem?

The component manager needs a feature to allow components to be modified without complaining about every minuscule thing. The prime example of why such a feature is important is code generation. Not only would I consider "in source" (so next to other actual source files) codegen to be quite common, but in some languages its even mandatory (Dart).

Describe the solution you'd like

my_component:
  version: master
  modifiable: true
  git: https://github.com/higaski/my_component

Describe alternatives you've considered

No response

Additional context

No response

I have checked existing list of Feature requests and I have provided enough information.

• [X] I confirm I have checked existing list of Feature requests and I have provided enough information.

[kumekay]

@higaski Thank you for opening the discussion,

Can you please elaborate a bit your example? Is generated code needed to be persisted, or will it be regenerated automatically every time? Do you have an example component that behaves like this?

If you want to keep generated code, you can move the component to the components' directory (as the error message suggests)

If code generated every time, the only option available now is to set IDF_COMPONENT_OVERWRITE_MANAGED_COMPONENTS=1, but in this case, component manager will remove all generated code on every run of CMake. We can make this setting per-component, but behavior will stay the same. Making sure that the component in the managed components' directory is always the same as in the lock file is one of the design goals of the tool. Otherwise, it won't know whether dependency is outdated or corrupted.

If you want a behavior when component manager will download components with modifiable: true only once, then it's the same as running CMake once and moving a component from managed_components to components.

[higaski]

I'm sorry I hadn't fully figured why that's happening when I initially created the issue. So what apparently does happen is that the component manager happily checks out my component which includes some CMake codegen using a custom command. This custom command leads to additional files getting generated during the build phase. The project builds, everyone is happy. The problem only occurs if for whatever reason CMake gets re-run. Sadly this happens quite often, e.g. a simple git commit... well... actually touch CMakeLists.txt is enough to re-run everything. The component manager now finds the new files and isn't too thrilled about them being there.

  ERROR: Some components (my_component) in the "managed_components" directory were
  modified on the disk since the last run of the CMake.  Content of this
  directory is managed automatically.

Moving the components to some other unmanaged directory is not an option.

[kumekay]

@higaski Thank you for explanation, For now, you can use IDF_COMPONENT_OVERWRITE_MANAGED_COMPONENTS=1 as a quick workaround.

As for a proper solution, it's not obvious whether something like modifiable: true should be set by a user of the component, or by the maintainer. The fact that the component can create some files in the source directory should be known beforehand and to my standpoint, it's better to free the end user from the hassle.

[higaski]

Ok thank you. I've added the environmental variable IDF_COMPONENT_OVERWRITE_MANAGED_COMPONENTS to my build.

Just for the record, I disagree with the rest, but let's not get too far into that. :grin:

[nicklasb]

 @kumekay

My reasons for this is that I develop a component that:

1. Is becoming quite large and complex. (https://github.com/RobustoFramework/Robusto)
2. Has a lot of dependencies.

I cannot use "Local development" (moving it to "components") as that disconnects the dependency management in some ways, which holds everything together, every time i have to temporarily change the dependencies of the project to not have it load both. Local mode is too local, basically.

And as I work, I frequently need to make changes in dependent libraries to create different contexts or test fixes to them to be able to make PR:s. Radiolib would be the typical example for me.

I really find no reason to not allow for directly changing the source of the components during development, that is only important in distribution of the component(s). To not be able to do this, not only makes me lose 20 minutes each time, but it forces me to sometimes make pointless commits and pushes to the framework to test stuff out.

On the flip side, it should be noted, the component manages doesn't alway detect changes, and sometimes I can make the error going away by just building again. I usually use it via PlatformIO, but I don't think that should matter.

So some way of disabling this check would be great. Either altogether or via more detailed parameter in idf_component.yml. Problem with having it there would be that people might forget to remove it when distributing the component...

[kumekay]

Hello @nicklasb I apologize for the slower turnaround. Could you please describe how is moving a component to the components⁣ directory disconnects it from dependency management? All the dependencies of these components are still processed and kept in managed_components, so it should only affect a single component you are moving.

The component manager is made in such a way that it decides whether it needs to download a component based on hash sum, so there is no “check” to be disabled to achieve the desired behavior. Quite the opposite, the check can be disabled by setting IDF_COMPONENT_OVERWRITE_MANAGED_COMPONENTS, but then it will always make sure that the component is the same as required by the manifest and overwrite changes unquestionably.

We can add support for some file markers, like the presence of .ignore_me file in the managed component, but then the behavior of the component manager will be essentially the same as to copying it to the components directory.

On the flip side, it should be noted, the component manages doesn't alway detect changes, and sometimes I can make the error going away by just building again. I usually use it via PlatformIO, but I don't think that should matter.

Could you please help us to describe the steps to reproduce this behavior? We are not aware of this problem yet.

[nicklasb]

So developing the framework that I use it in my test projects through dependency management and when I do this, because it is about communication and that becomes a lot of projects, basically.

If I instead move the component to /components everywhere, then I have have to need to change the idf_component.yml of the project to stop recreating the component in /managed_components.

To me, this is normally about testing different things, like temporarily adding extra logging, not only in my components but in many of its dependencies, to be able to exactly follow without debugging (debugging affects the processes and timings).

So I want to make a whole bunch of changes anywhere to find the problem, and then just take the fix and apply it upstream. And after that I want the checks and dependencies to start working and updating again, because then I want to test that my new code works when distributed. Also I don't want dependencies to update while i am tracking the error, quite confusing when that happens.

Could you please help us to describe the steps to reproduce this behavior? We are not aware of this problem yet.

Actually I don't want to help you with that as that is the only thing that keeps me from going crazy. :-) Also, all I know is that sometimes, after it detects the change, I can just hit "Verbose test" or whatever PIO thing I want to run again, and then it builds. And if I then accidentally runs menuconfig, it goes out of this state and then I have to either commit all these small changes to the branch and load again or remember.

I just want an easy way to without a lot of handling, It doesn't have to be official, just some hack to break the check temporarily.

[kumekay]

@nicklasb If after moving to components it keeps recreating in managed_components it looks like a bug, it should ignore it. I'll try to reproduce it tomorrow. Just in case, do you have set(COMPONENTS ...) in your project's CMakeLists.txt? (There is a bug which may cause such behavior, we are working on it)

[nicklasb]

Well, in my view, it should recreate it. If I don't remove the dependency, why wouldn't it? Either way, again that is not what I want to do as I just want to freely mess around in all the code in managed_components. That is works by moving it to components is of little use to me.

But I see no principal reason why that handling cannot be just disabled. Especially since you have the IDF_COMPONENT_OVERWRITE_MANAGED_COMPONENTS, which seem like a much more odd setting to have than having a disable setting. Which is like the most common setting of any component anywhere.

[higaski]

Have you even tried the IDF_COMPONENT_OVERWRITE_MANAGED_COMPONENTS option? I think it does exactly what you're looking for. I've edited files in one of my managed components yesterday and those changes were kept.

[nicklasb]

Have you even tried the IDF_COMPONENT_OVERWRITE_MANAGED_COMPONENTS option? I think ot does exactly what you're looking for. I've edited files in one of my managed components yesterday and those changes were kept.

According to Espressif above it does basically the opposite of what I am looking for and you repost, as it always overwrites the files. And while it at has happened that it hasn't, because I have even tried it (not sure how I have deserved that), that is not as that is just too weird to trust that it will fail. And at some points it did what it said anyway so.

My problem is that the build fails. I wan't to disable the check without relying on that IDF_COMPONENT_OVERWRITE_MANAGED_COMPONENTS doesn't seem to overwrite sometimes because that is worse than the build failing sometimes.