Workspace profiles

[paulbovbel]

Following from https://github.com/colcon/colcon-core/issues/52, it would be fantastic to have a system similar to catkin-tools profiles, which define build settings at the workspace level.

The command line arguments defaults mechanism looks like it provides most of the interesting options I'm familiar with from catkin-tools except one - --extend. With catkin-tools, it was extremely useful to be able to statically configure what the current space overlays, in a way that would persist through clean operations, and not be affected by a user's environment. Would you have any recommendations at what layer of colcon it would make sense to implement something like that?

[dirk-thomas]

Having such a feature would indeed be great and improve usability. I don't have a specific recommendation where and how it should be implemented. Before that can be determined it needs to be figured out how it should behave and interoperate with other similar features like:

• global defaults
• metadata
• mixins
• additional command line arguments

[paulbovbel]

How do those features currently interoperate? Is there a precedence order, or is there something more complex required for mutually exclusive arguments?

[dirk-thomas]

Metadata read from files can be used to set the metadata of packages directly or amend information like dependencies. Since metadata can't be affected through command line arguments it shouldn't matter for the order.

1. Global defaults are read from a file specified in an environment variable and set the default value for the argparse arguments.
2. Passed command line arguments are used to populate the argparse.Namespace. With argparse explicitly passed arguments will override the default values of step 1.

3. For mixins the semantic depends in the type of the argument:

   • For list values the mixin items are being prepended to the existing list in the argparse.Namespace.
   • For other values (bool, int, float, string) the value in argparse.Namespace is being overwritten when it is the default value. If the argument has been explicitly passed through the command line (even if the value is the same as the default value) it isn't changed and the mixin value is being ignored.

[mikepurvis]

I appreciate the effort that has gone into the explanation above and in the previous thread, however, I'm still having trouble wrapping my head around all of these features and the various ways they interact. The ideal scenario from my point of view (as a part-time stakeholder in tooling decisions for a medium-size development team) would be:

• Something I can install on everyone's computer which universally sets certain defaults in every newly-created workspace.
• Something I can ship with a given workspace bundle so that when that workspace is extended, the extending workspace inherits certain global defaults (thinking CMake args, especially around optimization level and arch-specific flags), as well as a suite of package-specific flags.
• All of this is exposed to users in a reasonably transparent way, so that they are empowered not just to override any particular setting, but also to understand how it got to be that way in the first place.

I don't want to have to set COLCON_HOME and have it end up pointing somewhere other than the expected location in the user's home directory. It would certainly be possible to set COLCON_DEFAULTS_FILE, especially if it was part of the env for a particular workspace to set that file pointing to workspace-specific defaults.

However, I do wonder if it might be more coherent to just make my own internal colcon-banana-defaults package which sets the options the way I want them and is able to add a supplemental option to --metas such as --banana-metas that points by default into the directory of package-specific metas used for a particular workspace, and then just make that package part of our standard setup— we may well still end up using the mixins capability to organize our flags, but the initial setup of them could be handled programmatically by the plugin rather than as something that has to be handed down to individual users ("be sure to manually add mixins X, Y, Z on every new workspace"). Does this make sense as a way forward?

[paulbovbel]

around all of these features and the various ways they interact

Me too! My naive understanding is that a 'persistent profile' a-la catkin-tools would sit in the order of precedence:

• global defaults
• ~metadata~ seems unrelated
• persistent profile
• mixins, and additional command line arguments (these seem to be basically equivalent, since mixins just resolve into args)

Something I can install on everyone's computer which universally sets certain defaults in every newly-created workspace.

That sounds like a good use for COLCON_DEFAULTS_FILE

Something I can ship with a given workspace bundle so that when that workspace is extended, the extending workspace inherits certain global defaults (thinking CMake args, especially around optimization level and arch-specific flags), as well as a suite of package-specific flags.

Ooh, now that's fancy, but totally outside the scope of what I had conceived. This sounds like it would be closely tied to your custom distribution method - is there a place to inject something like this in the wider hierarchy of colcon plugins?

[mikepurvis]

This sounds like it would be closely tied to your custom distribution method - is there a place to inject something like this in the wider hierarchy of colcon plugins?

I mean, our bundles by and large map into the larger ROS concept of a "distro", with the main caveat being that we identify them by tags and tag-like refs in our rosdistro repo rather than directories, so we have thousands of them (including many feature-branch ones) rather than just a few. The important part of putting build metadata into that asset would be ensuring that it's versioned alongside everything else— we don't want to have some outside authority on metas and CMake flags that can change and then mess up the consistency of older builds.

Without having given it too much more thought, my concept would be a utility package similar to cmake_modules, whose only purpose is to carry the global flag mixins and any metas that let us get out of patching upstream source (G2O_BUILD_EXAMPLES=0 and the like). There'd be some fancy tricks to get the flags into the initial bootstrap, but after that, it would basically just register some kind of hook that would make it findable by the "main" colcon plugin— whether that was an environment variable pointing to itself, or even just something that would show up as importable in the PYTHONPATH (edit: upon consideration, this would probably not be workable due to messing up situations where colcon and the build are not sharing a Python environment, for example if colcon is setup in a virtualenv).

It's likely that most of the logic for this could go in the colcon plugin side, and then it would be fairly easy for anyone else to supply their own data package for it. I'll give it some thought and see about knocking together a prototype to understand better how it would work.

[dirk-thomas]

Something I can install on everyone's computer which universally sets certain defaults in every newly-created workspace.

It sounds like putting those options into the defaults.yaml file or setting the environment variable COLCON_DEFAULTS_FILE to point to custom file solves the technical part within colcon. (I will just ignore the aspect how you could deploy that without your team since that doesn't seem to be a colcon specific question.)

Something I can ship with a given workspace bundle so that when that workspace is extended, the extending workspace inherits certain global defaults (thinking CMake args, especially around optimization level and arch-specific flags), as well as a suite of package-specific flags.

I am not sure if that is generally desirable. I would even expect that as soon as you use two sibling underlays you will quickly run into a situation where those inherited flags may conflict with each other.

Are these flags / options different from the above defaults? If yes, could they be declared in a mixin and the user actively decides to use that mixin? At least that would be the already available building blocks.

Obviously a custom extension could write information to a file in the install space as well as read that file from underlays to set different default values for overlays.

[dirk-thomas]

My naive understanding is that a 'persistent profile' a-la catkin-tools would sit in the order of precedence:

Technically I think only two things are missing:

• Profiles could be specified as simple as in a profiles.mixin file in the root of the workspace.
• A way to select a specific profile / mixin by name and make it the currently used one.

On top of that the questions are how to expose this in a nicer way to the user? E.g. how to pass a specific option and persist it in the current profile?

If there is any case around profiles which can't be satisfied with the existing technical building blocks those should be described in detail and being added to the existing building blocks (rather than implementing special logic for profiles).

[cottsay]

I'm fairly confident this was implemented in colcon/colcon-defaults#29. Please feel free to re-open if you believe otherwise.