workspace detection from catkin_tools

[wmmc88]

Feature: detect workspace root in a catkin workspace built with catkin_tools

Background https://github.com/Noam-Dori/ros-integrate/issues/52#issuecomment-716621973 Right now, the plugin uses a .catkin_workspace file generated by catkin_make to determine where the workspace root is. Most people use catkin_tools now and the tool doesnt generate a .catkin_workspace file, so the automatic workspace detection doesnt work.

Details The plugin could pick up on a catkin_tools folder in the same place as where the .catkin_workspace file is for catkin_make workspace. The plugin could use this instead of the .catkin_workspace to get the workspace root path.

[StefanFabian]

This would indeed be a great addition. I don't know why so many of these tools do it for catkin_make which has been deprecated for years. The folder is called .catkin_tools, though.

[Noam-Dori]

First of all, this is something I want to do, and have it completed by next release of this plugin. This is why it is already triaged and has the 0.1.7 milestone.

I don't know why so many of these tools do it for catkin_make which has been deprecated for years.

The official ROS website only documents catkin_make for all of its tutorials, I had to learn about the existence of catkin_tools from third party sources. At this point, catkin_tools itself is deprecated in favor of colcon.

So to determine the "sub-feature list" of this feature I need the following info:

• in regards to catkin_tools:
  • A big deal with this tool is that it saves configurations. Where can I find documentation on what the configuration directory contains? I looked and found nothing...
  • What should the user be able to control within the catkin_tools configurations if anything?
• in regards to colcon:
  • Does this tool have a configuration folder like catkin_tools? if so, what is it called?
  • What does this folder store if it exists?
  • just like before, what can the user control in this folder?
• should the user be able to switch between the build tools? Should the configuration folder be deleted when switching or should there be two of them in the same palce?

Of course, I might need more information in the future.

[StefanFabian]

The official ROS website only documents catkin_make for all of its tutorials, I had to learn about the existence of catkin_tools from third party sources. At this point, catkin_tools itself is deprecated in favor of colcon.

Ah, that explains a lot. Very unfortunate. colcon is the build system for ROS 2 so it's not really that catkin_tools is deprecated as it is still the main build system for ROS 1. However, as noetic is officially the last ROS 1 version, in the long run, it will probably lose relevance.

Catkin tools allows to create profiles with e.g. cmake, make or catkin make args, build, devel and source space and whitelisted / blacklisted packages. You can view the configuration using catkin config but apart from the settings none seem relevant for this plugin.

I have to say, though, that configuring from CLion is a very nice-to-have feature and in my opinion, the most important step for the next release would be basic support.

• should the user be able to switch between the build tools? Should the configuration folder be deleted when switching or should there be two of them in the same palce?

I don't think that anyone uses both of these build tools. It's normally one or the other.

[Noam-Dori]

Do you have a technical reference (or an example directory I can look at) of the files .catkin_tools has and what they contain? The plugin needs to be able to read the directory's content and write in user set configurations

[StefanFabian]

Unfortunately, not. I don't really understand what user configurations you want to write in there?

[wmmc88]

Unfortunately, not. I don't really understand what user configurations you want to write in there?

Perhaps he wants to generate clion build configurations automatically based off a workspace's catkin profile configurations? Each catlin profile can define make args, catlin specific make args, cmake args, devel vs install space setup, blacklist/whitelist etc

[Noam-Dori]

Exactly, I want the plugin to present the configurations in a GUI, then those will be written into the profile file

For example (I don't know how the directory is actually built) let's say that each profile has a name and is stored in this directory as that file

.catkin_tools
- local.yml
- docker.yml

The .yml file could be structured like this:

whitelist: [pkg_1,pkg_2]
cmake_args: "--message hello"
...

So in a UI CLion could display those two options, one as a list that can be edited and one as a text bar, and possibly add more things. When OK is clicked in CLion, it would edit the corresponding profile. It could also allow you to create new profiles from CLion if that's a thing.

I am just very unfamiliar with how catkin tools works, which is why both user input and a technical input are important.

the most important step for the next release would be basic support.

What do you consider as "base support"? All the plugin currently offers for catkin_make is automatic population of one plugin specific configuration

[StefanFabian]

I would consider base support that the workspace is recognized and packages from the workspace are found, though that doesn't really require support for catkin_tools workspaces. Perhaps also that you can build using catkin but that would I think already be a step ahead of existing plugins like Hatchery.

So in a UI CLion could display those two options, one as a list that can be edited and one as a text bar, and possibly add more things. When OK is clicked in CLion, it would edit the corresponding profile. It could also allow you to create new profiles from CLion if that's a thing.

You could do that but it's normally not the case that you change the profile often enough that it would justify the development efforts imho. I personally have changed it maybe four times in two years and that is probably more frequent than most people in my workgroup.

[wmmc88]

Do you have a technical reference (or an example directory I can look at) of the files .catkin_tools has and what they contain? The plugin needs to be able to read the directory's content and write in user set configurations

Heres some documentation about it: https://catkin-tools.readthedocs.io/en/latest/mechanics.html

I'd also say that personally I swap between profiles quite often. I configure profiles for a dev build that's more relaxed(generates debug symbols, doesnt treat warnings as errors, etc). And then I also have a profile that mimics CI for some of our projects (treats warnings as error, runs clang tidy, builds in release mode, etc).

[Noam-Dori]

I would consider base support that the workspace is recognized and packages from the workspace are found, though that doesn't really require support for catkin_tools workspaces.

This was already implemented in previous versions and was a very big part of the development of this plugin. Once of the neat things about this is that you can put any path as the workspace directory in the settings and the plugin will find all packages in it (basically any file containing a package.xml that the user did not explicitly filter out)

The extra "base" stuff I want to add is the automatic detection which will automatically set this plugin configuration

Heres some documentation about it: https://catkin-tools.readthedocs.io/en/latest/mechanics.html

This is a great starting off point. However, it only describes the concept of how it is structured, but not precisely what it contains. I need a very technical documentation of the very text and files that directory contains, if such documentation exists. For example, I used the REP when working on the package.xml part of the plugin.

I think an example .catkin_tools , in combination with the catkin_tools mechanics page, would be enough to do this right

[StefanFabian]

So the structure of .catkin_tools is:

+ .catkin_tools
| -+ profiles
| | -+ default
| | | -+ packages
| | | | -+ package_one
| | | | -+ ...
| | | -  build.yaml
| | | -  config.yaml
| | | -  devel_collisions.txt
| | -+ extra_profile1
| | -+ extra_profile2
| -  VERSION
| -  README
| -  CATKIN_IGNORE

VERSION contains the catkin version used (last?) with this workspace, the rest of the top-level files can be ignored. Each profile in profiles (only default has to exist), contains a build.yaml and config.yaml with options for that profile and a folder packages with a subfolder for each built package containing a devel_manifest.txt and the package's package.xml. No clue what the package folders are for. The devel_collisions.txt also seems to be irrelevant as it only contains the path to a cmake.lock which I assume is only relevant for catkin internals.

The build.yaml looks as follows:

blacklist:
- test2
- test
build_space: build
catkin_make_args: []
cmake_args:
- -DCMAKE_BUILD_TYPE=RelWithDebInfo
- -DCMAKE_BUILD_TYPE=RelWithDebInfo
devel_layout: linked
devel_space: devel
extend_path: null
install: false
install_space: install
isolate_install: false
jobs_args: []
log_space: logs
make_args: []
needs_force: true
source_space: src
use_env_cache: false
use_internal_make_jobserver: true
whitelist: []

and the config.yaml:

blacklist:
- test2
- test
build_space: build
catkin_make_args: []
cmake_args:
- -DCMAKE_BUILD_TYPE=RelWithDebInfo
devel_layout: linked
devel_space: devel
extend_path: null
install: false
install_space: install
isolate_install: false
jobs_args: []
log_space: logs
make_args: []
source_space: src
use_env_cache: false
use_internal_make_jobserver: true
whitelist: []

I assume build.yaml is the config of the last build since after running catkin build without arguments the files were the same except for the needs_force attribute which was added and set to false in the build.yaml.

Here's the second profile, I use to build packages for debian package builds: build.yaml

blacklist: []
build_space: /path/to/workspace/debs/build
catkin_make_args: []
cmake_args:
- -DCMAKE_BUILD_TYPE=RelWithDebInfo
devel_layout: linked
devel_space: /path/to/workspace/debs/devel
extend_path: null
install: false
install_space: /opt/hector
isolate_install: false
jobs_args: []
log_space: logs
make_args: []
source_space: src
use_env_cache: false
use_internal_make_jobserver: true
whitelist: []

config.yaml

blacklist: []
build_space: /path/to/workspace/debs/build
catkin_make_args: []
cmake_args:
- -DCMAKE_BUILD_TYPE=RelWithDebInfo
devel_layout: linked
devel_space: /path/to/workspace/debs/devel
extend_path: null
install: false
install_space: /opt/hector
isolate_install: false
jobs_args: []
log_space: logs
make_args: []
source_space: src
use_env_cache: false
use_internal_make_jobserver: true
whitelist: []

[Noam-Dori]

From this conversation I gathered this feature list:

• [x] basic support: .catkin_tools should be recognised as the root automatically. This should also be extendable to future tools like colcon
• profile support:
  • [x] Somewhere, either under the main ROS settings or under Build, Execution, or Deployment, have a tab dedicated to saving ROS profiles
  • [x] This list should read from the each buildtool's persistent component to create the available profiles
    • [x] for catkin_make, no such thing exists, so create a smiple default. The IDE will persist the entire profile in a configuration file (for each profile)
    • [x] for catkin_tools, simply read the data from .catkin_tools (see above for how)
    • [x] colcon is in a similar situation as catkin_make, but colcon does have a global defaults file which should be used to create the default profile. The entire colcon profile should be saved by the IDE in a persistent configuration component.
  • [x] Profiles can be added/removed/edited from CLion which will write data into a persistent component
    • [x] colcon uses the IDE persistent component entirely.
    • [x] catkin_make uses the IDE persistent component entirely.
    • [x] catkin_tools writes the edited/removed data into the .catkin_tools directory.
  • [x] Data contained in each profile form is independent of the buildtool:
    • [x] decide on build tool type (catkin_make, catkin_tools, colcon)
    • [x] whether or not to do installation step
    • [x] source, devel, build, install directories
    • [x] make args
    • [x] catkin specific args
    • [x] cmake args,
    • [x] blacklist/whitelist
    • [x] layout of the directories (isolated vs merged)
    • [x] This should be extendable since there are probably more settings
  • potentially, the IDE could allow triggering commands based on these profiles

[Noam-Dori]

On the topic of how colcon works: it seems that colcon does not implement profiles. It does have a "global default" which can be used as a template. Seems there is an intrest in adding colcon profiles though.

For now, I think implementing profiles via the IDE, like how catkin_make profiles are implemented, is the way to go.

[Noam-Dori]

Feature should be complete now. In the settings menu, there us a new tab under the Build, Execution, and Depolyment called ROS Profiles. From there you should be able to edit, create, delete, and migrate profiles. Using the information obtained from this, future plugin features could run profile-based actions like running the buildtool from the IDE.

feedback would be greatly appreciated before closing this issue