search

colcon / colcon-cmake

Extension for colcon to support CMake packages
http://colcon.readthedocs.io
Apache License 2.0
16 stars 25 forks source link

generate workspace-level compile_commands.json #69

Closed dirk-thomas closed 4 years ago

dirk-thomas commented 4 years ago

Requires colcon/colcon-core#326. The minimum dependency needs to be bumped once that patch has been released in colcon-core.

This patch only generates a workspace-level compile_commands.json if at least one package of the workspace does have such a file. The user needs to enable this by passing e.g. --cmake-args -DCMAKE_EXPORT_COMPILE_COMMANDS=1.

adlarkin commented 4 years ago

I tried testing this by doing the following (Ubuntu 18.04, no virtual env):

$ pip3 install -U git+https://github.com/colcon/colcon-core.git@dirk-thomas/unselected-package-event
$ pip3 install -U git+https://github.com/colcon/colcon-cmake.git@dirk-thomas/unselected-package-event

Then, I went to my workspace and entered the command colcon build --cmake-args -DCMAKE_EXPORT_COMPILE_COMMANDS=1 which resulted in the following output:

ERROR:colcon.colcon_core.entry_point:Exception loading extension 'colcon_core.event_handler.compile_commands': No module named 'colcon_cmake.event_handler'
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/colcon_core/entry_point.py", line 101, in load_entry_points
    extension_type = load_entry_point(entry_point)
  File "/usr/lib/python3/dist-packages/colcon_core/entry_point.py", line 143, in load_entry_point
    return entry_point.load()
  File "/home/adl/.local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2450, in load
    return self.resolve()
  File "/home/adl/.local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2456, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
ModuleNotFoundError: No module named 'colcon_cmake.event_handler'

[0.359s] ERROR:colcon.colcon_core.entry_point:Exception loading extension 'colcon_core.event_handler.compile_commands': No module named 'colcon_cmake.event_handler'
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/colcon_core/entry_point.py", line 101, in load_entry_points
    extension_type = load_entry_point(entry_point)
  File "/usr/lib/python3/dist-packages/colcon_core/entry_point.py", line 143, in load_entry_point
    return entry_point.load()
  File "/home/adl/.local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2450, in load
    return self.resolve()
  File "/home/adl/.local/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2456, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
ModuleNotFoundError: No module named 'colcon_cmake.event_handler'

The workspace continues to build after these error messages appear.

Did I run the wrong pip commands, or is there an issue with the pull request?

(I'm thinking I didn't do something correctly with pip because even after I enter

$ pip3 uninstall git+https://github.com/colcon/colcon-core.git@dirk-thomas/unselected-package-event
$ pip3 uninstall git+https://github.com/colcon/colcon-cmake.git@dirk-thomas/unselected-package-event

and run the colcon build command again, I still get the same error message)

dirk-thomas commented 4 years ago

Did I run the wrong pip commands

:+1:

is there an issue with the pull request?

I don't think so.

The workspace continues to build after these error messages appear.

The error messages basically say that there was a problem with the new compile_commands event handler. colcon then skips that extension and continues without it - so the build passes but it won't generate a workspace-level compile_commands.json.

Can you double check that you have the following files:

If no, something must have gone wrong when installing colcon-cmake. Maybe share the output of the command?

If yes, I guess there is something weird going on between the Debian installed packages and the pip installed ones. What does the following command output: ipython3 -c "import colcon_cmake; print(colcon_cmake.__file__)"?

A workaround would be to try a virtual env instead and install colcon there.

I'm thinking I didn't do something correctly with pip because even after I enter run the colcon build command again, I still get the same error message)

The uninstall command only takes the package name - not the full URL: e.g. pip3 uninstall colcon-core.

adlarkin commented 4 years ago

Can you double check that you have the following files:

  • /home/adl/.local/lib/python3.6/site-packages/colcon_cmake/colcon_cmake/event_handler/__init__.py

This file exists for me in /home/adl/.local/lib/python3.6/site-packages/colcon_cmake/event_handler/__init__.py which is slightly different than the path you listed (I do not have nested colcon_cmake directories)

  • /home/adl/.local/lib/python3.6/site-packages/colcon_cmake/colcon_cmake/event_handler/compile_commands.py

This file also exists for me in the same path as the __init__.py file above

If no, something must have gone wrong when installing colcon-cmake. Maybe share the output of the command?

If yes, I guess there is something weird going on between the Debian installed packages and the pip installed ones. What does the following command output: ipython3 -c "import colcon_cmake; print(colcon_cmake.__file__)"?

adl@osrf:~/ws/subt_solution_ws$ ipython3 -c "import colcon_cmake; print(colcon_cmake.__file__)"
/home/adl/.local/lib/python3.6/site-packages/colcon_cmake/__init__.py

A workaround would be to try a virtual env instead and install colcon there.

You may be right. Let me know what your thoughts are on the output I just shared above, and then we can decide what the best next step is to take.

dirk-thomas commented 4 years ago

The output looks fine - the package is installed in the right location. Though somehow pkg_resources is unable to load the entry point. I don't have a good idea how to debug that iteratively. Falling back to a virtual env is probably the easier path forward.

adlarkin commented 4 years ago

Okay. If I were to do something like a virtual env, would I just need something that would handle python packages like anaconda, or is there a specific setup you recommend?

dirk-thomas commented 4 years ago

ROS doesn't officially support anaconda so I wouldn't recommend using it for this case.

You simply create a virtual env with python3 -m venv /tmp/venv and active it by source /tmp/venv/bin/activate. After that you can install Python packages using pip as before - just that they go into the virtual env rather than into your system / user directory.

dirk-thomas commented 4 years ago

As long as you've given it some thought, I'm fine with leaving it as-is. Thanks for getting some tangible data on it.

Actually the original logic comes from when the code was collecting all compile commands in a list and then used yaml.dump to write the workspace-level file. Since it doesn't do that anymore there is no need to keep all compile commands in memory. I refactored it in 4b7754c838630ea58836fe8bf924f1dbb2172318.

mjcarroll commented 4 years ago

This correctly generates the file for me and works with my existing tooling with Vim and YouCompleteMe. This saves me some munging work when setting up a new workspace, so very much appreciated!

The first thing that I find somewhat unexpected is that it takes a successful build of the workspace to generate the top-level build/compile_commands.json. Based on my use of the compile_commands.json before, the expected behavior is for it to be available after the cmake invocation. I understand that this may be a limitation of the interaction between colcon and cmake in that there isn't really a configure vs build stage.

The other thing that seems unexpected is that the top-level file isn't generated during a filtered build (eg --packages-up-to). I suppose that's to keep from overwriting with a partial filtered compile_commands.json file after the whole thing has been generated, but it's a bit limited if you want to do a fresh partial build on a large workspace.

I don't think that either of these are show-stoppers for this PR, but could either be limitations or future enhancements.

dirk-thomas commented 4 years ago

The first thing that I find somewhat unexpected is that it takes a successful build of the workspace to generate the top-level build/compile_commands.json.

I am surprised by that. I just tried again to a) abort a build with Ctrl-C and b) fail the build due to an error in one package. In both cases the workspace-level compile_commands.json is being generated for me.

The other thing that seems unexpected is that the top-level file isn't generated during a filtered build (eg --packages-up-to).

That shouldn't be the case either. If you are also using the referenced already merged PR on colcon-core the workspace-level json file should contain the content of all package-level json files in the workspace. And even if you don't have that change the workspace-level json should still contain the content from the selected packages.

Just to double check: you do have the CMake option enabled, right (--cmake-args -DCMAKE_EXPORT_COMPILE_COMMANDS=ON)?

mjcarroll commented 4 years ago

Just to double check: you do have the CMake option enabled, right (--cmake-args -DCMAKE_EXPORT_COMPILE_COMMANDS=ON)?

Yes, I have that loaded via a defaults.yaml file in most of my workspaces.

That shouldn't be the case either. If you are also using the referenced already merged PR on colcon-core the workspace-level json file should contain the content of all package-level json files in the workspace.

I went ahead and did another venv using this PR and I am using colcon-core@master, since the other PR was merged. Things seem to be working as expected now, I must have gotten out of sync somehow.