Post install: only setup.sh exists, not setup.bash or setup.zsh

[zjeffer]

After installing the package, the post_install function in ros2-galactic.install prints instructions to add these lines to your shell file (~/.bashrc):

export ROS_DOMAIN_ID=42
source /opt/ros2/galactic/setup.bash

It also mentions to change the extension of the script to the correct one for your shell (.zsh if using zsh, etc).

The source command doesn't work as /opt/ros2/galactic/setup.bash (or *.zsh) doesn't exist. Only setup.sh exists.

After adding these lines (with setup.sh instead of my shell) to my ~/.zshrc, this is the output upon opening a new terminal:

The build time path "/home/zjeffer/.cache/paru/clone/ros2-galactic/src/install" doesn't exist. Either source a script for a different shell or set the environment variable "COLCON_CURRENT_PREFIX" explicitly.

How can I fix this? Why does my installation not include .zsh or .bash files in the /opt/... folder?

[m2-farzan]

This is at least partly related to this open issue: https://github.com/m2-farzan/ros2-galactic-PKGBUILD/issues/5

TLDR: we need to keep pkgbuild cache files. I don't use paru so I don't know if there's a way to prevent it from deleting intermediate build files. If there's not such option, you can manually build the package using 'makepkg' command.

[zjeffer]

I cloned the aur package from this link: https://aur.archlinux.org/ros2-galactic.git

I installed using makepkg -si, but installation was extremely quick and it only installed 0.02MiB. These are the only installed files according to pacman:

$ pacman -Ql ros2-galactic 
ros2-galactic /opt/
ros2-galactic /opt/ros2/
ros2-galactic /opt/ros2/galactic/
ros2-galactic /opt/ros2/galactic/COLCON_IGNORE
ros2-galactic /opt/ros2/galactic/_local_setup_util_sh.py
ros2-galactic /opt/ros2/galactic/local_setup.sh
ros2-galactic /opt/ros2/galactic/setup.sh

Sourcing the setup.sh script doesn't give any errors because the src dir now exists, but executing ros2 returns zsh: command not found: ros2. It obviously didn't install any tools, so what went wrong here? @m2-farzan

[m2-farzan]

We had a similar case of 0.02MiB package build (https://github.com/m2-farzan/ros2-foxy-PKGBUILD/issues/4#issuecomment-857257109). In that case, the issue was (1) The user was using a conda virtual env, shadowing system python packages & (2) they had to configure proxy (in that case it was probably due to china's internet limitations but you might need a similar configuration because of your workplace policies, etc.)

[m2-farzan]

I suggest reviewing your .bashrc looking for automatic virtual environment configs, etc....

[zjeffer]

I can't really see anything in my ~/.zshrc file that would indicate a Python virtual environment, here it is: https://github.com/zjeffer/dotfiles/blob/main/.zshrc

[zjeffer]

I opened a bash shell instead (~/.bashrc doesn't exist so it's definitely clean), and ran makepkg -si in a newly cloned repo. This was the output:

output.txt

[m2-farzan]

I opened a bash shell instead (~/.bashrc doesn't exist so it's definitely clean), and ran makepkg -si in a newly cloned repo. This was the output:

Yeah that's my fault... I had pushed a bad commit, breaking the AUR package for 10 minutes. It's now fixed by a follow-up commit. Just try again in bash.

[zjeffer]

Okay now it builds without errors, but the resulting package is still only 0.02MiB.

Here's the output: output.txt

I noticed that when makepkg clones the ros2 git repo, it's less than a MiB big:

[zjeffer@zjeffer ros2-galactic]$ makepkg -si
==> Making package: ros2-galactic 2021.07.16-10 (Thu 24 Feb 2022 09:54:08 PM CET)
==> Checking runtime dependencies...
==> Checking buildtime dependencies...
==> Retrieving sources...
  -> Cloning ros2 git repo...
Cloning into bare repository '/home/zjeffer/Downloads/ros2-galactic/ros2-galactic/ros2'...
remote: Enumerating objects: 3243, done.
remote: Counting objects: 100% (259/259), done.
remote: Compressing objects: 100% (186/186), done.
remote: Total 3243 (delta 136), reused 151 (delta 71), pack-reused 2984
Receiving objects: 100% (3243/3243), 771.86 KiB | 1.90 MiB/s, done.
Resolving deltas: 100% (1767/1767), done.
  -> Found google_benchmark_vendor.patch
  -> Found rviz_assimp_vendor.patch
==> Validating source files with sha256sums...
    ros2 ... Skipped
    google_benchmark_vendor.patch ... Passed
    rviz_assimp_vendor.patch ... Passed
==> Extracting sources...
  -> Creating working copy of ros2 git repo...
Cloning into 'ros2'...
done.
...

(see the Receiving objects line)

Is this normal?

[m2-farzan]

Yes, ros2/ros2 is a small repo because it only contains a list of other repos that should be downloaded in the next step (but in your case they don't get downloaded)

I mostly suspect that your python environment is lacking some required packages. A helper like paru should have installed them automatically but if you had modified your python version at the moment, it could have caused them to be installed in a wrong place. I suggest trying to install all the dependencies again using paru, this time making sure they get installed for python 10.

[m2-farzan]

Another suggestion... Try this:

1. Download the sources without building them: makepkg --nobuild
2. Go to ./src/ros2/src and make sure there are ~10 source directories (ament, eclipse-..., osrf, ros, ros2, ros-...)
3. Go back to ./src and run colcon build --merge-install and make sure that ~330 packages get built.

I hope that in one of these steps the problem shows itself.

[zjeffer]

1. makepkg --nobuild failed because I had uninstalled dependencies using paru -Rsn python-colcon-common-extensions ros2-arch-deps ros2-galactic. I then reinstalled the deps: paru -S ros2-arch-deps. makepkg --nobuild then worked fine.
2. There were 12 directories, everything looked fine.
3. Running colcon build --merge-install just returns with no output. There's the culprit.

Here's the DEBUG output of colcon --log-level=DEBUG build --merge-install:

colcon --log-level=DEBUG build --merge-install
DEBUG:colcon:Parsed command line arguments: Namespace(log_base=None, log_level=10, verb_name='build', build_base='build', install_base='install', merge_install=True, symlink_install=False, test_result_base=None, continue_on_error=False, allow_overriding=[], executor='sequential', event_handlers=None, paths=['.'], verb_parser=CustomArgumentParser(prog='colcon build', usage=None, description='Build a set of packages.', formatter_class=<class 'colcon_core.command.CustomFormatter'>, conflict_handler='error', add_help=True), verb_extension=<colcon_core.verb.build.BuildVerb object at 0x7f10d8fc66a0>, main=<bound method BuildVerb.main of <colcon_core.verb.build.BuildVerb object at 0x7f10d8fc66a0>>)
INFO:colcon.colcon_core.location:Using log path 'log/build_2022-02-25_15-06-48'
[0.301s] INFO:colcon.colcon_core.executor:Executing jobs using 'sequential' executor
[0.305s] DEBUG:colcon.colcon_core.executor.sequential:closing loop
[0.305s] DEBUG:colcon.colcon_core.executor.sequential:loop closed
[0.305s] DEBUG:colcon.colcon_core.event_reactor:joining thread
[0.305s] DEBUG:colcon.colcon_core.event_reactor:joined thread
[0.311s] INFO:colcon.colcon_core.plugin_system:Skipping extension 'colcon_core.shell.bat': Not used on non-Windows systems
[0.311s] INFO:colcon.colcon_core.shell:Creating prefix script '/home/zjeffer/Downloads/ros2-galactic/ros2-galactic/src/install/local_setup.sh'
[0.312s] INFO:colcon.colcon_core.shell:Creating prefix util module '/home/zjeffer/Downloads/ros2-galactic/ros2-galactic/src/install/_local_setup_util_sh.py'
[0.313s] INFO:colcon.colcon_core.shell:Creating prefix chain script '/home/zjeffer/Downloads/ros2-galactic/ros2-galactic/src/install/setup.sh'```

[m2-farzan]

From this link I suspect you're missing some colcon packages. Can you run this:

python -m pip freeze | grep colcon

and make sure the output looks like this:

colcon-argcomplete==0.3.3
colcon-bash==0.4.2
colcon-cmake==0.2.26
colcon-common-extensions==0.2.1
colcon-core==0.6.1
colcon-defaults==0.2.5
colcon-devtools==0.2.2
colcon-library-path==0.2.1
colcon-metadata==0.2.5
colcon-notification==0.2.13
colcon-output==0.2.12
colcon-package-information==0.3.3
colcon-package-selection==0.2.10
colcon-parallel-executor==0.2.4
colcon-pkg-config==0.1.0
colcon-powershell==0.3.6
colcon-python-setup-py==0.2.7
colcon-recursive-crawl==0.2.1
colcon-ros==0.3.21
colcon-test-result==0.3.8
colcon-zsh==0.4.0

[zjeffer]

Here's the output:

colcon-argcomplete==0.3.3
colcon-bash==0.4.2
colcon-cmake==0.2.26
colcon-common-extensions==0.2.1
colcon-core==0.7.1
colcon-defaults==0.2.6
colcon-devtools==0.2.3
colcon-library-path==0.2.1
colcon-metadata==0.2.5
colcon-notification==0.2.13
colcon-output==0.2.12
colcon-package-information==0.3.3
colcon-package-selection==0.2.10
colcon-parallel-executor==0.2.4
colcon-pkg-config==0.1.0
colcon-powershell==0.3.7
colcon-python-setup-py==0.2.7
colcon-recursive-crawl==0.2.1
colcon-ros==0.3.22
colcon-test-result==0.3.8
colcon-zsh==0.4.0

Some of these packages have different versions but they seem to all be there.

[m2-farzan]

Strange...

By the way, did you revert this before re-installing colcon:

I fixed the issue by symlinking Python 3.9 instead of 3.10 like this:

sudo ln -sf /usr/bin/python3.9 /usr/bin/python3

Just sharing any clue that comes to my mind...

[m2-farzan]

Also compare $ colcon extensions output with this:

$ colcon extensions
colcon_argcomplete.argcomplete_completer
  catkin_cmake_args: Completion of catkin CMake arguments
  cmake_args: Completion of CMake arguments
colcon_core.argument_parser
  argcomplete: Argcomplete-based completions for the command line tool
  colcon: Default command line arguments from global config file
colcon_core.environment
  cmake_module_path: Extend the `CMAKE_MODULE_PATH` variable to find CMake modules
  cmake_prefix_path: Extend the `CMAKE_PREFIX_PATH` variable to find CMake config files
  library_path: Extend the platform specific environment variable to find shared libraries
  path: Extend the `PATH` variable to find executables
  pkg_config: Extend the environment variable PKG_CONFIG_PATH
  pythonpath: Extend the `PYTHONPATH` variable to find Python modules
colcon_core.event_handler
  compile_commands: Generate a `compile_commands.json` file for the whole workspace
  console_cohesion: Pass task output at once to stdout
  console_direct: Pass output directly to stdout/err
  console_package_list: Output list of queued job names
  console_start_end: Output task name on start/end
  console_stderr: Output all stderr of a task at once
  desktop_notification: Desktop notification of the summary
  event_log: Log all events to a global log file
  log: Output task specific log files
  log_command: Log a 'debug' message for each command
  status: Continuously update a status line
  store_result: Persist the result of a job in a file in its build directory
  summary: Output summary of all tasks
  terminal_title: Show status in the terminal title
colcon_core.executor
  parallel: Process multiple packages in parallel
  sequential: Process one package at a time
colcon_core.package_augmentation
  check_dependency_constraint: Check package dependency constraints
  colcon_meta: Provide meta information from `.colcon` files
  colcon_pkg: Provide meta information from `colcon.pkg` files
  ignore: Ignore discovered packages based on cli arguments
  python: Augment Python packages with information from `setup.cfg` files
  python_setup_py: Augment Python packages with information from `setup.py` files
  ros: Identify ROS packages with `package.xml` files
colcon_core.package_discovery
  colcon_meta: Provide meta information from `.meta` files
  ignore: Ignore discovered packages based on cli arguments
  path: Check specific paths for packages
  recursive: Crawl paths recursively for packages
colcon_core.package_identification
  cmake: Identify CMake packages with `CMakeLists.txt` files
  colcon_meta: Provide meta information from `.colcon` files
  colcon_pkg: Provide meta information from `colcon.pkg` files
  ignore: Ignore paths containing a `COLCON_IGNORE` file
  ignore_ament_install: Ignore paths containing files belonging to `catkin` or `ament_tools`
  python: Identify Python packages with `setup.cfg` files
  python_setup_py: Identify Python packages with `setup.py` files
  ros: Identify ROS packages with `package.xml` files
colcon_core.package_selection
  dependencies: Select packages based on their dependencies
  previous: Skip a set of packages based on results of previous invocations
  select_skip: Select a set of packages based on select / skip criteria
  start_end: Select a range of packages based on flattened topological ordering
colcon_core.prefix_path
  ament: Prefix path defined in the `AMENT_PREFIX_PATH` environment variable
  catkin: Prefix path defined in the `CMAKE_PREFIX_PATH` environment variable
  colcon: Prefix path defined in the `COLCON_PREFIX_PATH` environment variable
colcon_core.python_testing
  pytest: Use `pytest` to test Python packages
  setuppy_test: Use `unittest` to test packages
colcon_core.shell
  bash: Generate `.bash` scripts to extend the environment
  dsv: Generate `.dsv` files describing the intended environment change
  powershell: Generate `.ps1` scripts to extend the environment
  sh: Generate `.sh` scripts to extend the environment
  zsh: Generate `.zsh` scripts to extend the environment
colcon_core.task.build
  cmake: Build CMake packages
  python: Build Python packages
  ros.ament_cmake: Build ROS packages with the build type 'ament_cmake'
  ros.ament_python: Build ROS packages with the build type 'ament_python'
  ros.catkin: Build ROS packages with the build type 'catkin'
  ros.cmake: Build ROS packages with the build type 'cmake'
colcon_core.task.test
  cmake: Test CMake packages
  python: Test Python packages
  ros.ament_cmake: Test ROS packages with the build type 'ament_cmake'
  ros.ament_python: Test ROS packages with the build type 'ament_python'
  ros.catkin: Test ROS packages with the build type 'catkin'
  ros.cmake: Test ROS packages with the build type 'cmake'
colcon_core.verb
  build: Build a set of packages
  extension-points: List extension points
  extensions: List extensions
  graph: Generate a visual representation of the dependency graph
  info: Package information
  list: List packages, optionally in topological ordering
  metadata: Manage metadata of packages
  test: Test a set of packages
  test-result: Show the test results generated when testing a set of packages
  version-check: Compare local package versions with PyPI
colcon_metadata.subverb
  add: Add the URL of a repository index
  list: List all repositories and their metadata
  remove: Remove a repository from the list of indexes
  update: Update the metadata from the repository indexes
colcon_notification.desktop_notification
  notify2: Use `notify2` to show notifications
  notify_send: Use `notify-send` to show notifications
colcon_test_result.test_result
  ctest: Collect the CTest results generated when testing a set of CMake packages
  xunit: Collect the xUnit results generated when testing a set of packages

[zjeffer]

By the way, did you revert this before re-installing colcon

Yes, I did.

Also compare $ colcon extensions output with this:

Well this is interesting...:

colcon extensions                             
usage: colcon [-h] [--log-base LOG_BASE] [--log-level LOG_LEVEL] {build,test} ...
colcon: error: argument verb_name: invalid choice: 'extensions' (choose from 'build', 'test')

python-colcon-common-extensions is installed. What's happening here?

[m2-farzan]

Now we're getting somewhere.

Run:

pacman -Ql python-colcon-common-extensions

You should get:

python-colcon-common-extensions /usr/
python-colcon-common-extensions /usr/lib/
python-colcon-common-extensions /usr/lib/python3.10/
python-colcon-common-extensions /usr/lib/python3.10/site-packages/
python-colcon-common-extensions /usr/lib/python3.10/site-packages/colcon_common_extensions-0.2.1-py3.10.egg-info/
python-colcon-common-extensions /usr/lib/python3.10/site-packages/colcon_common_extensions-0.2.1-py3.10.egg-info/PKG-INFO
python-colcon-common-extensions /usr/lib/python3.10/site-packages/colcon_common_extensions-0.2.1-py3.10.egg-info/SOURCES.txt
python-colcon-common-extensions /usr/lib/python3.10/site-packages/colcon_common_extensions-0.2.1-py3.10.egg-info/dependency_links.txt
python-colcon-common-extensions /usr/lib/python3.10/site-packages/colcon_common_extensions-0.2.1-py3.10.egg-info/requires.txt
python-colcon-common-extensions /usr/lib/python3.10/site-packages/colcon_common_extensions-0.2.1-py3.10.egg-info/top_level.txt
python-colcon-common-extensions /usr/lib/python3.10/site-packages/colcon_common_extensions-0.2.1-py3.10.egg-info/zip-safe
python-colcon-common-extensions /usr/lib/python3.10/site-packages/colcon_common_extensions/
python-colcon-common-extensions /usr/lib/python3.10/site-packages/colcon_common_extensions/__init__.py
python-colcon-common-extensions /usr/lib/python3.10/site-packages/colcon_common_extensions/__pycache__/
python-colcon-common-extensions /usr/lib/python3.10/site-packages/colcon_common_extensions/__pycache__/__init__.cpython-310.opt-1.pyc
python-colcon-common-extensions /usr/lib/python3.10/site-packages/colcon_common_extensions/__pycache__/__init__.cpython-310.pyc
python-colcon-common-extensions /usr/share/
python-colcon-common-extensions /usr/share/doc/
python-colcon-common-extensions /usr/share/doc/python-colcon-common-extensions/
python-colcon-common-extensions /usr/share/doc/python-colcon-common-extensions/README.rst

Note /usr/lib/python3.10/site-packages in the paths. If your installation is not in that path, that could be the issue. In that case, fully reinstalling colcon packages should do the job. Alternatively, you can install colcon packages using pip (e.g. pip install colcon-common-extensions) but I recommend trying the aur packages first.

[zjeffer]

I have the exact same files. Installing colcon-common-extensions with pip made colcon extensions work.

I then ran paru -S ros2-galactic. It's looking like it's building everything. I'll edit this comment when it's done.

EDIT:

IT WORKS! So the solution was to install colcon-common-extensions with pip. I now have the other setup.*sh files, so I can use setup.zsh with my shell.

Thank you very much for all your help, @m2-farzan !

[m2-farzan]

No problem!