Can't use `rmw_cyclonedds_cpp` with ros2 `humble` because path to `*.so` can't be resolved

[zatevakhin]

Trying to list topics for example, or use any ros2 command:

# ivan @ nixos in /projects/work/nix-ros [12:53:18] 
$ nix develop   
You use 'humble' with 'rmw_cyclonedds_cpp' on domain '1'.

[ivan@nixos:/projects/work/nix-ros]$ ros2 topic list
[ERROR] [1713873209.974130528] [rcl]: Error getting RMW implementation identifier / RMW implementation not installed (expected identifier of 'rmw_cyclonedds_cpp'), with error message 'failed to load shared library 'librmw_cyclonedds_cpp.so' due to dlopen error: librmw_cyclonedds_cpp.so: cannot open shared object file: No such file or directory, at /build/rcutils-release-release-humble-rcutils-5.1.5-1/src/shared_library.c:99, at /build/rmw_implementation-release-release-humble-rmw_implementation-2.8.2-1/src/functions.cpp:65', exiting with 1., at /build/rcl-release-release-humble-rcl-5.3.7-1/src/rcl/rmw_implementation_identifier_check.c:139

Here is my flake.nix which I use:

{
  inputs = {
    nix-ros-overlay.url = "github:lopsided98/nix-ros-overlay";
    nixpkgs.follows = "nix-ros-overlay/nixpkgs";  # IMPORTANT!!!
  };
  outputs = { self, nix-ros-overlay, nixpkgs }:
    nix-ros-overlay.inputs.flake-utils.lib.eachDefaultSystem (system:
      let
        ROS_DISTRO = "humble";
        pkgs = import nixpkgs {
          inherit system;
          overlays = [ nix-ros-overlay.overlays.default ];
        };
      in {
        devShells.default = pkgs.mkShell {
          name = "Example project";
          packages = with pkgs.rosPackages.${ROS_DISTRO}; [
            pkgs.colcon
            ros-core
            rmw-cyclonedds-cpp
          ];

          RMW_IMPLEMENTATION = "rmw_cyclonedds_cpp";
          ROS_DOMAIN_ID = 1;
          shellHook = ''
          echo "You using '$ROS_DISTRO' with '$RMW_IMPLEMENTATION' on domain '$ROS_DOMAIN_ID'."
          '';
        };
      });

  nixConfig = {
    trusted-substituters = [ "https://ros.cachix.org" ];
    extra-trusted-public-keys = [ "ros.cachix.org-1:dSyZxI8geDCJrwgvCOHDoAfOm5sV1wCPjBkKL+38Rvo=" ];
  };
}

I found the librmw_cyclonedds_cpp.so exists in the Nix store, but it seems it can't be linked, mb something wrong with nix-ld.

# ivan @ nixos in /projects/work/nix-ros [13:00:37] C:1
$ locate librmw_cyclonedds_cpp.so
/nix/store/nvgv329v69kmb2j5bvwzmir9xyjy9kyd-ros-humble-rmw-cyclonedds-cpp-1.3.4-r1/lib/librmw_cyclonedds_cpp.so

[zatevakhin]

Checked my env and path to /nix/store/nvgv329v69kmb2j5bvwzmir9xyjy9kyd-ros-humble-rmw-cyclonedds-cpp-1.3.4-r1/ is present in next vars:

AMENT_PREFIX_PATH= ... :/nix/store/nvgv329v69kmb2j5bvwzmir9xyjy9kyd-ros-humble-rmw-cyclonedds-cpp-1.3.4-r1: ...
CMAKE_PREFIX_PATH= ... :/nix/store/nvgv329v69kmb2j5bvwzmir9xyjy9kyd-ros-humble-rmw-cyclonedds-cpp-1.3.4-r1: ...
XDG_DATA_DIRS= ... :/nix/store/nvgv329v69kmb2j5bvwzmir9xyjy9kyd-ros-humble-rmw-cyclonedds-cpp-1.3.4-r1/share: ...
NIX_LDFLAGS=-rpath /projects/work/nix-ros/outputs/out/lib  ... -L/nix/store/nvgv329v69kmb2j5bvwzmir9xyjy9kyd-ros-humble-rmw-cyclonedds-cpp-1.3.4-r1/lib ... -L/nix/store/nvgv329v69kmb2j5bvwzmir9xyjy9kyd-ros-humble-rmw-cyclonedds-cpp-1.3.4-r1/lib 
nativeBuildInputs=/nix/store/nvgv329v69kmb2j5bvwzmir9xyjy9kyd-ros-humble-rmw-cyclonedds-cpp-1.3.4-r1 ...
CMAKE_LIBRARY_PATH=... :/nix/store/nvgv329v69kmb2j5bvwzmir9xyjy9kyd-ros-humble-rmw-cyclonedds-cpp-1.3.4-r1/lib: ... :/nix/store/nvgv329v69kmb2j5bvwzmir9xyjy9kyd-ros-humble-rmw-cyclonedds-cpp-1.3.4-r1/lib: ...

[zatevakhin]

But there is no cyclonedds rmw in LD_LIBRARY_PATH

echo $LD_LIBRARY_PATH | rg rmw
/nix/store/vhxxkjfnmwzvn1zdwdnwm8yigv3qqx8h-ros-humble-rclcpp-components-16.0.8-r1/lib:/nix/store/jql7jpzf38f006pyfqcsdgzwi77bzcbz-ros-humble-rclcpp-lifecycle-16.0.8-r1/lib:/nix/store/fmc3xz21r0rpfzxv64spvfabrjzk0s71-ros-humble-rclcpp-action-16.0.8-r1/lib:/nix/store/gk1gy5ykm5ihr23xhg2h3xkzimii34wq-ros-humble-statistics-msgs-1.2.1-r1/lib:/nix/store/3sdlf4xs4mr34nv85kswzq5qppr6384g-ros-humble-libstatistics-collector-1.3.1-r1/lib:/nix/store/p795qh46c8lbv4pdfg6wffflm6rf7q65-ros-humble-rclcpp-16.0.8-r1/lib:/nix/store/2pxbr043gcd1fqa5xif9z8182ra5szzf-ros-humble-rosgraph-msgs-1.2.1-r1/lib:/nix/store/rqznjh6h2y6pxyivvalir05kmfqhawd9-ros-humble-rcl-lifecycle-5.3.7-r1/lib:/nix/store/kk3f117hfd4d9lahyhii24rdf1rm426a-ros-humble-unique-identifier-msgs-2.2.1-r3/lib:/nix/store/s1g909ip65x9y8c3yzk8acc4jkwcs5gy-ros-humble-action-msgs-1.2.1-r1/lib:/nix/store/c2hxgng7nnvr1mmmmqqbxhipn8vjkg5k-ros-humble-rcl-action-5.3.7-r1/lib:/nix/store/x8iriys3cm8d4pdn5gfvlcds60r36mzm-ros-humble-tracetools-4.1.1-r1/lib:/nix/store/7p7lf033dq2p68xgrv4fx5wfwdbjz33m-ros-humble-rmw-fastrtps-shared-cpp-6.2.6-r1/lib:/nix/store/0d7sz1mbd9bc8bddd5kafg23rykxl4vb-ros-humble-rmw-dds-common-1.6.0-r2/lib:/nix/store/ma7myf6hai606m5iijwwivid92f4jaxi-ros-humble-rmw-fastrtps-cpp-6.2.6-r1/lib:/nix/store/x377mm7dvzzmsn9jwfgg3qz23vl2sz1c-ros-humble-libyaml-vendor-1.2.2-r2/lib:/nix/store/bdz5lkcrmzj7ynrbg004pnddj4kdww26-ros-humble-rcl-yaml-param-parser-5.3.7-r1/lib:/nix/store/a845kk860c92y825zxn97c97sxx7xxjl-ros-humble-rcl-logging-spdlog-2.3.1-r1/lib:/nix/store/qzprvyz2rjb8nsp1dr28g8wgdrq76g9f-ros-humble-rcl-5.3.7-r1/lib:/nix/store/vpimf2rhcng6bx74camsklpxqm6m406s-ros-humble-rclpy-3.3.12-r1/lib:/nix/store/gii4axzmvx8rzaqmijzmynsy5rqc3sdw-ros-humble-lifecycle-msgs-1.2.1-r1/lib:/nix/store/4m92m4iksqdw5fn2hq8d1i8fyqyy0h63-ros-humble-rcl-interfaces-1.2.1-r1/lib:/nix/store/hjlgg1xxsljiwzwsqpzl7rkcycfk72l0-ros-humble-composition-interfaces-1.2.1-r1/lib:/nix/store/r8k1cmh52q9zqqnhh3kyq4ggwrxq5r8v-ros-humble-visualization-msgs-4.2.3-r1/lib:/nix/store/097qkj76d8h1n5d022mq58i21hk0q4xb-ros-humble-trajectory-msgs-4.2.3-r1/lib:/nix/store/dq6liv0j4mhmq1ks2fpwydwlvdzlwphj-ros-humble-stereo-msgs-4.2.3-r1/lib:/nix/store/n64ai5wp70d686ksrj3drm09kr84mb5s-ros-humble-std-srvs-4.2.3-r1/lib:/nix/store/x5l96fk17fyihawbrmvb64a26mqp96jm-ros-humble-shape-msgs-4.2.3-r1/lib:/nix/store/m7z1640vvc0kfmgyzpjl3bw3vim16zy7-ros-humble-sensor-msgs-4.2.3-r1/lib:/nix/store/xbmvary848c59cff2w0xs9lsvzmafkbm-ros-humble-nav-msgs-4.2.3-r1/lib:/nix/store/61hc2v3g5994m707p943kh95qsm76d92-ros-humble-geometry-msgs-4.2.3-r1/lib:/nix/store/vm1gl2q8qpkm6bh7qab43rxifzjimcs9-ros-humble-diagnostic-msgs-4.2.3-r1/lib:/nix/store/0a1zkdyi0zwdn2is6dzs5bfr0vjck9bs-ros-humble-std-msgs-4.2.3-r1/lib:/nix/store/nwk37g5l2dp5sgxxz8h2ghd79kzgnqfy-ros-humble-rosidl-typesupport-introspection-cpp-3.1.5-r2/lib:/nix/store/77q55fw9851digzn197qslyl4h9m84f2-ros-humble-rosidl-typesupport-introspection-c-3.1.5-r2/lib:/nix/store/z3f54fnd4a0abj7iwgwv9kgrbmkrmc4a-ros-humble-rosidl-typesupport-fastrtps-cpp-2.2.2-r2/lib:/nix/store/2pmp5xsg6g1x3rm1kcm25r9lbcpki9af-ros-humble-rosidl-typesupport-fastrtps-c-2.2.2-r2/lib:/nix/store/m32ikdwm8dy7hmqhwyqrp1s8ppnj49c5-ros-humble-rosidl-typesupport-cpp-2.0.1-r1/lib:/nix/store/l3dd5ym75l0ywbgrsfvfnh5mdglnimvd-ros-humble-rosidl-typesupport-c-2.0.1-r1/lib:/nix/store/l0l9lw65wiicjirxwrwb4cw5maa012b3-ros-humble-rosidl-runtime-c-3.1.5-r2/lib:/nix/store/xkngsa8vjbxz8ahyx3bks9k7xzq4h4rn-ros-humble-rmw-6.1.1-r1/lib:/nix/store/6h9sf98qq8fm4i7mlkcfagqgl6rqsr4d-ros-humble-builtin-interfaces-1.2.1-r1/lib:/nix/store/i5abgfrr3l32z5z7r85lmnfg0n6gsp2j-ros-humble-actionlib-msgs-4.2.3-r1/lib:/nix/store/ifhrfqdfc8hn4hx0b05gjw9ycycfkm2s-ros-humble-rcutils-5.1.5-r1/lib:/nix/store/lmn09g5qlq5z5njd60yjrb69lfr7892n-ros-humble-rcpputils-2.4.2-r1/lib:/nix/store/ql9si7g8zkh4x9gw78yfspdzp5d465bw-ros-humble-ament-index-cpp-1.4.0-r2/lib

And it works when I manually add it to LD_LIBRARY_PATH.

[ivan@nixos:/projects/work/nix-ros]$ export LD_LIBRARY_PATH="/nix/store/nvgv329v69kmb2j5bvwzmir9xyjy9kyd-ros-humble-rmw-cyclonedds-cpp-1.3.4-r1/lib:$LD_LIBRARY_PATH"

[ivan@nixos:/projects/work/nix-ros]$ ros2 topic list
/parameter_events
/rosout

But fastrtps works fine and exists in LD_LIBRARY_PATH. As /nix/store/7p7lf033dq2p68xgrv4fx5wfwdbjz33m-ros-humble-rmw-fastrtps-shared-cpp-6.2.6-r1/lib.

Is this the expected behavior, and should I add it to LD_LIBRARY_PATH in my flake.nix?

[zatevakhin]

Kinda solved. Is this a good way to solve this?

{
  inputs = {
    nix-ros-overlay.url = "github:lopsided98/nix-ros-overlay";
    nixpkgs.follows = "nix-ros-overlay/nixpkgs";  # IMPORTANT!!!
  };
  outputs = { self, nix-ros-overlay, nixpkgs }:
    nix-ros-overlay.inputs.flake-utils.lib.eachDefaultSystem (system:
      let
        ROS_DISTRO = "humble";
        pkgs = import nixpkgs {
          inherit system;
          overlays = [ nix-ros-overlay.overlays.default ];
        };
      in {
        devShells.default = pkgs.mkShell {
          name = "ros-${ROS_DISTRO}-dev-environment";

          packages = with pkgs.rosPackages.${ROS_DISTRO}; [
            pkgs.colcon
            ros-core
            rmw-cyclonedds-cpp
            # ...
          ];

          LD_LIBRARY_PATH="${pkgs.rosPackages.${ROS_DISTRO}.rmw-cyclonedds-cpp}/lib";
          RMW_IMPLEMENTATION = "rmw_cyclonedds_cpp";
          ROS_DOMAIN_ID = 1;
          shellHook = ''
            echo "You using '$ROS_DISTRO' with '$RMW_IMPLEMENTATION' on domain '$ROS_DOMAIN_ID'."
          '';
        };
      });

  nixConfig = {
    trusted-substituters = [ "https://ros.cachix.org" ];
    extra-trusted-public-keys = [ "ros.cachix.org-1:dSyZxI8geDCJrwgvCOHDoAfOm5sV1wCPjBkKL+38Rvo=" ];
  };
}

[wentasah]

Wrap your ROS packages to buildEnv like this:

          packages = with pkgs.rosPackages.${ROS_DISTRO}; [
            pkgs.colcon
            (buildEnv {
              paths = [
                ros-core
                rmw-cyclonedds-cpp
              ];
            })
          ];

This ensures that LD_LIBRARY_PATH is set correctly.

We should fix the example flake.nix as many people seem to experience similar problems. I'll do it.

[zatevakhin]

@wentasah Thanks! Works perfectly for me. Now is no need to set LD_LIBRARY_PATH manually.