Since b6676bf cmake_modules cannot be found

[kjeremy]

Packages can't find cmake-modules anymore. On dde852a

Bisected to b6676bf7dc1b03afe9582c5fd99025b3a4b7f4f9

[jkolb@nixos:~/nix-ros-overlay]$ nix build .#noetic.pluginlib
error: builder for '/nix/store/rqnrly7cjmxylva3fypjn9dq6blbny3r-ros-noetic-class-loader-0.5.0-r1.drv' failed with exit code 1;
       last 25 log lines:
       > -- Using PYTHON_EXECUTABLE: /nix/store/h3i0acpmr8mrjx07519xxmidv8mpax4y-python3-3.12.5/bin/python
       > -- Using default Python package layout
       > -- Found PY_em: /nix/store/293x651b1sv2w9nvi8mxh5av2rcyp18j-python3.12-empy-4.1/lib/python3.12/site-packages/em.py
       > -- Using empy: /nix/store/293x651b1sv2w9nvi8mxh5av2rcyp18j-python3.12-empy-4.1/lib/python3.12/site-packages/em.py
       > -- Using CATKIN_ENABLE_TESTING: OFF
       > -- catkin 0.8.10
       > -- BUILD_SHARED_LIBS is on
       > -- Could NOT find cmake_modules (missing: cmake_modules_DIR)
       > -- Could not find the required component 'cmake_modules'. The following CMake error indicates that you either need to install the package with the same name or change your environment so that it can be found.
       > CMake Error at /nix/store/d6y7v13kik3kl2a673qxpkw0hsm56zxs-ros-noetic-catkin-0.8.10-r1/share/catkin/cmake/catkinConfig.cmake:83 (find_package):
       >   Could not find a package configuration file provided by "cmake_modules"
       >   with any of the following names:
       >
       >     cmake_modulesConfig.cmake
       >     cmake_modules-config.cmake
       >
       >   Add the installation prefix of "cmake_modules" to CMAKE_PREFIX_PATH or set
       >   "cmake_modules_DIR" to a directory containing one of the above files.  If
       >   "cmake_modules" provides a separate development package or SDK, be sure it
       >   has been installed.
       > Call Stack (most recent call first):
       >   CMakeLists.txt:22 (find_package)
       >
       > 

[wentasah]

https://github.com/NixOS/nixpkgs/commit/8c9c8ade2f88a85ccdd4858cc802d7b7d6c48fe0 might be the reason. Now, cmake outputs looks like this:

-- The CXX compiler identification is GNU 13.3.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /nix/store/zznja5f8v3jafffyah1rk46vpfcn38dv-gcc-wrapper-13.3.0/bin/g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Boost: /nix/store/aw1aybfjs5bgx9v362vjmmzw63nzy61d-boost-1.81.0-dev/lib/cmake/Boost-1.81.0/BoostConfig.cmake (found version "1.81.0") found components: thread system
-- Using CATKIN_DEVEL_PREFIX: /home/wsh/src/ros/nix-ros-overlay/class_loader-release-release-noetic-class_loader-0.5.0-1/build/devel
-- Using CMAKE_PREFIX_PATH:
...

CMAKE_PREFIX_PATH is empty, everything is now in NIXPKGS_CMAKE_PREFIX_PATH. Maybe, this somehow breaks catkin-setup-hook. I'm not much familiar with catkin so I don't know where the problem might be.

[lopsided98]

Yeah, I noticed this and came to the same conclusion, although I haven't figured out why it broke either.

[paparodeo]

CMAKE_PREFIX_PATH is empty, everything is now in NIXPKGS_CMAKE_PREFIX_PATH. Maybe, this somehow breaks catkin-setup-hook. I'm not much familiar with catkin so I don't know where the problem might be.

this shouldn't matter much. CMAKE_PREFIX_PATH still works as before and NIXPKGS_CMAKE_PREFIX_PATH is just like CMAKE_PREFIX_PATH except it is not searched for programs. this can change the order of how cmake finds exes.

to verify one thing you can try is to just copy (cp -r) cmake from by-name/cm/cmake -> by-name/cm/cmake2. then in the setup hook for cmake2 change NIXPKGS_CMAKE_PREFIX_PATH -> CMAKE_PREFIX_PATH and in the derivation that is failing use the cmake2 rather than cmake and see if that builds.

if it builds there is probably some executable which is getting run that is both in the CMAKE_PREFIX_PATH + bin and a similarly named executable in PATH. you might need to add something to nativeBuildInputs so the order of the path is correct or remove something, like was done in https://github.com/NixOS/nixpkgs/pull/337119, that is now covering the correct exe.

[edit] tho looking at this line

CMake Error at /nix/store/d6y7v13kik3kl2a673qxpkw0hsm56zxs-ros-noetic-catkin-0.8.10-r1/share/catkin/cmake/catkinConfig.cmake:83 (find_package):

it is using find_package and not find_program so there shouldn't be any changes to the path. so one thing to verify is that the path to cmake_modulesConfig.cmake or similar is located in one of the paths in NIXPKGS_CMAKE_PREFIX_PATH -- can add something like

preConfigure = ''
  echo $NIXPKGS_CMAKE_PREFIX_PATH
'';

or just drop into the builder using nix-shell an inspect the enviroment.

[kjeremy]

In a dev shell for .#noetic.class-loader after the cmakeConfigurePhase I see that $NIXPKGS_CMAKE_PREFIX_PATH includes /nix/store/5938zylqp/nix/store/5938zylqphqxbphv5h0y09sjgiymdlfn-ros-noetic-cmake-modules-0.5.0-r1/. The cmake_modulesConfig.cmake file for that package is located under that path at ./share/cmake_modules/cmake/cmake_modules/cmake_modulesConfig.cmake./share/cmake_modules/cmake/cmake_modulesConfig.cmake.

$CMAKE_PREFIX_PATH is unset.

[wentasah]

$CMAKE_PREFIX_PATH is unset.

Yes. That's what the cmake change mentioned above does.

I've tried the following:

nix develop github:lopsided98/nix-ros-overlay/develop#noetic.class-loader
genericBuild
# fails with Could not find a package configuration file provided by "cmake_modules"
cmake .. $cmakeFlags --debug-find

This prints:

  find_package considered the following locations for cmake_modules's Config
  module:

    /home/wsh/src/ros/nix-ros-overlay/class_loader-release-release-noetic-class_loader-0.5.0-1/build/CMakeFiles/pkgRedirects/cmake_modulesConfig.cmake
    /home/wsh/src/ros/nix-ros-overlay/class_loader-release-release-noetic-class_loader-0.5.0-1/build/CMakeFiles/pkgRedirects/cmake_modules-config.cmake

  The file was not found.

If I continue with:

export CMAKE_PREFIX_PATH=$NIXPKGS_CMAKE_PREFIX_PATH
cmake .. $cmakeFlags --debug-find

I prints:

find_package considered the following locations for cmake_modules's Config
  module:

    /home/wsh/src/ros/nix-ros-overlay/class_loader-release-release-noetic-class_loader-0.5.0-1/build/CMakeFiles/pkgRedirects/cmake_modulesConfig.cmake
    /home/wsh/src/ros/nix-ros-overlay/class_loader-release-release-noetic-class_loader-0.5.0-1/build/CMakeFiles/pkgRedirects/cmake_modules-config.cmake
    /nix/store/5938zylqphqxbphv5h0y09sjgiymdlfn-ros-noetic-cmake-modules-0.5.0-r1/share/cmake_modules/cmake/cmake_modulesConfig.cmake

  The file was found at

    /nix/store/5938zylqphqxbphv5h0y09sjgiymdlfn-ros-noetic-cmake-modules-0.5.0-r1/share/cmake_modules/cmake/cmake_modulesConfig.cmake

So there is a difference.

When looking at the CMakeLists.txt, there is:

  find_package(catkin)
...
  find_package(catkin REQUIRED COMPONENTS cmake_modules)

So it seems that catkin is found, but cmake_modules from the same package not. I guess we'll have to dig into catkin's cmake files to see what's happening there. I don't have more time right now and I can continue later. Maybe, somebody else will see something there in the mean time.

[paparodeo]

In a dev shell for .#noetic.class-loader after the cmakeConfigurePhase I see that $NIXPKGS_CMAKE_PREFIX_PATH includes /nix/store/5938zylqp/nix/store/5938zylqphqxbphv5h0y09sjgiymdlfn-ros-noetic-cmake-modules-0.5.0-r1/. The cmake_modulesConfig.cmake file for that package is located under that path at ./share/cmake_modules/cmake/cmake_modules/cmake_modulesConfig.cmake.

$CMAKE_PREFIX_PATH is unset.

so i believe that https://github.com/Kitware/CMake/blob/53f3e5da1e6bc15baedb1053976b677909564c6f/Source/cmFindPackageCommand.cxx#L2720 contains the permutations of sub-directories appended to (NIXPKGS_)CMAKE_PREFIX_PATH and ./share/cmake_modules/cmake/cmake_modules/cmake_modulesConfig.cmake is not one of them -- it seems like there is one cmake_modules directory too many.

[edit]

just saw https://github.com/lopsided98/nix-ros-overlay/issues/491#issuecomment-2377162615 where things work when export CMAKE_PREFIX_PATH=$NIXPKGS_CMAKE_PREFIX_PATH.

[kjeremy]

it seems like there is one cmake_modules directory too many.

Sorry I mistyped that. The path is actually /nix/store/5938zylqphqxbphv5h0y09sjgiymdlfn-ros-noetic-cmake-modules-0.5.0-r1/share/cmake_modules/cmake/cmake_modulesConfig.cmake which I think is correct

[paparodeo]

find_package(catkin REQUIRED COMPONENTS cmake_modules)

so i think that uses the framework in https://github.com/ros/catkin/blob/24e9b0a27b7889a29698fa4c90a2b0bf90df1c0b/cmake/all.cmake#L42 which seems to want to look at CMAKE_PREFIX_PATH, which is a problem.

i can add find_package(cmake_modules REQUIRED) to CMakefiles.list and it finds the pkg just fine.

[kjeremy]

find_package(catkin REQUIRED COMPONENTS cmake_modules)

so i think that uses the framework in https://github.com/ros/catkin/blob/24e9b0a27b7889a29698fa4c90a2b0bf90df1c0b/cmake/all.cmake#L42 which seems to want to look at CMAKE_PREFIX_PATH, which is a problem.

i can add find_package(cmake_modules REQUIRED) to CMakefiles.list and it finds the pkg just fine.

Do we need to patch those lines to look at NIXPKGS_CMAKE_PREFIX_PATH instead? There are a lot of cmake assumptions in the ROS ecosystem.

[paparodeo]

find_package(catkin REQUIRED COMPONENTS cmake_modules)

so i think that uses the framework in https://github.com/ros/catkin/blob/24e9b0a27b7889a29698fa4c90a2b0bf90df1c0b/cmake/all.cmake#L42 which seems to want to look at CMAKE_PREFIX_PATH, which is a problem. i can add find_package(cmake_modules REQUIRED) to CMakefiles.list and it finds the pkg just fine.

Do we need to patch those lines to look at NIXPKGS_CMAKE_PREFIX_PATH instead? There are a lot of cmake assumptions in the ROS ecosystem.

either that or run export CMAKE_PREFIX_PATH=$NIXPKGS_CMAKE_PREFIX_PATH at some point after the cmake setup hook. it looks like the catkin pkg exports cmake in propagatedNativeBuildInputs so i think just adding the setup-hook after could work.

perhaps try https://ryantm.github.io/nixpkgs/builders/special/makesetuphook/ or i think even just using writeTextFile might work: https://ryantm.github.io/nixpkgs/builders/trivial-builders/

[kjeremy]

either that or run export CMAKE_PREFIX_PATH=$NIXPKGS_CMAKE_PREFIX_PATH at some point after the cmake setup hook. it looks like the catkin pkg exports cmake in propagatedNativeBuildInputs so i think just adding the setup-hook after could work.

That seemed to work. I added it to the existing catkin setup hook.

[kjeremy]

I think this also broke at least sdformat