nix build stuck at pythonCatchConflictsPhase

[i-am-logger]

after a nix flake update

nix build is getting stuck (1 thread is working)

I tried to let it continue but after 5 hours and no change i cancelled the build...

[lopsided98]

Please provide a self contained example that reproduces this, as it builds fine when I try it.

[i-am-logger]

https://github.com/realsnick/sky360/tree/main/nixOS is what i'm using...

[i-am-logger]

services.ros2 = {
    enable = true;
    distro = "iron";
    domainId = 0;
    # systemPackages = p: with p; [ ros2cli ros2run ];
    # # pkgs = 
    nodes = {
      talker = {
        package = "demo_nodes_cpp";
        node = "talker";
      };
      listener2 = {
        package = "demo_nodes_cpp";
        node = "listener";
      };
    };
  };

without any nodes, everything builds fine. The moment i enable the nodes (such as talker/listener) , then i get the pythonCatchConflictsPhase running (very slow) on ros2run

after more tests, without any nodes

    systemPackages = p: with p; [ ros2run ];

this seems to cause the issue as well.

[i-am-logger]

seems i just needed to wait (ages) and it builds...

[lopsided98]

The build times are normal in my nix-ros-overlay builds that use the pinned nixpkgs version, but I'm observing the long build times with one of my NixOS machines that uses a newer nixpkgs. I wonder if this is a bug introduced in nixpkgs recently.

[lopsided98]

If I had to guess: https://github.com/NixOS/nixpkgs/pull/287957

[i-am-logger]

i did use the latest nixpkgs but also tried pointing nixpkgs to lopsided98/nixpkgs .. didn't seem to help to be honest but it does feel its a new thing that was changed....

I also noticed that this overlay stopped working...

      sky360-ros-packages-overlay = (final: prev: {
        sky360-ros-iron-heartbeat = prev.callPackage ./ros/iron/heartbeat {};
      });
      sky360-packages-overlay = (self: super: {
        rosPackages = super.rosPackages // {
          iron = super.rosPackages.iron.overrideScope sky360-ros-packages-overlay;
        };
      });   

the error that i get is this:

the ros2.nix is this

{ pkgs, ... }:
{
  environment.systemPackages = with pkgs; [
    pkgs.rosPackages.iron.sky360-ros-iron-heartbeat
  ];

  services.ros2 = {
    enable = true;
    distro = "iron";
    domainId = 0;
    # systemPackages = p: with p; [ python3Packages.pyyaml ros2cli ros2run ];
    # # pkgs = 
    nodes = {
      heartbeat = {
        # package = pkgs.sky360.sky360-ros-iron-heartbeat.pname;
        package = pkgs.rosPackages.iron.sky360-ros-iron-heartbeat.pname;
        node = "heartbeat";
        args = [ ];
        rosArgs = [ ];
        params = { };
      };
    };
  };
}

could be related to the other issue i opened #367 where i fixed the overlay to my own namespace but now the package failed to run with yaml module not found

[hacker1024]

We've hit this as well - many of our own ROS packages are taking a long time to build after upgrading nixos-unstable. I'll try and look into it over the weekend.

[lopsided98]

After some investigation, it appears the new version of pythonCatchConflictsHook traverses all paths in the closure DAG, so it has exponential time complexity.

[hacker1024]

If you don't want to wait for https://github.com/NixOS/nixpkgs/pull/305619, and don't want to rebuild a great portion of Nixpkgs by merging it yourself, you can override just buildRosPackage to use the updated hook.

{
  # pythonCatchConflictsHook: prevent exponential worst-case
  # https://github.com/NixOS/nixpkgs/pull/305619
  buildRosPackage = args: rosSuper.buildRosPackage (args // self.lib.optionalAttrs (args.buildType == "ament_python" && self.lib.versionAtLeast rosSelf.python.pythonVersion "3") {
    catchConflicts = false;
    nativeBuildInputs = args.nativeBuildInputs or [ ] ++ [
      (rosSelf.pythonPackages.makePythonHook
        {
          name = "python-catch-conflicts-hook";
          substitutions = {
            pythonInterpreter = rosSelf.python.pythonOnBuildForHost.interpreter;
            pythonSitePackages = rosSelf.python.sitePackages;
            catchConflicts = (
              # The file catch_conflicts.py is in a subdirectory because, if it isn't, the /nix/store/ directory is added to sys.path causing a delay when building.
              "${self.fetchurl {
                url = "https://github.com/lopsided98/nixpkgs/raw/8d18ed3af4b30d5e83deb4e31a8a83c60e6f4b08/pkgs/development/interpreters/python/catch_conflicts/catch_conflicts.py";
                downloadToTemp = true;
                postFetch = ''
                  mkdir -p "$out/catch_conflicts"
                  mv "$downloadedFile" "$out/catch_conflicts/catch_conflicts.py"
                '';
                recursiveHash = true;
                hash = "sha256-OMelWMXE2UXL0lHqtpMcsnAgLOqiZ1BTEvsyXP5MfOs=";
              }}/catch_conflicts/catch_conflicts.py"
            );
          };
        }
        (self.path + "/pkgs/development/interpreters/python/hooks/python-catch-conflicts-hook.sh"))
    ];
  });
}

[Pleune]

@hacker1024, how can I apply this override? I cannot figure out how to use your snippet.