debugpy halts any build at the check phase

[GetPsyched]

Describe the bug

I use debugpy indirectly using vscode-extensions.ms-python.python. Whenever I build my configuration with this extension, it halts at pytestCheckPhase at 94%. Even after waiting for hours, it remains like this.

Steps To Reproduce

Steps to reproduce the behavior:

1. Build debugpy using nix-shell -p python310Packages.debugpy
2. See build halt at 94%

Expected behavior

The package builds without hanging.

Screenshots

Additional context

I have tested this with debugpy using python311 instead of python310 and it works just fine.

Notify maintainers

debugpy - @kira-bruneau, if possible, fix this issue with Python 3.10 ms-python - @jraygauthier @jfchevrette, if possible, use Python 3.11 so that this bug is fixed until it's fixed in debugpy itself

Metadata

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 6.1.55, NixOS, 23.11 (Tapir), 23.11.20230925.6500b45`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.17.0`
 - nixpkgs: `/nix/store/i6hq63805y784dml4j8pgkrz3lryxzld-source`

[kira-bruneau]

Are you still able to reproduce this? I'm guessing that it was either affected indirectly by a dependency or the tests non-deterministically hang.

Hydra is building it fine now, but there was a point where it also hanged: https://hydra.nixos.org/build/236258670.

[GetPsyched]

Yup, I reproduced it again just now; attaching a screenshot as well.

[sersorrel]

Building debugpy as a dependency of another package (obs-studio-plugins.obs-backgroundremoval), at fd309df083ec8a904f2ccb3fceb39bfb523c32be, appears to have hung even earlier, at 17%.

[charmoniumQ]

I think this is the same problem. Mine is hanging with the following output:

$ nix build --print-build-logs --option substitute false 'nixpkgs#python310Packages.debugpy'
...
python3.10-debugpy> rootdir: /build/source
python3.10-debugpy> configfile: pytest.ini
python3.10-debugpy> testpaths: tests
python3.10-debugpy> plugins: timeout-2.2.0, xdist-3.5.0
python3.10-debugpy> 2 workers [1216 items] m

[charmoniumQ]

Could this be related to https://github.com/NixOS/nixpkgs/pull/252070? Timeouts make the code flaky fail, so they were removed, but they would at least turn infinite hanging into a crash.

[totoroot]

Still running into this, no matter how often I update my flake inputs... Not sure what it even is a dependency of, but seems like the pytestCheckPhase of python3.10-debugpy-1.8.0 just does not finish 🫤

[charmoniumQ]

I can build it by disabling the pytest check:

pkgs.python310.debugpy.overrideAttrs (self: super: {
    pytestCheckPhase = ''true'';
});

but I don't know how to overlay this in other packages (I don't really understand that aspect of Nix).

[charmoniumQ]

@kira-bruneau Are you able to reproduce an infinite hang during pytestCheckPhase:

nix build --print-build-logs --option substitute false 'nixpkgs#python310Packages.debugpy'

[DataHearth]

@charmoniumQ Hey! I've got the same issue. Building python3.11-debugpy-1.8.0 stuck at 17%. I've worked around by overriding the package debugpy by setting doCheck = false. In my case, for an overlay usage, here's my config

final: prev: with prev; {
  ...
  python3 = lib.recursiveUpdate python3 {
    pkgs = lib.recursiveUpdate python3.pkgs (callPackage ./python3 { });
  };
}

After digging a little bit, I've found it was used by ms-python.python (at least in my case). I don't know what is the python3 package for now, but it is different from python311Packages, python310Packages and I guess python3Packages. I've tried override it using them without success.

After applying the overlay and the doCheck = false;, everything works fine on my end :+1: !

[kugland]

As a temporary workaround, I added the following overlay:

nixpkgs.overlays = [
  (final: prev: {
    python3 = prev.python3.override {
      packageOverrides = pfinal: pprev: {
        debugpy = pprev.debugpy.overrideAttrs (oldAttrs: {
          pytestCheckPhase = "true";
        });
      };
    };
    python3Packages = final.python3.pkgs;
  })
];

[SuperSandro2000]

Can you try using an overlay with pytest-timeout (https://pypi.org/project/pytest-timeout/#:~:text=pytest%20%2D%2Dtimeout%3D300) and see which test times out, so that we can inform upstream about this and disable the test?

[mweinelt]

I can reliably reproduce hangs across variety of Intel and AMD CPUs. With a timeout of 15 minutes both with and without xdist I see for example tests/debugpy/test_args.py::test_args[attach_pid-program] hang.

https://gist.github.com/mweinelt/1d89eca9f13f6c8d2b0d5f42606592fb

[999eagle]

Running with a timeout of 5 minutes, these are all the failing tests I have: https://gist.github.com/999eagle/99344e055b307e549d8e60426bbcad9f All of these do some kind of attach_pid, so maybe that is the cause here?

[mweinelt]

They all block in

debugpy>   File "/nix/store/sxr2igfkwhxbagri49b8krmcqz168sim-python3-3.11.8/lib/python3.11/threading.py", line 327, in wait
debugpy>     waiter.acquire()

which could imply a deadlock. But I also ran it without xdist, which didn't help one bit.

[GetPsyched]

Seems to have been fixed by #296478. Feel free to re-open if this recurs.