search

NixOS / nixpkgs

Nix Packages collection & NixOS
MIT License
16.47k stars 12.97k forks source link

python3Packages.joblib fails on Darwin #65286

Closed kalbasit closed 4 years ago

kalbasit commented 4 years ago

Describe the bug

nix-build -A python3Packages.joblib fails on Darwin.

To Reproduce Steps to reproduce the behavior:

  1. nix-build -A python3Packages.joblib

Expected behavior Package builds

Additional context

Error log:

Error log ```console running install tests ============================= test session starts ============================== platform darwin -- Python 3.7.4, pytest-4.6.3, py-1.7.0, pluggy-0.12.0 rootdir: /private/var/folders/ck/ht80_l5d4c9_tnwmd5lqsxj40000gn/T/nix-build-python3.7-joblib-0.13.2.drv-0/joblib-0.13.2, inifile: setup.cfg, testpaths: joblib collected 1109 items / 1 deselected / 2 skipped / 1106 selected joblib/__init__.py s [ 0%] joblib/parallel.py ss [ 0%] joblib/test/test_backports.py s.... [ 0%] joblib/test/test_disk.py ........ [ 1%] joblib/test/test_format_stack.py s... [ 1%] joblib/test/test_func_inspect.py s.................................... [ 5%] joblib/test/test_hashing.py sssssssssss................................. [ 9%] ........................................................................ [ 15%] ........................................................................ [ 22%] ........................................................................ [ 28%] ........................................................................ [ 35%] ........................................................................ [ 41%] ........................................................................ [ 48%] ....................ss............. [ 51%] joblib/test/test_init.py . [ 51%] joblib/test/test_logger.py . [ 51%] joblib/test/test_memmapping.py ssssssssssssss.. [ 52%] joblib/test/test_memory.py ss........................................... [ 56%] ...... [ 57%] joblib/test/test_module.py .. [ 57%] joblib/test/test_my_exceptions.py ... [ 57%] joblib/test/test_numpy_pickle.py sssssssssssssssssssss.................. [ 61%] ...............................................................s [ 67%] joblib/test/test_numpy_pickle_compat.py . [ 67%] joblib/test/test_numpy_pickle_utils.py .. [ 67%] joblib/test/test_parallel.py sssss...................................... [ 71%] ........................................................................ [ 77%] ........................................................................ [ 84%] ........FFF.......[DEBUG:MainProcess:Thread-505] worker handler exiting [DEBUG:MainProcess:Thread-508] worker handler exiting [DEBUG:MainProcess:Thread-506] task handler got sentinel [DEBUG:MainProcess:Thread-506] task handler sending sentinel to result handler [DEBUG:MainProcess:Thread-506] task handler sending sentinel to workers [DEBUG:MainProcess:Thread-506] task handler exiting [DEBUG:MainProcess:Thread-486] joining task handler [DEBUG:MainProcess:Thread-487] joining task handler ...................................................... [ 90%] ....s.ss.s...s.ss.s...s.ss.s............................................ [ 97%] ...............ss... [ 99%] joblib/test/test_store_backends.py ... [ 99%] joblib/test/test_testing.py ..... [ 99%] joblib/test/data/create_numpy_pickle.py s [100%] =================================== FAILURES =================================== ________________ test_dispatch_multiprocessing[multiprocessing] ________________ backend = 'multiprocessing' @with_multiprocessing @parametrize('backend', PARALLEL_BACKENDS) def test_dispatch_multiprocessing(backend): """ Check that using pre_dispatch Parallel does indeed dispatch items lazily. """ > manager = mp.Manager() joblib/test/test_parallel.py:448: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /nix/store/pwqcjzvh0slsrx1qylc9b95k216bd6zs-python3-3.7.4/lib/python3.7/multiprocessing/context.py:56: in Manager m.start() /nix/store/pwqcjzvh0slsrx1qylc9b95k216bd6zs-python3-3.7.4/lib/python3.7/multiprocessing/managers.py:567: in start self._address = reader.recv() /nix/store/pwqcjzvh0slsrx1qylc9b95k216bd6zs-python3-3.7.4/lib/python3.7/multiprocessing/connection.py:250: in recv buf = self._recv_bytes() /nix/store/pwqcjzvh0slsrx1qylc9b95k216bd6zs-python3-3.7.4/lib/python3.7/multiprocessing/connection.py:407: in _recv_bytes buf = self._recv(4) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = , size = 4 read = def _recv(self, size, read=_read): buf = io.BytesIO() handle = self._handle remaining = size while remaining > 0: chunk = read(handle, remaining) n = len(chunk) if n == 0: if remaining == size: > raise EOFError E EOFError /nix/store/pwqcjzvh0slsrx1qylc9b95k216bd6zs-python3-3.7.4/lib/python3.7/multiprocessing/connection.py:383: EOFError ----------------------------- Captured stderr call ----------------------------- [INFO:SyncManager-338:MainThread] child process calling self.run() [INFO:SyncManager-338:MainThread] created temp directory /private/var/folders/ck/ht80_l5d4c9_tnwmd5lqsxj40000gn/T/nix-build-python3.7-joblib-0.13.2.drv-0/pymp-grdx4c0r [INFO:SyncManager-338:MainThread] process shutting down [DEBUG:SyncManager-338:MainThread] running all "atexit" finalizers with priority >= 0 [DEBUG:SyncManager-338:MainThread] running the remaining "atexit" finalizers Process SyncManager-338: Traceback (most recent call last): File "/nix/store/pwqcjzvh0slsrx1qylc9b95k216bd6zs-python3-3.7.4/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/nix/store/pwqcjzvh0slsrx1qylc9b95k216bd6zs-python3-3.7.4/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/nix/store/pwqcjzvh0slsrx1qylc9b95k216bd6zs-python3-3.7.4/lib/python3.7/multiprocessing/managers.py", line 589, in _run_server server = cls._Server(registry, address, authkey, serializer) File "/nix/store/pwqcjzvh0slsrx1qylc9b95k216bd6zs-python3-3.7.4/lib/python3.7/multiprocessing/managers.py", line 145, in __init__ self.listener = Listener(address=address, backlog=16) File "/nix/store/pwqcjzvh0slsrx1qylc9b95k216bd6zs-python3-3.7.4/lib/python3.7/multiprocessing/connection.py", line 438, in __init__ self._listener = SocketListener(address, family, backlog) File "/nix/store/pwqcjzvh0slsrx1qylc9b95k216bd6zs-python3-3.7.4/lib/python3.7/multiprocessing/connection.py", line 581, in __init__ self._socket.bind(address) OSError: AF_UNIX path too long [INFO:SyncManager-338:MainThread] process exiting with exitcode 1 _____________________ test_dispatch_multiprocessing[loky] ______________________ backend = 'loky' @with_multiprocessing @parametrize('backend', PARALLEL_BACKENDS) def test_dispatch_multiprocessing(backend): """ Check that using pre_dispatch Parallel does indeed dispatch items lazily. """ > manager = mp.Manager() joblib/test/test_parallel.py:448: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /nix/store/pwqcjzvh0slsrx1qylc9b95k216bd6zs-python3-3.7.4/lib/python3.7/multiprocessing/context.py:56: in Manager m.start() /nix/store/pwqcjzvh0slsrx1qylc9b95k216bd6zs-python3-3.7.4/lib/python3.7/multiprocessing/managers.py:567: in start self._address = reader.recv() /nix/store/pwqcjzvh0slsrx1qylc9b95k216bd6zs-python3-3.7.4/lib/python3.7/multiprocessing/connection.py:250: in recv buf = self._recv_bytes() /nix/store/pwqcjzvh0slsrx1qylc9b95k216bd6zs-python3-3.7.4/lib/python3.7/multiprocessing/connection.py:407: in _recv_bytes buf = self._recv(4) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = , size = 4 read = def _recv(self, size, read=_read): buf = io.BytesIO() handle = self._handle remaining = size while remaining > 0: chunk = read(handle, remaining) n = len(chunk) if n == 0: if remaining == size: > raise EOFError E EOFError /nix/store/pwqcjzvh0slsrx1qylc9b95k216bd6zs-python3-3.7.4/lib/python3.7/multiprocessing/connection.py:383: EOFError ----------------------------- Captured stderr call ----------------------------- [INFO:SyncManager-339:MainThread] child process calling self.run() [INFO:SyncManager-339:MainThread] created temp directory /private/var/folders/ck/ht80_l5d4c9_tnwmd5lqsxj40000gn/T/nix-build-python3.7-joblib-0.13.2.drv-0/pymp-e04hzocl [INFO:SyncManager-339:MainThread] process shutting down [DEBUG:SyncManager-339:MainThread] running all "atexit" finalizers with priority >= 0 [DEBUG:SyncManager-339:MainThread] running the remaining "atexit" finalizers Process SyncManager-339: Traceback (most recent call last): File "/nix/store/pwqcjzvh0slsrx1qylc9b95k216bd6zs-python3-3.7.4/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/nix/store/pwqcjzvh0slsrx1qylc9b95k216bd6zs-python3-3.7.4/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/nix/store/pwqcjzvh0slsrx1qylc9b95k216bd6zs-python3-3.7.4/lib/python3.7/multiprocessing/managers.py", line 589, in _run_server server = cls._Server(registry, address, authkey, serializer) File "/nix/store/pwqcjzvh0slsrx1qylc9b95k216bd6zs-python3-3.7.4/lib/python3.7/multiprocessing/managers.py", line 145, in __init__ self.listener = Listener(address=address, backlog=16) File "/nix/store/pwqcjzvh0slsrx1qylc9b95k216bd6zs-python3-3.7.4/lib/python3.7/multiprocessing/connection.py", line 438, in __init__ self._listener = SocketListener(address, family, backlog) File "/nix/store/pwqcjzvh0slsrx1qylc9b95k216bd6zs-python3-3.7.4/lib/python3.7/multiprocessing/connection.py", line 581, in __init__ self._socket.bind(address) OSError: AF_UNIX path too long [INFO:SyncManager-339:MainThread] process exiting with exitcode 1 ___________________ test_dispatch_multiprocessing[threading] ___________________ backend = 'threading' @with_multiprocessing @parametrize('backend', PARALLEL_BACKENDS) def test_dispatch_multiprocessing(backend): """ Check that using pre_dispatch Parallel does indeed dispatch items lazily. """ > manager = mp.Manager() joblib/test/test_parallel.py:448: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /nix/store/pwqcjzvh0slsrx1qylc9b95k216bd6zs-python3-3.7.4/lib/python3.7/multiprocessing/context.py:56: in Manager m.start() /nix/store/pwqcjzvh0slsrx1qylc9b95k216bd6zs-python3-3.7.4/lib/python3.7/multiprocessing/managers.py:567: in start self._address = reader.recv() /nix/store/pwqcjzvh0slsrx1qylc9b95k216bd6zs-python3-3.7.4/lib/python3.7/multiprocessing/connection.py:250: in recv buf = self._recv_bytes() /nix/store/pwqcjzvh0slsrx1qylc9b95k216bd6zs-python3-3.7.4/lib/python3.7/multiprocessing/connection.py:407: in _recv_bytes buf = self._recv(4) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = , size = 4 read = def _recv(self, size, read=_read): buf = io.BytesIO() handle = self._handle remaining = size while remaining > 0: chunk = read(handle, remaining) n = len(chunk) if n == 0: if remaining == size: > raise EOFError E EOFError /nix/store/pwqcjzvh0slsrx1qylc9b95k216bd6zs-python3-3.7.4/lib/python3.7/multiprocessing/connection.py:383: EOFError ----------------------------- Captured stderr call ----------------------------- [INFO:SyncManager-340:MainThread] child process calling self.run() [INFO:SyncManager-340:MainThread] created temp directory /private/var/folders/ck/ht80_l5d4c9_tnwmd5lqsxj40000gn/T/nix-build-python3.7-joblib-0.13.2.drv-0/pymp-c4ghaxyf [INFO:SyncManager-340:MainThread] process shutting down [DEBUG:SyncManager-340:MainThread] running all "atexit" finalizers with priority >= 0 [DEBUG:SyncManager-340:MainThread] running the remaining "atexit" finalizers Process SyncManager-340: Traceback (most recent call last): File "/nix/store/pwqcjzvh0slsrx1qylc9b95k216bd6zs-python3-3.7.4/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/nix/store/pwqcjzvh0slsrx1qylc9b95k216bd6zs-python3-3.7.4/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/nix/store/pwqcjzvh0slsrx1qylc9b95k216bd6zs-python3-3.7.4/lib/python3.7/multiprocessing/managers.py", line 589, in _run_server server = cls._Server(registry, address, authkey, serializer) File "/nix/store/pwqcjzvh0slsrx1qylc9b95k216bd6zs-python3-3.7.4/lib/python3.7/multiprocessing/managers.py", line 145, in __init__ self.listener = Listener(address=address, backlog=16) File "/nix/store/pwqcjzvh0slsrx1qylc9b95k216bd6zs-python3-3.7.4/lib/python3.7/multiprocessing/connection.py", line 438, in __init__ self._listener = SocketListener(address, family, backlog) File "/nix/store/pwqcjzvh0slsrx1qylc9b95k216bd6zs-python3-3.7.4/lib/python3.7/multiprocessing/connection.py", line 581, in __init__ self._socket.bind(address) OSError: AF_UNIX path too long [INFO:SyncManager-340:MainThread] process exiting with exitcode 1 ======= 3 failed, 1028 passed, 79 skipped, 1 deselected in 81.40 seconds ======= [INFO:MainProcess:MainThread] process shutting down [DEBUG:MainProcess:MainThread] running all "atexit" finalizers with priority >= 0 [DEBUG:MainProcess:MainThread] Interpreter shutting down. Waking up queue_manager_threads [(, )] [DEBUG:MainProcess:QueueManagerThread] queue management thread shutting down [DEBUG:MainProcess:QueueManagerThread] closing call_queue [DEBUG:MainProcess:QueueManagerThread] telling queue thread to quit [DEBUG:MainProcess:QueueManagerThread] joining processes [DEBUG:MainProcess:QueueFeederThread] feeder thread got sentinel -- exiting [DEBUG:MainProcess:QueueManagerThread] queue management thread clean shutdown of worker processes: [] [DEBUG:MainProcess:MainThread] running the remaining "atexit" finalizers builder for '/nix/store/42791m8ybfxma4cz058haj5criwiwlvk-python3.7-joblib-0.13.2.drv' failed with exit code 1 error: build of '/nix/store/42791m8ybfxma4cz058haj5criwiwlvk-python3.7-joblib-0.13.2.drv' failed ```

Metadata Please run nix run nixpkgs.nix-info -c nix-info -m and paste the result.

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute: python3Packages.joblib

Possibly related to #21437

kalbasit commented 4 years ago

Never mind. This is caused by #64962. I'll fix it there.

On the master branch, I removed the derivation from the nix store with nix-store --delete --ignore-liveness /nix/store/dix39a6m4w7a5vw6bg0h85ip8dr538in-python3.7-joblib-0.13.2 and I built it without substitutes; nix-build --option build-use-substitutes false -A python3Packages.joblib failed with the same message. This is not caused by #63962, but help me discover it due to the update of the requests python package.

zimbatm commented 4 years ago

maintainer: @costrouc

costrouc commented 4 years ago

@zimbatm I don't have a way to test on joblib for darwin so this is why I didn't have input on this PR. Is there an easy way to test nixpkgs on darwin? Hackintosh via kvm?

zimbatm commented 4 years ago

There is https://github.com/kholia/OSX-KVM that works with some tweaking but I wouldn't describe it as easy. Another option is to ask @ GrahamcOfBorg to do the builds on the PR if there are not too many iterations to do.