search

NixOS / nix

Nix, the purely functional package manager
https://nixos.org/
GNU Lesser General Public License v2.1
12.58k stars 1.51k forks source link

Recursive Nix deadlocks often #4216

Open grahamc opened 3 years ago

grahamc commented 3 years ago

When using https://github.com/grahamc/netboot.nix/ and 2.4pre20201102_550e11f on 20.09, Nix often deadlocks in the daemon. Usually it'l lmake some progress then I'll pkill -9 nix-daemon and try again until it finishes.

Here are the nix-daemon processes:

[grahamc@Petunia:~]$ pgrep -af nix-daemon
536 nix-daemon 26379
537 nix-daemon 26379
26388 nix-daemon --daemon
26397 nix-daemon 26379
Process statuses: ``` PID: 536 Name: nix-daemon Umask: 0022 State: S (sleeping) Tgid: 536 Ngid: 0 Pid: 536 PPid: 26397 TracerPid: 0 Uid: 0 0 0 0 Gid: 0 0 0 0 FDSize: 512 Groups: NStgid: 536 1 NSpid: 536 1 NSpgid: 536 1 NSsid: 536 1 VmPeak: 1250552 kB VmSize: 1250552 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 23860 kB VmRSS: 23860 kB RssAnon: 22748 kB RssFile: 1112 kB RssShmem: 0 kB VmData: 620432 kB VmStk: 132 kB VmExe: 1636 kB VmLib: 16944 kB VmPTE: 376 kB VmSwap: 0 kB HugetlbPages: 0 kB CoreDumping: 0 THP_enabled: 1 Threads: 1 SigQ: 0/62567 SigPnd: 0000000000000000 ShdPnd: 0000000000000000 SigBlk: 0000000000000000 SigIgn: 0000000000001000 SigCgt: 00000001a0800600 CapInh: 0000000000000000 CapPrm: 000000ffffffffff CapEff: 000000ffffffffff CapBnd: 000000ffffffffff CapAmb: 0000000000000000 NoNewPrivs: 1 Seccomp: 2 Speculation_Store_Bypass: thread force mitigated Cpus_allowed: ff Cpus_allowed_list: 0-7 Mems_allowed: 00000000,00000001 Mems_allowed_list: 0 voluntary_ctxt_switches: 4 nonvoluntary_ctxt_switches: 0 PID: 537 Name: nix-daemon Umask: 0022 State: S (sleeping) Tgid: 537 Ngid: 0 Pid: 537 PPid: 26397 TracerPid: 0 Uid: 0 0 0 0 Gid: 0 0 0 0 FDSize: 512 Groups: NStgid: 537 1 NSpid: 537 1 NSpgid: 537 1 NSsid: 537 1 VmPeak: 1250552 kB VmSize: 1250552 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 23704 kB VmRSS: 23704 kB RssAnon: 22748 kB RssFile: 956 kB RssShmem: 0 kB VmData: 620432 kB VmStk: 132 kB VmExe: 1636 kB VmLib: 16944 kB VmPTE: 376 kB VmSwap: 0 kB HugetlbPages: 0 kB CoreDumping: 0 THP_enabled: 1 Threads: 1 SigQ: 0/62567 SigPnd: 0000000000000000 ShdPnd: 0000000000000000 SigBlk: 0000000000000000 SigIgn: 0000000000001000 SigCgt: 00000001a0800600 CapInh: 0000000000000000 CapPrm: 000000ffffffffff CapEff: 000000ffffffffff CapBnd: 000000ffffffffff CapAmb: 0000000000000000 NoNewPrivs: 1 Seccomp: 2 Speculation_Store_Bypass: thread force mitigated Cpus_allowed: ff Cpus_allowed_list: 0-7 Mems_allowed: 00000000,00000001 Mems_allowed_list: 0 voluntary_ctxt_switches: 3 nonvoluntary_ctxt_switches: 0 PID: 26388 Name: nix-daemon Umask: 0022 State: S (sleeping) Tgid: 26388 Ngid: 0 Pid: 26388 PPid: 1 TracerPid: 0 Uid: 0 0 0 0 Gid: 0 0 0 0 FDSize: 512 Groups: NStgid: 26388 NSpid: 26388 NSpgid: 26388 NSsid: 26388 VmPeak: 495944 kB VmSize: 495944 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 13776 kB VmRSS: 13776 kB RssAnon: 2864 kB RssFile: 10904 kB RssShmem: 8 kB VmData: 463640 kB VmStk: 132 kB VmExe: 1636 kB VmLib: 16944 kB VmPTE: 148 kB VmSwap: 0 kB HugetlbPages: 0 kB CoreDumping: 0 THP_enabled: 1 Threads: 9 SigQ: 0/62567 SigPnd: 0000000000000000 ShdPnd: 0000000000000000 SigBlk: 0000000008005003 SigIgn: 0000000000001000 SigCgt: 00000001a0810600 CapInh: 0000000000000000 CapPrm: 000000ffffffffff CapEff: 000000ffffffffff CapBnd: 000000ffffffffff CapAmb: 0000000000000000 NoNewPrivs: 0 Seccomp: 0 Speculation_Store_Bypass: thread vulnerable Cpus_allowed: ff Cpus_allowed_list: 0-7 Mems_allowed: 00000000,00000001 Mems_allowed_list: 0 voluntary_ctxt_switches: 10 nonvoluntary_ctxt_switches: 2 PID: 26397 Name: nix-daemon Umask: 0022 State: S (sleeping) Tgid: 26397 Ngid: 0 Pid: 26397 PPid: 26388 TracerPid: 0 Uid: 0 0 0 0 Gid: 0 0 0 0 FDSize: 512 Groups: NStgid: 26397 NSpid: 26397 NSpgid: 26397 NSsid: 26397 VmPeak: 1332480 kB VmSize: 1249528 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 28180 kB VmRSS: 28180 kB RssAnon: 22732 kB RssFile: 5436 kB RssShmem: 12 kB VmData: 619408 kB VmStk: 132 kB VmExe: 1636 kB VmLib: 16944 kB VmPTE: 380 kB VmSwap: 0 kB HugetlbPages: 0 kB CoreDumping: 0 THP_enabled: 1 Threads: 6 SigQ: 0/62567 SigPnd: 0000000000000000 ShdPnd: 0000000000000000 SigBlk: 0000000008005003 SigIgn: 0000000000001000 SigCgt: 00000001a0800600 CapInh: 0000000000000000 CapPrm: 000000ffffffffff CapEff: 000000ffffffffff CapBnd: 000000ffffffffff CapAmb: 0000000000000000 NoNewPrivs: 0 Seccomp: 0 Speculation_Store_Bypass: thread vulnerable Cpus_allowed: ff Cpus_allowed_list: 0-7 Mems_allowed: 00000000,00000001 Mems_allowed_list: 0 voluntary_ctxt_switches: 103 nonvoluntary_ctxt_switches: 11 ```
Stack info: ``` [nix-shell:~]$ sudo eu-stack -p 536 PID 536 - process TID 536: #0 0x00007f1b165fbf74 read #1 0x00007f1b16a261ab nix::drainFD(int, nix::Sink&, bool) #2 0x00007f1b16a2795b nix::drainFD[abi:cxx11](int, bool, unsigned long) #3 0x00007f1b16b9471d nix::DerivationGoal::runChild() #4 0x00007f1b16b96879 nix::childEntry(void*) #5 0x00007f1b16522d2f __clone [nix-shell:~]$ sudo eu-stack -p 537 PID 537 - process TID 537: #0 0x00007f1b165fbf74 read #1 0x00007f1b16a261ab nix::drainFD(int, nix::Sink&, bool) #2 0x00007f1b16a2795b nix::drainFD[abi:cxx11](int, bool, unsigned long) #3 0x00007f1b16b9471d nix::DerivationGoal::runChild() #4 0x00007f1b16b96879 nix::childEntry(void*) #5 0x00007f1b16522d2f __clone [nix-shell:~]$ sudo eu-stack -p 26388 PID 26388 - process TID 26388: #0 0x00007f1b165fc097 accept #1 0x00000000004b56ac main_nix_daemon(int, char**) #2 0x000000000056dddb nix::mainWrapped(int, char**) #3 0x00007f1b16e81edd nix::handleExceptions(std::__cxx11::basic_string, std::allocator > const&, std::function) #4 0x00000000004713bb main #5 0x00007f1b1644ec7d __libc_start_main #6 0x0000000000472b7a _start TID 26389: #0 0x00007f1b16463c7c __sigtimedwait #1 0x00007f1b165fcae4 sigwait #2 0x00007f1b16a22f45 nix::signalHandlerThread(__sigset_t) #3 0x00007f1b16a2a6ad std::thread::_State_impl > >::_M_run() #4 0x00007f1b1683f930 #5 0x00007f1b165f1ead start_thread #6 0x00007f1b16522d2f __clone TID 26390: #0 0x00007f1b165f888d pthread_cond_wait@@GLIBC_2.3.2 #1 0x00007f1b16eb1ec7 GC_wait_marker #2 0x00007f1b16eb234a GC_help_marker #3 0x00007f1b16eb241f GC_mark_thread #4 0x00007f1b165f1ead start_thread #5 0x00007f1b16522d2f __clone TID 26391: #0 0x00007f1b165f888d pthread_cond_wait@@GLIBC_2.3.2 #1 0x00007f1b16eb1ec7 GC_wait_marker #2 0x00007f1b16eb234a GC_help_marker #3 0x00007f1b16eb241f GC_mark_thread #4 0x00007f1b165f1ead start_thread #5 0x00007f1b16522d2f __clone TID 26392: #0 0x00007f1b165f888d pthread_cond_wait@@GLIBC_2.3.2 #1 0x00007f1b16eb1ec7 GC_wait_marker #2 0x00007f1b16eb234a GC_help_marker #3 0x00007f1b16eb241f GC_mark_thread #4 0x00007f1b165f1ead start_thread #5 0x00007f1b16522d2f __clone TID 26393: #0 0x00007f1b165f888d pthread_cond_wait@@GLIBC_2.3.2 #1 0x00007f1b16eb1ec7 GC_wait_marker #2 0x00007f1b16eb234a GC_help_marker #3 0x00007f1b16eb241f GC_mark_thread #4 0x00007f1b165f1ead start_thread #5 0x00007f1b16522d2f __clone TID 26394: #0 0x00007f1b165f888d pthread_cond_wait@@GLIBC_2.3.2 #1 0x00007f1b16eb1ec7 GC_wait_marker #2 0x00007f1b16eb234a GC_help_marker #3 0x00007f1b16eb241f GC_mark_thread #4 0x00007f1b165f1ead start_thread #5 0x00007f1b16522d2f __clone TID 26395: #0 0x00007f1b165f888d pthread_cond_wait@@GLIBC_2.3.2 #1 0x00007f1b16eb1ec7 GC_wait_marker #2 0x00007f1b16eb234a GC_help_marker #3 0x00007f1b16eb241f GC_mark_thread #4 0x00007f1b165f1ead start_thread #5 0x00007f1b16522d2f __clone TID 26396: #0 0x00007f1b165f888d pthread_cond_wait@@GLIBC_2.3.2 #1 0x00007f1b16eb1ec7 GC_wait_marker #2 0x00007f1b16eb234a GC_help_marker #3 0x00007f1b16eb241f GC_mark_thread #4 0x00007f1b165f1ead start_thread #5 0x00007f1b16522d2f __clone [nix-shell:~]$ sudo eu-stack -p 26397 PID 26397 - process TID 26397: #0 0x00007f1b16518dd9 __poll #1 0x00007f1b16be732f nix::Worker::waitForInput() #2 0x00007f1b16be85e9 nix::Worker::run(std::set, nix::CompareGoalPtrs, std::allocator > > const&) #3 0x00007f1b16bdbebb nix::LocalStore::buildPaths(std::vector > const&, nix::BuildMode) #4 0x00007f1b16bfb139 nix::daemon::processConnection(nix::ref, nix::FdSource&, nix::FdSink&, nix::daemon::TrustedFlag, nix::daemon::RecursiveFlag, std::function) #5 0x00000000004b52d5 std::_Function_handler::_M_invoke(std::_Any_data const&) #6 0x00007f1b16a23d7c std::_Function_handler, nix::ProcessOptions const&)::{lambda()#1}>::_M_invoke(std::_Any_data const&) #7 0x00007f1b16a20729 nix::doFork(bool, std::function) #8 0x00007f1b16a23c69 nix::startProcess(std::function, nix::ProcessOptions const&) #9 0x00000000004b6493 main_nix_daemon(int, char**) #10 0x000000000056dddb nix::mainWrapped(int, char**) #11 0x00007f1b16e81edd nix::handleExceptions(std::__cxx11::basic_string, std::allocator > const&, std::function) #12 0x00000000004713bb main #13 0x00007f1b1644ec7d __libc_start_main #14 0x0000000000472b7a _start TID 26398: #0 0x00007f1b16518dd9 __poll #1 0x00000000004ee79e std::thread::_State_impl > >::_M_run() #2 0x00007f1b1683f930 #3 0x00007f1b165f1ead start_thread #4 0x00007f1b16522d2f __clone TID 26422: #0 0x00007f1b165fc097 accept #1 0x00007f1b16b93dad nix::DerivationGoal::startDaemon()::{lambda()#1}::operator()() const #2 0x00007f1b1683f930 #3 0x00007f1b165f1ead start_thread #4 0x00007f1b16522d2f __clone TID 26444: #0 0x00007f1b165fbf74 read #1 0x00007f1b16a25f3f nix::readLine[abi:cxx11](int) #2 0x00007f1b16ba44c2 nix::DerivationGoal::startBuilder() #3 0x00007f1b16ba81da nix::DerivationGoal::tryLocalBuild() #4 0x00007f1b16be8563 nix::Worker::run(std::set, nix::CompareGoalPtrs, std::allocator > > const&) #5 0x00007f1b16bdbebb nix::LocalStore::buildPaths(std::vector > const&, nix::BuildMode) #6 0x00007f1b16bc7673 nix::RestrictedStore::buildPaths(std::vector > const&, nix::BuildMode) #7 0x00007f1b16bfb139 nix::daemon::processConnection(nix::ref, nix::FdSource&, nix::FdSink&, nix::daemon::TrustedFlag, nix::daemon::RecursiveFlag, std::function) #8 0x00007f1b16b84c80 std::thread::_State_impl > >::_M_run() #9 0x00007f1b1683f930 #10 0x00007f1b165f1ead start_thread #11 0x00007f1b16522d2f __clone TID 26445: #0 0x00007f1b165fc097 accept #1 0x00007f1b16b93dad nix::DerivationGoal::startDaemon()::{lambda()#1}::operator()() const #2 0x00007f1b1683f930 #3 0x00007f1b165f1ead start_thread #4 0x00007f1b16522d2f __clone TID 26538: #0 0x00007f1b165fbf74 read #1 0x00007f1b16a25f3f nix::readLine[abi:cxx11](int) #2 0x00007f1b16ba44c2 nix::DerivationGoal::startBuilder() #3 0x00007f1b16ba81da nix::DerivationGoal::tryLocalBuild() #4 0x00007f1b16be8563 nix::Worker::run(std::set, nix::CompareGoalPtrs, std::allocator > > const&) #5 0x00007f1b16bdbebb nix::LocalStore::buildPaths(std::vector > const&, nix::BuildMode) #6 0x00007f1b16bc7673 nix::RestrictedStore::buildPaths(std::vector > const&, nix::BuildMode) #7 0x00007f1b16bfb139 nix::daemon::processConnection(nix::ref, nix::FdSource&, nix::FdSink&, nix::daemon::TrustedFlag, nix::daemon::RecursiveFlag, std::function) #8 0x00007f1b16b84c80 std::thread::_State_impl > >::_M_run() #9 0x00007f1b1683f930 #10 0x00007f1b165f1ead start_thread #11 0x00007f1b16522d2f __clone ```
`fdinfo` statuses: ``` PID: 536 PID 536 FD 0: pos: 0 flags: 0100002 mnt_id: 3307 PID 536 FD 1: pos: 0 flags: 0100002 mnt_id: 376 PID 536 FD 10: pos: 495705 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY READ 26397 00:1c:2458720 0 EOF PID 536 FD 11: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:335005 0 EOF PID 536 FD 114: pos: 0 flags: 02100001 mnt_id: 365 PID 536 FD 118: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 6 PID 536 FD 119: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:5067831 0 EOF PID 536 FD 12: pos: 0 flags: 02000002 mnt_id: 9 scm_fds: 0 PID 536 FD 120: pos: 0 flags: 0100000 mnt_id: 3 PID 536 FD 124: pos: 0 flags: 02100001 mnt_id: 365 PID 536 FD 125: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 12 PID 536 FD 127: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:5068146 0 EOF PID 536 FD 128: pos: 0 flags: 0100000 mnt_id: 3 PID 536 FD 13: pos: 0 flags: 02100001 mnt_id: 365 PID 536 FD 133: pos: 0 flags: 02100001 mnt_id: 365 PID 536 FD 135: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 13 PID 536 FD 137: pos: 0 flags: 0100000 mnt_id: 3 PID 536 FD 14: pos: 0 flags: 02000002 mnt_id: 9 scm_fds: 0 PID 536 FD 141: pos: 0 flags: 0100000 mnt_id: 3 PID 536 FD 15: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 2 PID 536 FD 16: pos: 0 flags: 02100002 mnt_id: 366 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:2555099 0 EOF PID 536 FD 17: pos: 0 flags: 0100000 mnt_id: 3 PID 536 FD 18: pos: 0 flags: 02100002 mnt_id: 366 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:2555100 0 EOF PID 536 FD 19: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:335295 0 EOF PID 536 FD 2: pos: 0 flags: 0100002 mnt_id: 376 PID 536 FD 20: pos: 0 flags: 02000002 mnt_id: 9 scm_fds: 0 PID 536 FD 21: pos: 0 flags: 02100001 mnt_id: 365 PID 536 FD 22: pos: 0 flags: 02000002 mnt_id: 9 scm_fds: 0 PID 536 FD 23: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 3 PID 536 FD 24: pos: 0 flags: 02000002 mnt_id: 14 PID 536 FD 26: pos: 0 flags: 0100000 mnt_id: 3 PID 536 FD 27: pos: 0 flags: 02100002 mnt_id: 366 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:2729283 0 EOF PID 536 FD 28: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:337168 0 EOF PID 536 FD 29: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:337536 0 EOF PID 536 FD 3: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY READ 26397 00:1c:385 0 EOF PID 536 FD 30: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:338700 0 EOF PID 536 FD 306: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:335306 0 EOF PID 536 FD 307: pos: 0 flags: 02100001 mnt_id: 365 PID 536 FD 308: pos: 0 flags: 0100000 mnt_id: 3 PID 536 FD 309: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 4 PID 536 FD 31: pos: 0 flags: 0100000 mnt_id: 3 PID 536 FD 32: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:5067588 0 EOF PID 536 FD 33: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:5070745 0 EOF PID 536 FD 34: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:337921 0 EOF PID 536 FD 35: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:1877196 0 EOF PID 536 FD 36: pos: 0 flags: 02100001 mnt_id: 365 PID 536 FD 37: pos: 0 flags: 02100001 mnt_id: 365 PID 536 FD 38: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 7 PID 536 FD 39: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:337156 0 EOF PID 536 FD 4: pos: 0 flags: 02000002 mnt_id: 9 scm_fds: 0 PID 536 FD 40: pos: 0 flags: 0100002 mnt_id: 376 PID 536 FD 41: pos: 0 flags: 02100001 mnt_id: 365 PID 536 FD 42: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:5068994 0 EOF PID 536 FD 43: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 5 PID 536 FD 44: pos: 0 flags: 02100001 mnt_id: 365 PID 536 FD 45: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 16 PID 536 FD 46: pos: 0 flags: 0100000 mnt_id: 3 PID 536 FD 47: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 8 PID 536 FD 48: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:5069201 0 EOF PID 536 FD 49: pos: 0 flags: 02100002 mnt_id: 366 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:2461780 0 EOF PID 536 FD 5: pos: 0 flags: 02500002 mnt_id: 365 PID 536 FD 50: pos: 0 flags: 02100002 mnt_id: 366 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:2720603 0 EOF PID 536 FD 51: pos: 0 flags: 02100002 mnt_id: 366 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:2462330 0 EOF PID 536 FD 52: pos: 0 flags: 02000000 mnt_id: 13 PID 536 FD 53: pos: 0 flags: 02100001 mnt_id: 365 PID 536 FD 55: pos: 0 flags: 02100001 mnt_id: 365 PID 536 FD 56: pos: 0 flags: 02100001 mnt_id: 365 PID 536 FD 57: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 14 PID 536 FD 58: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 9 PID 536 FD 59: pos: 0 flags: 02100001 mnt_id: 365 PID 536 FD 6: pos: 0 flags: 02500002 mnt_id: 365 PID 536 FD 60: pos: 0 flags: 0100000 mnt_id: 3 PID 536 FD 61: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 11 PID 536 FD 62: pos: 0 flags: 0100002 mnt_id: 376 PID 536 FD 63: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 18 PID 536 FD 64: pos: 0 flags: 0100000 mnt_id: 3 PID 536 FD 65: pos: 0 flags: 02000000 mnt_id: 13 PID 536 FD 66: pos: 0 flags: 02000001 mnt_id: 13 PID 536 FD 69: pos: 0 flags: 0100000 mnt_id: 3 PID 536 FD 7: pos: 0 flags: 02500002 mnt_id: 365 PID 536 FD 70: pos: 0 flags: 0100000 mnt_id: 3 PID 536 FD 72: pos: 0 flags: 02100001 mnt_id: 365 PID 536 FD 74: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 17 PID 536 FD 8: pos: 0 flags: 02500002 mnt_id: 364 PID 536 FD 9: pos: 0 flags: 02100002 mnt_id: 366 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:2547754 0 EOF PID: 537 PID 537 FD 0: pos: 0 flags: 0100002 mnt_id: 3338 PID 537 FD 1: pos: 0 flags: 0100002 mnt_id: 376 PID 537 FD 10: pos: 495705 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY READ 26397 00:1c:2458720 0 EOF PID 537 FD 11: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:335005 0 EOF PID 537 FD 114: pos: 0 flags: 02100001 mnt_id: 365 PID 537 FD 118: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 6 PID 537 FD 119: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:5067831 0 EOF PID 537 FD 12: pos: 0 flags: 02000002 mnt_id: 9 scm_fds: 0 PID 537 FD 120: pos: 0 flags: 0100000 mnt_id: 3 PID 537 FD 124: pos: 0 flags: 02100001 mnt_id: 365 PID 537 FD 125: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 12 PID 537 FD 127: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:5068146 0 EOF PID 537 FD 128: pos: 0 flags: 0100000 mnt_id: 3 PID 537 FD 13: pos: 0 flags: 02100001 mnt_id: 365 PID 537 FD 133: pos: 0 flags: 02100001 mnt_id: 365 PID 537 FD 135: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 13 PID 537 FD 137: pos: 0 flags: 0100000 mnt_id: 3 PID 537 FD 14: pos: 0 flags: 02000002 mnt_id: 9 scm_fds: 0 PID 537 FD 141: pos: 0 flags: 0100000 mnt_id: 3 PID 537 FD 15: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 2 PID 537 FD 16: pos: 0 flags: 02100002 mnt_id: 366 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:2555099 0 EOF PID 537 FD 17: pos: 0 flags: 0100000 mnt_id: 3 PID 537 FD 18: pos: 0 flags: 02100002 mnt_id: 366 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:2555100 0 EOF PID 537 FD 19: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:335295 0 EOF PID 537 FD 2: pos: 0 flags: 0100002 mnt_id: 376 PID 537 FD 20: pos: 0 flags: 02000002 mnt_id: 9 scm_fds: 0 PID 537 FD 21: pos: 0 flags: 02100001 mnt_id: 365 PID 537 FD 22: pos: 0 flags: 02000002 mnt_id: 9 scm_fds: 0 PID 537 FD 23: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 3 PID 537 FD 24: pos: 0 flags: 02000002 mnt_id: 14 PID 537 FD 26: pos: 0 flags: 0100000 mnt_id: 3 PID 537 FD 27: pos: 0 flags: 02100002 mnt_id: 366 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:2729283 0 EOF PID 537 FD 28: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:337168 0 EOF PID 537 FD 29: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:337536 0 EOF PID 537 FD 3: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY READ 26397 00:1c:385 0 EOF PID 537 FD 30: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:338700 0 EOF PID 537 FD 306: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:335306 0 EOF PID 537 FD 307: pos: 0 flags: 02100001 mnt_id: 365 PID 537 FD 308: pos: 0 flags: 0100000 mnt_id: 3 PID 537 FD 309: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 4 PID 537 FD 31: pos: 0 flags: 0100000 mnt_id: 3 PID 537 FD 32: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:5067588 0 EOF PID 537 FD 33: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:5070745 0 EOF PID 537 FD 34: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:337921 0 EOF PID 537 FD 35: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:1877196 0 EOF PID 537 FD 36: pos: 0 flags: 02100001 mnt_id: 365 PID 537 FD 37: pos: 0 flags: 02100001 mnt_id: 365 PID 537 FD 38: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 7 PID 537 FD 39: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:337156 0 EOF PID 537 FD 4: pos: 0 flags: 02000002 mnt_id: 9 scm_fds: 0 PID 537 FD 40: pos: 0 flags: 0100002 mnt_id: 376 PID 537 FD 41: pos: 0 flags: 02100001 mnt_id: 365 PID 537 FD 42: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:5068994 0 EOF PID 537 FD 43: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 5 PID 537 FD 44: pos: 0 flags: 02100001 mnt_id: 365 PID 537 FD 45: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 16 PID 537 FD 46: pos: 0 flags: 0100000 mnt_id: 3 PID 537 FD 47: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 8 PID 537 FD 48: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:5069201 0 EOF PID 537 FD 49: pos: 0 flags: 02100002 mnt_id: 366 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:2461780 0 EOF PID 537 FD 5: pos: 0 flags: 02500002 mnt_id: 365 PID 537 FD 50: pos: 0 flags: 02100002 mnt_id: 366 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:2720603 0 EOF PID 537 FD 51: pos: 0 flags: 02100002 mnt_id: 366 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:2462330 0 EOF PID 537 FD 52: pos: 0 flags: 02000000 mnt_id: 13 PID 537 FD 53: pos: 0 flags: 02100001 mnt_id: 365 PID 537 FD 54: pos: 0 flags: 02000001 mnt_id: 13 PID 537 FD 55: pos: 0 flags: 02100001 mnt_id: 365 PID 537 FD 56: pos: 0 flags: 02100001 mnt_id: 365 PID 537 FD 57: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 14 PID 537 FD 58: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 9 PID 537 FD 59: pos: 0 flags: 02100001 mnt_id: 365 PID 537 FD 6: pos: 0 flags: 02500002 mnt_id: 365 PID 537 FD 60: pos: 0 flags: 0100000 mnt_id: 3 PID 537 FD 61: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 11 PID 537 FD 62: pos: 0 flags: 0100002 mnt_id: 376 PID 537 FD 63: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 18 PID 537 FD 64: pos: 0 flags: 0100000 mnt_id: 3 PID 537 FD 65: pos: 0 flags: 02000000 mnt_id: 13 PID 537 FD 69: pos: 0 flags: 0100000 mnt_id: 3 PID 537 FD 7: pos: 0 flags: 02500002 mnt_id: 365 PID 537 FD 70: pos: 0 flags: 0100000 mnt_id: 3 PID 537 FD 72: pos: 0 flags: 02100001 mnt_id: 365 PID 537 FD 74: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 17 PID 537 FD 8: pos: 0 flags: 02500002 mnt_id: 364 PID 537 FD 9: pos: 0 flags: 02100002 mnt_id: 366 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:2547754 0 EOF PID: 26388 PID 26388 FD 0: pos: 0 flags: 0100000 mnt_id: 21 PID 26388 FD 1: pos: 0 flags: 02 mnt_id: 9 scm_fds: 0 PID 26388 FD 2: pos: 0 flags: 02 mnt_id: 9 scm_fds: 0 PID 26388 FD 3: pos: 0 flags: 02000002 mnt_id: 9 scm_fds: 0 PID: 26397 PID 26397 FD 0: pos: 0 flags: 0100000 mnt_id: 21 PID 26397 FD 1: pos: 0 flags: 02 mnt_id: 9 scm_fds: 0 PID 26397 FD 10: pos: 495705 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY READ 26397 00:1c:2458720 0 EOF PID 26397 FD 11: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:335005 0 EOF PID 26397 FD 114: pos: 0 flags: 02100001 mnt_id: 365 PID 26397 FD 118: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 6 PID 26397 FD 119: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:5067831 0 EOF PID 26397 FD 12: pos: 0 flags: 02000002 mnt_id: 9 scm_fds: 0 PID 26397 FD 120: pos: 0 flags: 0100000 mnt_id: 3 PID 26397 FD 124: pos: 0 flags: 02100001 mnt_id: 365 PID 26397 FD 125: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 12 PID 26397 FD 127: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:5068146 0 EOF PID 26397 FD 128: pos: 0 flags: 0100000 mnt_id: 3 PID 26397 FD 13: pos: 0 flags: 02100001 mnt_id: 365 PID 26397 FD 133: pos: 0 flags: 02100001 mnt_id: 365 PID 26397 FD 135: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 13 PID 26397 FD 137: pos: 0 flags: 0100000 mnt_id: 3 PID 26397 FD 14: pos: 0 flags: 02000002 mnt_id: 9 scm_fds: 0 PID 26397 FD 141: pos: 0 flags: 0100000 mnt_id: 3 PID 26397 FD 15: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 2 PID 26397 FD 16: pos: 0 flags: 02100002 mnt_id: 366 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:2555099 0 EOF PID 26397 FD 17: pos: 0 flags: 0100000 mnt_id: 3 PID 26397 FD 18: pos: 0 flags: 02100002 mnt_id: 366 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:2555100 0 EOF PID 26397 FD 19: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:335295 0 EOF PID 26397 FD 2: pos: 0 flags: 02 mnt_id: 9 scm_fds: 0 PID 26397 FD 20: pos: 0 flags: 02000002 mnt_id: 9 scm_fds: 0 PID 26397 FD 21: pos: 0 flags: 02100001 mnt_id: 365 PID 26397 FD 22: pos: 0 flags: 02000002 mnt_id: 9 scm_fds: 0 PID 26397 FD 23: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 3 PID 26397 FD 26: pos: 0 flags: 0100000 mnt_id: 3 PID 26397 FD 27: pos: 0 flags: 02100002 mnt_id: 366 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:2729283 0 EOF PID 26397 FD 28: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:337168 0 EOF PID 26397 FD 29: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:337536 0 EOF PID 26397 FD 3: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY READ 26397 00:1c:385 0 EOF PID 26397 FD 30: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:338700 0 EOF PID 26397 FD 306: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:335306 0 EOF PID 26397 FD 307: pos: 0 flags: 02100001 mnt_id: 365 PID 26397 FD 308: pos: 0 flags: 0100000 mnt_id: 3 PID 26397 FD 309: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 4 PID 26397 FD 31: pos: 0 flags: 0100000 mnt_id: 3 PID 26397 FD 32: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:5067588 0 EOF PID 26397 FD 33: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:5070745 0 EOF PID 26397 FD 34: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:337921 0 EOF PID 26397 FD 35: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:1877196 0 EOF PID 26397 FD 36: pos: 0 flags: 02100001 mnt_id: 365 PID 26397 FD 37: pos: 0 flags: 02100001 mnt_id: 365 PID 26397 FD 38: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 7 PID 26397 FD 39: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:337156 0 EOF PID 26397 FD 4: pos: 0 flags: 02000002 mnt_id: 9 scm_fds: 0 PID 26397 FD 40: pos: 0 flags: 0100000 mnt_id: 3 PID 26397 FD 41: pos: 0 flags: 02100001 mnt_id: 365 PID 26397 FD 42: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:5068994 0 EOF PID 26397 FD 43: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 5 PID 26397 FD 44: pos: 0 flags: 02100001 mnt_id: 365 PID 26397 FD 45: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 16 PID 26397 FD 46: pos: 0 flags: 0100000 mnt_id: 3 PID 26397 FD 47: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 8 PID 26397 FD 48: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:5069201 0 EOF PID 26397 FD 49: pos: 0 flags: 02100002 mnt_id: 366 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:2461780 0 EOF PID 26397 FD 5: pos: 0 flags: 02500002 mnt_id: 365 lock: 1: POSIX ADVISORY READ 26397 00:1c:386 1073741826 1073742335 PID 26397 FD 50: pos: 0 flags: 02100002 mnt_id: 366 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:2720603 0 EOF PID 26397 FD 51: pos: 0 flags: 02100002 mnt_id: 366 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:2462330 0 EOF PID 26397 FD 52: pos: 0 flags: 0100000 mnt_id: 3 PID 26397 FD 53: pos: 0 flags: 02100001 mnt_id: 365 PID 26397 FD 55: pos: 0 flags: 02100001 mnt_id: 365 PID 26397 FD 56: pos: 0 flags: 02100001 mnt_id: 365 PID 26397 FD 57: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 14 PID 26397 FD 58: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 9 PID 26397 FD 59: pos: 0 flags: 02100001 mnt_id: 365 PID 26397 FD 6: pos: 0 flags: 02500002 mnt_id: 365 PID 26397 FD 60: pos: 0 flags: 0100000 mnt_id: 3 PID 26397 FD 61: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 11 PID 26397 FD 63: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 18 PID 26397 FD 64: pos: 0 flags: 0100000 mnt_id: 3 PID 26397 FD 69: pos: 0 flags: 0100000 mnt_id: 3 PID 26397 FD 7: pos: 0 flags: 02500002 mnt_id: 365 lock: 1: POSIX ADVISORY READ 26397 00:1c:2941973 128 128 PID 26397 FD 70: pos: 0 flags: 0100000 mnt_id: 3 PID 26397 FD 72: pos: 0 flags: 02100001 mnt_id: 365 PID 26397 FD 74: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 17 PID 26397 FD 8: pos: 0 flags: 02500002 mnt_id: 364 PID 26397 FD 9: pos: 0 flags: 02100002 mnt_id: 366 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:2547754 0 EOF ```
nrdxp commented 3 years ago

Ran into this recently as well on a Hydra server. Not 100% sure if Nix is being called recursively in my case though. We are using the same nix-unstable version. For now, I'm running a transient systemd service to check if the lock is held for more than 5 minutes and simply kill and restart the nix-daemon and the hydra-queue-runner if so. This seems to be enough to get things going at least.

stale[bot] commented 3 years ago

I marked this as stale due to inactivity. → More info

matthewbauer commented 2 years ago

I've gotten this too. I'm not sure if the cause is actually recursive-nix itself, or just the fact that recursive-nix makes it really easy to spawn a bunch of derivations. I've gotten something similar outside of recursive-nix , but it seems like it's a little more common in recursive-nix.