Open grahamc opened 3 years ago
Ran into this recently as well on a Hydra server. Not 100% sure if Nix is being called recursively in my case though. We are using the same nix-unstable version. For now, I'm running a transient systemd service to check if the lock is held for more than 5 minutes and simply kill and restart the nix-daemon and the hydra-queue-runner if so. This seems to be enough to get things going at least.
I marked this as stale due to inactivity. → More info
I've gotten this too. I'm not sure if the cause is actually recursive-nix itself, or just the fact that recursive-nix makes it really easy to spawn a bunch of derivations. I've gotten something similar outside of recursive-nix , but it seems like it's a little more common in recursive-nix.
When using https://github.com/grahamc/netboot.nix/ and 2.4pre20201102_550e11f on 20.09, Nix often deadlocks in the daemon. Usually it'l lmake some progress then I'll pkill -9 nix-daemon and try again until it finishes.
Here are the nix-daemon processes:
Process statuses:
``` PID: 536 Name: nix-daemon Umask: 0022 State: S (sleeping) Tgid: 536 Ngid: 0 Pid: 536 PPid: 26397 TracerPid: 0 Uid: 0 0 0 0 Gid: 0 0 0 0 FDSize: 512 Groups: NStgid: 536 1 NSpid: 536 1 NSpgid: 536 1 NSsid: 536 1 VmPeak: 1250552 kB VmSize: 1250552 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 23860 kB VmRSS: 23860 kB RssAnon: 22748 kB RssFile: 1112 kB RssShmem: 0 kB VmData: 620432 kB VmStk: 132 kB VmExe: 1636 kB VmLib: 16944 kB VmPTE: 376 kB VmSwap: 0 kB HugetlbPages: 0 kB CoreDumping: 0 THP_enabled: 1 Threads: 1 SigQ: 0/62567 SigPnd: 0000000000000000 ShdPnd: 0000000000000000 SigBlk: 0000000000000000 SigIgn: 0000000000001000 SigCgt: 00000001a0800600 CapInh: 0000000000000000 CapPrm: 000000ffffffffff CapEff: 000000ffffffffff CapBnd: 000000ffffffffff CapAmb: 0000000000000000 NoNewPrivs: 1 Seccomp: 2 Speculation_Store_Bypass: thread force mitigated Cpus_allowed: ff Cpus_allowed_list: 0-7 Mems_allowed: 00000000,00000001 Mems_allowed_list: 0 voluntary_ctxt_switches: 4 nonvoluntary_ctxt_switches: 0 PID: 537 Name: nix-daemon Umask: 0022 State: S (sleeping) Tgid: 537 Ngid: 0 Pid: 537 PPid: 26397 TracerPid: 0 Uid: 0 0 0 0 Gid: 0 0 0 0 FDSize: 512 Groups: NStgid: 537 1 NSpid: 537 1 NSpgid: 537 1 NSsid: 537 1 VmPeak: 1250552 kB VmSize: 1250552 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 23704 kB VmRSS: 23704 kB RssAnon: 22748 kB RssFile: 956 kB RssShmem: 0 kB VmData: 620432 kB VmStk: 132 kB VmExe: 1636 kB VmLib: 16944 kB VmPTE: 376 kB VmSwap: 0 kB HugetlbPages: 0 kB CoreDumping: 0 THP_enabled: 1 Threads: 1 SigQ: 0/62567 SigPnd: 0000000000000000 ShdPnd: 0000000000000000 SigBlk: 0000000000000000 SigIgn: 0000000000001000 SigCgt: 00000001a0800600 CapInh: 0000000000000000 CapPrm: 000000ffffffffff CapEff: 000000ffffffffff CapBnd: 000000ffffffffff CapAmb: 0000000000000000 NoNewPrivs: 1 Seccomp: 2 Speculation_Store_Bypass: thread force mitigated Cpus_allowed: ff Cpus_allowed_list: 0-7 Mems_allowed: 00000000,00000001 Mems_allowed_list: 0 voluntary_ctxt_switches: 3 nonvoluntary_ctxt_switches: 0 PID: 26388 Name: nix-daemon Umask: 0022 State: S (sleeping) Tgid: 26388 Ngid: 0 Pid: 26388 PPid: 1 TracerPid: 0 Uid: 0 0 0 0 Gid: 0 0 0 0 FDSize: 512 Groups: NStgid: 26388 NSpid: 26388 NSpgid: 26388 NSsid: 26388 VmPeak: 495944 kB VmSize: 495944 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 13776 kB VmRSS: 13776 kB RssAnon: 2864 kB RssFile: 10904 kB RssShmem: 8 kB VmData: 463640 kB VmStk: 132 kB VmExe: 1636 kB VmLib: 16944 kB VmPTE: 148 kB VmSwap: 0 kB HugetlbPages: 0 kB CoreDumping: 0 THP_enabled: 1 Threads: 9 SigQ: 0/62567 SigPnd: 0000000000000000 ShdPnd: 0000000000000000 SigBlk: 0000000008005003 SigIgn: 0000000000001000 SigCgt: 00000001a0810600 CapInh: 0000000000000000 CapPrm: 000000ffffffffff CapEff: 000000ffffffffff CapBnd: 000000ffffffffff CapAmb: 0000000000000000 NoNewPrivs: 0 Seccomp: 0 Speculation_Store_Bypass: thread vulnerable Cpus_allowed: ff Cpus_allowed_list: 0-7 Mems_allowed: 00000000,00000001 Mems_allowed_list: 0 voluntary_ctxt_switches: 10 nonvoluntary_ctxt_switches: 2 PID: 26397 Name: nix-daemon Umask: 0022 State: S (sleeping) Tgid: 26397 Ngid: 0 Pid: 26397 PPid: 26388 TracerPid: 0 Uid: 0 0 0 0 Gid: 0 0 0 0 FDSize: 512 Groups: NStgid: 26397 NSpid: 26397 NSpgid: 26397 NSsid: 26397 VmPeak: 1332480 kB VmSize: 1249528 kB VmLck: 0 kB VmPin: 0 kB VmHWM: 28180 kB VmRSS: 28180 kB RssAnon: 22732 kB RssFile: 5436 kB RssShmem: 12 kB VmData: 619408 kB VmStk: 132 kB VmExe: 1636 kB VmLib: 16944 kB VmPTE: 380 kB VmSwap: 0 kB HugetlbPages: 0 kB CoreDumping: 0 THP_enabled: 1 Threads: 6 SigQ: 0/62567 SigPnd: 0000000000000000 ShdPnd: 0000000000000000 SigBlk: 0000000008005003 SigIgn: 0000000000001000 SigCgt: 00000001a0800600 CapInh: 0000000000000000 CapPrm: 000000ffffffffff CapEff: 000000ffffffffff CapBnd: 000000ffffffffff CapAmb: 0000000000000000 NoNewPrivs: 0 Seccomp: 0 Speculation_Store_Bypass: thread vulnerable Cpus_allowed: ff Cpus_allowed_list: 0-7 Mems_allowed: 00000000,00000001 Mems_allowed_list: 0 voluntary_ctxt_switches: 103 nonvoluntary_ctxt_switches: 11 ```Stack info:
``` [nix-shell:~]$ sudo eu-stack -p 536 PID 536 - process TID 536: #0 0x00007f1b165fbf74 read #1 0x00007f1b16a261ab nix::drainFD(int, nix::Sink&, bool) #2 0x00007f1b16a2795b nix::drainFD[abi:cxx11](int, bool, unsigned long) #3 0x00007f1b16b9471d nix::DerivationGoal::runChild() #4 0x00007f1b16b96879 nix::childEntry(void*) #5 0x00007f1b16522d2f __clone [nix-shell:~]$ sudo eu-stack -p 537 PID 537 - process TID 537: #0 0x00007f1b165fbf74 read #1 0x00007f1b16a261ab nix::drainFD(int, nix::Sink&, bool) #2 0x00007f1b16a2795b nix::drainFD[abi:cxx11](int, bool, unsigned long) #3 0x00007f1b16b9471d nix::DerivationGoal::runChild() #4 0x00007f1b16b96879 nix::childEntry(void*) #5 0x00007f1b16522d2f __clone [nix-shell:~]$ sudo eu-stack -p 26388 PID 26388 - process TID 26388: #0 0x00007f1b165fc097 accept #1 0x00000000004b56ac main_nix_daemon(int, char**) #2 0x000000000056dddb nix::mainWrapped(int, char**) #3 0x00007f1b16e81edd nix::handleExceptions(std::__cxx11::basic_string`fdinfo` statuses:
``` PID: 536 PID 536 FD 0: pos: 0 flags: 0100002 mnt_id: 3307 PID 536 FD 1: pos: 0 flags: 0100002 mnt_id: 376 PID 536 FD 10: pos: 495705 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY READ 26397 00:1c:2458720 0 EOF PID 536 FD 11: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:335005 0 EOF PID 536 FD 114: pos: 0 flags: 02100001 mnt_id: 365 PID 536 FD 118: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 6 PID 536 FD 119: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:5067831 0 EOF PID 536 FD 12: pos: 0 flags: 02000002 mnt_id: 9 scm_fds: 0 PID 536 FD 120: pos: 0 flags: 0100000 mnt_id: 3 PID 536 FD 124: pos: 0 flags: 02100001 mnt_id: 365 PID 536 FD 125: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 12 PID 536 FD 127: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:5068146 0 EOF PID 536 FD 128: pos: 0 flags: 0100000 mnt_id: 3 PID 536 FD 13: pos: 0 flags: 02100001 mnt_id: 365 PID 536 FD 133: pos: 0 flags: 02100001 mnt_id: 365 PID 536 FD 135: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 13 PID 536 FD 137: pos: 0 flags: 0100000 mnt_id: 3 PID 536 FD 14: pos: 0 flags: 02000002 mnt_id: 9 scm_fds: 0 PID 536 FD 141: pos: 0 flags: 0100000 mnt_id: 3 PID 536 FD 15: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 2 PID 536 FD 16: pos: 0 flags: 02100002 mnt_id: 366 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:2555099 0 EOF PID 536 FD 17: pos: 0 flags: 0100000 mnt_id: 3 PID 536 FD 18: pos: 0 flags: 02100002 mnt_id: 366 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:2555100 0 EOF PID 536 FD 19: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:335295 0 EOF PID 536 FD 2: pos: 0 flags: 0100002 mnt_id: 376 PID 536 FD 20: pos: 0 flags: 02000002 mnt_id: 9 scm_fds: 0 PID 536 FD 21: pos: 0 flags: 02100001 mnt_id: 365 PID 536 FD 22: pos: 0 flags: 02000002 mnt_id: 9 scm_fds: 0 PID 536 FD 23: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 3 PID 536 FD 24: pos: 0 flags: 02000002 mnt_id: 14 PID 536 FD 26: pos: 0 flags: 0100000 mnt_id: 3 PID 536 FD 27: pos: 0 flags: 02100002 mnt_id: 366 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:2729283 0 EOF PID 536 FD 28: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:337168 0 EOF PID 536 FD 29: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:337536 0 EOF PID 536 FD 3: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY READ 26397 00:1c:385 0 EOF PID 536 FD 30: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:338700 0 EOF PID 536 FD 306: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:335306 0 EOF PID 536 FD 307: pos: 0 flags: 02100001 mnt_id: 365 PID 536 FD 308: pos: 0 flags: 0100000 mnt_id: 3 PID 536 FD 309: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 4 PID 536 FD 31: pos: 0 flags: 0100000 mnt_id: 3 PID 536 FD 32: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:5067588 0 EOF PID 536 FD 33: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:5070745 0 EOF PID 536 FD 34: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:337921 0 EOF PID 536 FD 35: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:1877196 0 EOF PID 536 FD 36: pos: 0 flags: 02100001 mnt_id: 365 PID 536 FD 37: pos: 0 flags: 02100001 mnt_id: 365 PID 536 FD 38: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 7 PID 536 FD 39: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:337156 0 EOF PID 536 FD 4: pos: 0 flags: 02000002 mnt_id: 9 scm_fds: 0 PID 536 FD 40: pos: 0 flags: 0100002 mnt_id: 376 PID 536 FD 41: pos: 0 flags: 02100001 mnt_id: 365 PID 536 FD 42: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:5068994 0 EOF PID 536 FD 43: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 5 PID 536 FD 44: pos: 0 flags: 02100001 mnt_id: 365 PID 536 FD 45: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 16 PID 536 FD 46: pos: 0 flags: 0100000 mnt_id: 3 PID 536 FD 47: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 8 PID 536 FD 48: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:5069201 0 EOF PID 536 FD 49: pos: 0 flags: 02100002 mnt_id: 366 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:2461780 0 EOF PID 536 FD 5: pos: 0 flags: 02500002 mnt_id: 365 PID 536 FD 50: pos: 0 flags: 02100002 mnt_id: 366 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:2720603 0 EOF PID 536 FD 51: pos: 0 flags: 02100002 mnt_id: 366 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:2462330 0 EOF PID 536 FD 52: pos: 0 flags: 02000000 mnt_id: 13 PID 536 FD 53: pos: 0 flags: 02100001 mnt_id: 365 PID 536 FD 55: pos: 0 flags: 02100001 mnt_id: 365 PID 536 FD 56: pos: 0 flags: 02100001 mnt_id: 365 PID 536 FD 57: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 14 PID 536 FD 58: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 9 PID 536 FD 59: pos: 0 flags: 02100001 mnt_id: 365 PID 536 FD 6: pos: 0 flags: 02500002 mnt_id: 365 PID 536 FD 60: pos: 0 flags: 0100000 mnt_id: 3 PID 536 FD 61: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 11 PID 536 FD 62: pos: 0 flags: 0100002 mnt_id: 376 PID 536 FD 63: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 18 PID 536 FD 64: pos: 0 flags: 0100000 mnt_id: 3 PID 536 FD 65: pos: 0 flags: 02000000 mnt_id: 13 PID 536 FD 66: pos: 0 flags: 02000001 mnt_id: 13 PID 536 FD 69: pos: 0 flags: 0100000 mnt_id: 3 PID 536 FD 7: pos: 0 flags: 02500002 mnt_id: 365 PID 536 FD 70: pos: 0 flags: 0100000 mnt_id: 3 PID 536 FD 72: pos: 0 flags: 02100001 mnt_id: 365 PID 536 FD 74: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 17 PID 536 FD 8: pos: 0 flags: 02500002 mnt_id: 364 PID 536 FD 9: pos: 0 flags: 02100002 mnt_id: 366 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:2547754 0 EOF PID: 537 PID 537 FD 0: pos: 0 flags: 0100002 mnt_id: 3338 PID 537 FD 1: pos: 0 flags: 0100002 mnt_id: 376 PID 537 FD 10: pos: 495705 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY READ 26397 00:1c:2458720 0 EOF PID 537 FD 11: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:335005 0 EOF PID 537 FD 114: pos: 0 flags: 02100001 mnt_id: 365 PID 537 FD 118: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 6 PID 537 FD 119: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:5067831 0 EOF PID 537 FD 12: pos: 0 flags: 02000002 mnt_id: 9 scm_fds: 0 PID 537 FD 120: pos: 0 flags: 0100000 mnt_id: 3 PID 537 FD 124: pos: 0 flags: 02100001 mnt_id: 365 PID 537 FD 125: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 12 PID 537 FD 127: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:5068146 0 EOF PID 537 FD 128: pos: 0 flags: 0100000 mnt_id: 3 PID 537 FD 13: pos: 0 flags: 02100001 mnt_id: 365 PID 537 FD 133: pos: 0 flags: 02100001 mnt_id: 365 PID 537 FD 135: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 13 PID 537 FD 137: pos: 0 flags: 0100000 mnt_id: 3 PID 537 FD 14: pos: 0 flags: 02000002 mnt_id: 9 scm_fds: 0 PID 537 FD 141: pos: 0 flags: 0100000 mnt_id: 3 PID 537 FD 15: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 2 PID 537 FD 16: pos: 0 flags: 02100002 mnt_id: 366 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:2555099 0 EOF PID 537 FD 17: pos: 0 flags: 0100000 mnt_id: 3 PID 537 FD 18: pos: 0 flags: 02100002 mnt_id: 366 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:2555100 0 EOF PID 537 FD 19: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:335295 0 EOF PID 537 FD 2: pos: 0 flags: 0100002 mnt_id: 376 PID 537 FD 20: pos: 0 flags: 02000002 mnt_id: 9 scm_fds: 0 PID 537 FD 21: pos: 0 flags: 02100001 mnt_id: 365 PID 537 FD 22: pos: 0 flags: 02000002 mnt_id: 9 scm_fds: 0 PID 537 FD 23: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 3 PID 537 FD 24: pos: 0 flags: 02000002 mnt_id: 14 PID 537 FD 26: pos: 0 flags: 0100000 mnt_id: 3 PID 537 FD 27: pos: 0 flags: 02100002 mnt_id: 366 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:2729283 0 EOF PID 537 FD 28: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:337168 0 EOF PID 537 FD 29: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:337536 0 EOF PID 537 FD 3: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY READ 26397 00:1c:385 0 EOF PID 537 FD 30: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:338700 0 EOF PID 537 FD 306: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:335306 0 EOF PID 537 FD 307: pos: 0 flags: 02100001 mnt_id: 365 PID 537 FD 308: pos: 0 flags: 0100000 mnt_id: 3 PID 537 FD 309: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 4 PID 537 FD 31: pos: 0 flags: 0100000 mnt_id: 3 PID 537 FD 32: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:5067588 0 EOF PID 537 FD 33: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:5070745 0 EOF PID 537 FD 34: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:337921 0 EOF PID 537 FD 35: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:1877196 0 EOF PID 537 FD 36: pos: 0 flags: 02100001 mnt_id: 365 PID 537 FD 37: pos: 0 flags: 02100001 mnt_id: 365 PID 537 FD 38: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 7 PID 537 FD 39: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:337156 0 EOF PID 537 FD 4: pos: 0 flags: 02000002 mnt_id: 9 scm_fds: 0 PID 537 FD 40: pos: 0 flags: 0100002 mnt_id: 376 PID 537 FD 41: pos: 0 flags: 02100001 mnt_id: 365 PID 537 FD 42: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:5068994 0 EOF PID 537 FD 43: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 5 PID 537 FD 44: pos: 0 flags: 02100001 mnt_id: 365 PID 537 FD 45: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 16 PID 537 FD 46: pos: 0 flags: 0100000 mnt_id: 3 PID 537 FD 47: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 8 PID 537 FD 48: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:5069201 0 EOF PID 537 FD 49: pos: 0 flags: 02100002 mnt_id: 366 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:2461780 0 EOF PID 537 FD 5: pos: 0 flags: 02500002 mnt_id: 365 PID 537 FD 50: pos: 0 flags: 02100002 mnt_id: 366 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:2720603 0 EOF PID 537 FD 51: pos: 0 flags: 02100002 mnt_id: 366 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:2462330 0 EOF PID 537 FD 52: pos: 0 flags: 02000000 mnt_id: 13 PID 537 FD 53: pos: 0 flags: 02100001 mnt_id: 365 PID 537 FD 54: pos: 0 flags: 02000001 mnt_id: 13 PID 537 FD 55: pos: 0 flags: 02100001 mnt_id: 365 PID 537 FD 56: pos: 0 flags: 02100001 mnt_id: 365 PID 537 FD 57: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 14 PID 537 FD 58: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 9 PID 537 FD 59: pos: 0 flags: 02100001 mnt_id: 365 PID 537 FD 6: pos: 0 flags: 02500002 mnt_id: 365 PID 537 FD 60: pos: 0 flags: 0100000 mnt_id: 3 PID 537 FD 61: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 11 PID 537 FD 62: pos: 0 flags: 0100002 mnt_id: 376 PID 537 FD 63: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 18 PID 537 FD 64: pos: 0 flags: 0100000 mnt_id: 3 PID 537 FD 65: pos: 0 flags: 02000000 mnt_id: 13 PID 537 FD 69: pos: 0 flags: 0100000 mnt_id: 3 PID 537 FD 7: pos: 0 flags: 02500002 mnt_id: 365 PID 537 FD 70: pos: 0 flags: 0100000 mnt_id: 3 PID 537 FD 72: pos: 0 flags: 02100001 mnt_id: 365 PID 537 FD 74: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 17 PID 537 FD 8: pos: 0 flags: 02500002 mnt_id: 364 PID 537 FD 9: pos: 0 flags: 02100002 mnt_id: 366 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:2547754 0 EOF PID: 26388 PID 26388 FD 0: pos: 0 flags: 0100000 mnt_id: 21 PID 26388 FD 1: pos: 0 flags: 02 mnt_id: 9 scm_fds: 0 PID 26388 FD 2: pos: 0 flags: 02 mnt_id: 9 scm_fds: 0 PID 26388 FD 3: pos: 0 flags: 02000002 mnt_id: 9 scm_fds: 0 PID: 26397 PID 26397 FD 0: pos: 0 flags: 0100000 mnt_id: 21 PID 26397 FD 1: pos: 0 flags: 02 mnt_id: 9 scm_fds: 0 PID 26397 FD 10: pos: 495705 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY READ 26397 00:1c:2458720 0 EOF PID 26397 FD 11: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:335005 0 EOF PID 26397 FD 114: pos: 0 flags: 02100001 mnt_id: 365 PID 26397 FD 118: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 6 PID 26397 FD 119: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:5067831 0 EOF PID 26397 FD 12: pos: 0 flags: 02000002 mnt_id: 9 scm_fds: 0 PID 26397 FD 120: pos: 0 flags: 0100000 mnt_id: 3 PID 26397 FD 124: pos: 0 flags: 02100001 mnt_id: 365 PID 26397 FD 125: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 12 PID 26397 FD 127: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:5068146 0 EOF PID 26397 FD 128: pos: 0 flags: 0100000 mnt_id: 3 PID 26397 FD 13: pos: 0 flags: 02100001 mnt_id: 365 PID 26397 FD 133: pos: 0 flags: 02100001 mnt_id: 365 PID 26397 FD 135: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 13 PID 26397 FD 137: pos: 0 flags: 0100000 mnt_id: 3 PID 26397 FD 14: pos: 0 flags: 02000002 mnt_id: 9 scm_fds: 0 PID 26397 FD 141: pos: 0 flags: 0100000 mnt_id: 3 PID 26397 FD 15: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 2 PID 26397 FD 16: pos: 0 flags: 02100002 mnt_id: 366 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:2555099 0 EOF PID 26397 FD 17: pos: 0 flags: 0100000 mnt_id: 3 PID 26397 FD 18: pos: 0 flags: 02100002 mnt_id: 366 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:2555100 0 EOF PID 26397 FD 19: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:335295 0 EOF PID 26397 FD 2: pos: 0 flags: 02 mnt_id: 9 scm_fds: 0 PID 26397 FD 20: pos: 0 flags: 02000002 mnt_id: 9 scm_fds: 0 PID 26397 FD 21: pos: 0 flags: 02100001 mnt_id: 365 PID 26397 FD 22: pos: 0 flags: 02000002 mnt_id: 9 scm_fds: 0 PID 26397 FD 23: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 3 PID 26397 FD 26: pos: 0 flags: 0100000 mnt_id: 3 PID 26397 FD 27: pos: 0 flags: 02100002 mnt_id: 366 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:2729283 0 EOF PID 26397 FD 28: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:337168 0 EOF PID 26397 FD 29: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:337536 0 EOF PID 26397 FD 3: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY READ 26397 00:1c:385 0 EOF PID 26397 FD 30: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:338700 0 EOF PID 26397 FD 306: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:335306 0 EOF PID 26397 FD 307: pos: 0 flags: 02100001 mnt_id: 365 PID 26397 FD 308: pos: 0 flags: 0100000 mnt_id: 3 PID 26397 FD 309: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 4 PID 26397 FD 31: pos: 0 flags: 0100000 mnt_id: 3 PID 26397 FD 32: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:5067588 0 EOF PID 26397 FD 33: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:5070745 0 EOF PID 26397 FD 34: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:337921 0 EOF PID 26397 FD 35: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:1877196 0 EOF PID 26397 FD 36: pos: 0 flags: 02100001 mnt_id: 365 PID 26397 FD 37: pos: 0 flags: 02100001 mnt_id: 365 PID 26397 FD 38: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 7 PID 26397 FD 39: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:337156 0 EOF PID 26397 FD 4: pos: 0 flags: 02000002 mnt_id: 9 scm_fds: 0 PID 26397 FD 40: pos: 0 flags: 0100000 mnt_id: 3 PID 26397 FD 41: pos: 0 flags: 02100001 mnt_id: 365 PID 26397 FD 42: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:5068994 0 EOF PID 26397 FD 43: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 5 PID 26397 FD 44: pos: 0 flags: 02100001 mnt_id: 365 PID 26397 FD 45: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 16 PID 26397 FD 46: pos: 0 flags: 0100000 mnt_id: 3 PID 26397 FD 47: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 8 PID 26397 FD 48: pos: 0 flags: 02100002 mnt_id: 365 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:5069201 0 EOF PID 26397 FD 49: pos: 0 flags: 02100002 mnt_id: 366 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:2461780 0 EOF PID 26397 FD 5: pos: 0 flags: 02500002 mnt_id: 365 lock: 1: POSIX ADVISORY READ 26397 00:1c:386 1073741826 1073742335 PID 26397 FD 50: pos: 0 flags: 02100002 mnt_id: 366 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:2720603 0 EOF PID 26397 FD 51: pos: 0 flags: 02100002 mnt_id: 366 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:2462330 0 EOF PID 26397 FD 52: pos: 0 flags: 0100000 mnt_id: 3 PID 26397 FD 53: pos: 0 flags: 02100001 mnt_id: 365 PID 26397 FD 55: pos: 0 flags: 02100001 mnt_id: 365 PID 26397 FD 56: pos: 0 flags: 02100001 mnt_id: 365 PID 26397 FD 57: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 14 PID 26397 FD 58: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 9 PID 26397 FD 59: pos: 0 flags: 02100001 mnt_id: 365 PID 26397 FD 6: pos: 0 flags: 02500002 mnt_id: 365 PID 26397 FD 60: pos: 0 flags: 0100000 mnt_id: 3 PID 26397 FD 61: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 11 PID 26397 FD 63: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 18 PID 26397 FD 64: pos: 0 flags: 0100000 mnt_id: 3 PID 26397 FD 69: pos: 0 flags: 0100000 mnt_id: 3 PID 26397 FD 7: pos: 0 flags: 02500002 mnt_id: 365 lock: 1: POSIX ADVISORY READ 26397 00:1c:2941973 128 128 PID 26397 FD 70: pos: 0 flags: 0100000 mnt_id: 3 PID 26397 FD 72: pos: 0 flags: 02100001 mnt_id: 365 PID 26397 FD 74: pos: 0 flags: 0100002 mnt_id: 375 tty-index: 17 PID 26397 FD 8: pos: 0 flags: 02500002 mnt_id: 364 PID 26397 FD 9: pos: 0 flags: 02100002 mnt_id: 366 lock: 1: FLOCK ADVISORY WRITE 26397 00:1c:2547754 0 EOF ```