search

lxc / lxcfs

FUSE filesystem for LXC
https://linuxcontainers.org/lxcfs
Other
1.02k stars 246 forks source link

LXCFS crashes, need help diagnosing #644

Open webdock-io opened 1 month ago

webdock-io commented 1 month ago

Ubuntu Noble LXD v5.21.1 LTS + whatever version of lxcfs is packaged with this

We have a system with about 86 containers and no significant activity, a bit of load but nothing major. Out of our fleet of 60+ hosts which are pretty much identical to this system with some thousands of containers; on this host lxcfs crashes every couple of days. We have checked all logs we can think of, but all we can find is the crash dump itself in the syslog, as seen below.

Any help diagnosing these crashes is much appreciated, we suspect it's tied to some "unique" customer workload as it's only happening here. It's really rather painful having to reboot the entire host and bring down all instances with it, whenever this happens.

Crash data ``` 2024-06-05T14:07:40.930342+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: *** signal 11 2024-06-05T14:07:40.930748+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: Register dump: 2024-06-05T14:07:40.930813+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: RAX: 00000007d89ac102 RBX: ffffffffffffff80 RCX: 00007d8a0e27d769 2024-06-05T14:07:40.930860+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: RDX: 00007d89ed5ffc90 RSI: 00000007d89ac0f2 RDI: 00000007d89ac102 2024-06-05T14:07:40.930920+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: RBP: 00007d89ed5ffb20 R8 : 00007d8a0e262198 R9 : 000000000000000f 2024-06-05T14:07:40.930972+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: R10: 00007d8a0e262198 R11: 0000000000000000 R12: 0000000000000003 2024-06-05T14:07:40.931016+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: R13: 00007d8978107df0 R14: 00006437266e5708 R15: 0000000000000000 2024-06-05T14:07:40.931059+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: RSP: 00007d89ed5ffad0 2024-06-05T14:07:40.931119+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: RIP: 00007d8a0e3493fe EFLAGS: 00010202 2024-06-05T14:07:40.931163+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: CS: 0033 FS: 0000 GS: 0000 2024-06-05T14:07:40.931230+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: Trap: 0000000e Error: 00000004 OldMask: 00004007 CR2: d89ac0fa 2024-06-05T14:07:40.931282+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: FPUCW: 0000037f FPUSW: 00000000 TAG: 00000000 2024-06-05T14:07:40.931336+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: RIP: 00000000 RDP: 00000000 2024-06-05T14:07:40.931379+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: ST(0) 0000 0000000000000000 ST(1) 0000 0000000000000000 2024-06-05T14:07:40.931430+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: ST(2) 0000 0000000000000000 ST(3) 0000 0000000000000000 2024-06-05T14:07:40.931479+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: ST(4) 0000 0000000000000000 ST(5) 0000 0000000000000000 2024-06-05T14:07:40.931529+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: ST(6) 0000 0000000000000000 ST(7) 0000 0000000000000000 2024-06-05T14:07:40.931578+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: mxcsr: 1fa0 2024-06-05T14:07:40.931628+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: XMM0: 00000000000000000000000000000000 XMM1: 00000000000000000000000000000000 2024-06-05T14:07:40.931700+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: XMM2: 00000000000000000000000000000000 XMM3: 00000000000000000000000000000000 2024-06-05T14:07:40.931743+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: XMM4: 00000000000000000000000000000000 XMM5: 00000000000000000000000000000000 2024-06-05T14:07:40.931794+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: XMM6: 00000000000000000000000000000000 XMM7: 00000000000000000000000000000000 2024-06-05T14:07:40.931958+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: XMM8: 00000000000000000000000000000000 XMM9: 00000000000000000000000000000000 2024-06-05T14:07:40.932017+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: XMM10: 00000000000000000000000000000000 XMM11: 00000000000000000000000000000000 2024-06-05T14:07:40.932066+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: XMM12: 00000000000000000000000000000000 XMM13: 00000000000000000000000000000000 2024-06-05T14:07:40.932115+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: XMM14: 00000000000000000000000000000000 XMM15: 00000000000000000000000000000000 2024-06-05T14:07:40.932164+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: Backtrace: 2024-06-05T14:07:40.932228+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: /lib/x86_64-linux-gnu/libc.so.6(free+0x1e)[0x7d8a0e3493fe] 2024-06-05T14:07:40.932285+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: /snap/lxd/current/lib/liblxcfs.so(do_release_file_info+0x42)[0x7d8a0e2880c8] 2024-06-05T14:07:40.932335+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: /snap/lxd/current/lib/liblxcfs.so(proc_release+0x20)[0x7d8a0e27d789] 2024-06-05T14:07:40.932385+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: lxcfs(+0x2f2c)[0x64372561cf2c] 2024-06-05T14:07:40.932434+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: lxcfs(+0x3c7f)[0x64372561dc7f] 2024-06-05T14:07:40.932708+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: /snap/lxd/current/lib/x86_64-linux-gnu/libfuse3.so.3(+0xbb6a)[0x7d8a0e4fbb6a] 2024-06-05T14:07:40.932765+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: /snap/lxd/current/lib/x86_64-linux-gnu/libfuse3.so.3(+0x104fa)[0x7d8a0e5004fa] 2024-06-05T14:07:40.932801+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: /snap/lxd/current/lib/x86_64-linux-gnu/libfuse3.so.3(+0x11738)[0x7d8a0e501738] 2024-06-05T14:07:40.932892+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: /snap/lxd/current/lib/x86_64-linux-gnu/libfuse3.so.3(+0x1e13f)[0x7d8a0e50e13f] 2024-06-05T14:07:40.932957+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: /snap/lxd/current/lib/x86_64-linux-gnu/libfuse3.so.3(+0x167a7)[0x7d8a0e5067a7] 2024-06-05T14:07:40.933000+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7d8a0e338ac3] 2024-06-05T14:07:40.933043+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: /lib/x86_64-linux-gnu/libc.so.6(+0x126850)[0x7d8a0e3ca850] 2024-06-05T14:07:40.933091+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: Memory map: 2024-06-05T14:07:40.933142+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 64372561a000-643725622000 r-xp 00000000 07:01 44 /snap/lxd/28463/bin/lxcfs 2024-06-05T14:07:40.933216+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 643725622000-643725623000 r--p 00007000 07:01 44 /snap/lxd/28463/bin/lxcfs 2024-06-05T14:07:40.933275+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 643725623000-643725624000 rw-p 00008000 07:01 44 /snap/lxd/28463/bin/lxcfs 2024-06-05T14:07:40.933325+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 6437266d8000-643726727000 rw-p 00000000 00:00 0 [heap] 2024-06-05T14:07:40.933361+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d88f8000000-7d88f8203000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.933404+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d88f8203000-7d88fc000000 ---p 00000000 00:00 0 2024-06-05T14:07:40.933447+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8900000000-7d8900204000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.933502+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8900204000-7d8904000000 ---p 00000000 00:00 0 2024-06-05T14:07:40.933551+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8904000000-7d8904203000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.933600+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8904203000-7d8908000000 ---p 00000000 00:00 0 2024-06-05T14:07:40.933705+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d890c000000-7d890c203000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.933765+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d890c203000-7d8910000000 ---p 00000000 00:00 0 2024-06-05T14:07:40.933817+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8910000000-7d8910213000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.933859+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8910213000-7d8914000000 ---p 00000000 00:00 0 2024-06-05T14:07:40.933908+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8918000000-7d8918206000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.933957+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8918206000-7d891c000000 ---p 00000000 00:00 0 2024-06-05T14:07:40.934015+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d891c000000-7d891c21d000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.934064+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d891c21d000-7d8920000000 ---p 00000000 00:00 0 2024-06-05T14:07:40.934121+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8924000000-7d8924177000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.934169+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8924177000-7d8928000000 ---p 00000000 00:00 0 2024-06-05T14:07:40.934245+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8928000000-7d8928172000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.934294+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8928172000-7d892c000000 ---p 00000000 00:00 0 2024-06-05T14:07:40.934351+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8930000000-7d8930203000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.934400+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8930203000-7d8934000000 ---p 00000000 00:00 0 2024-06-05T14:07:40.934449+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8934000000-7d8934165000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.934507+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8934165000-7d8938000000 ---p 00000000 00:00 0 2024-06-05T14:07:40.934557+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d893c000000-7d893c17b000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.934598+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d893c17b000-7d8940000000 ---p 00000000 00:00 0 2024-06-05T14:07:40.934674+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8940000000-7d894016d000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.934732+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d894016d000-7d8944000000 ---p 00000000 00:00 0 2024-06-05T14:07:40.934789+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8948000000-7d8948207000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.934838+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8948207000-7d894c000000 ---p 00000000 00:00 0 2024-06-05T14:07:40.934950+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d894c000000-7d894c203000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.935009+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d894c203000-7d8950000000 ---p 00000000 00:00 0 2024-06-05T14:07:40.935723+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8954000000-7d8954203000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.935778+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8954203000-7d8958000000 ---p 00000000 00:00 0 2024-06-05T14:07:40.935824+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8958000000-7d895820d000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.935875+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d895820d000-7d895c000000 ---p 00000000 00:00 0 2024-06-05T14:07:40.935921+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8960000000-7d8960178000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.935971+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8960178000-7d8964000000 ---p 00000000 00:00 0 2024-06-05T14:07:40.936030+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8964000000-7d8964203000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.936087+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8964203000-7d8968000000 ---p 00000000 00:00 0 2024-06-05T14:07:40.936133+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d896c000000-7d896c21a000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.936178+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d896c21a000-7d8970000000 ---p 00000000 00:00 0 2024-06-05T14:07:40.936247+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8970000000-7d8970204000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.936293+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8970204000-7d8974000000 ---p 00000000 00:00 0 2024-06-05T14:07:40.936343+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8978000000-7d8978203000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.936395+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8978203000-7d897c000000 ---p 00000000 00:00 0 2024-06-05T14:07:40.936446+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d897c000000-7d897c203000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.936497+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d897c203000-7d8980000000 ---p 00000000 00:00 0 2024-06-05T14:07:40.936547+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8984000000-7d898417e000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.936597+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d898417e000-7d8988000000 ---p 00000000 00:00 0 2024-06-05T14:07:40.936660+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8988000000-7d8988203000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.936712+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8988203000-7d898c000000 ---p 00000000 00:00 0 2024-06-05T14:07:40.936763+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8990000000-7d8990203000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.936818+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8990203000-7d8994000000 ---p 00000000 00:00 0 2024-06-05T14:07:40.936875+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8994000000-7d8994203000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.936930+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8994203000-7d8998000000 ---p 00000000 00:00 0 2024-06-05T14:07:40.936975+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d899c000000-7d899c204000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.937025+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d899c204000-7d89a0000000 ---p 00000000 00:00 0 2024-06-05T14:07:40.937070+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d89a0000000-7d89a0172000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.937115+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d89a0172000-7d89a4000000 ---p 00000000 00:00 0 2024-06-05T14:07:40.937165+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d89a8000000-7d89a8205000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.937226+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d89a8205000-7d89ac000000 ---p 00000000 00:00 0 2024-06-05T14:07:40.937290+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d89ac000000-7d89ac17b000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.937341+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d89ac17b000-7d89b0000000 ---p 00000000 00:00 0 2024-06-05T14:07:40.937397+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d89b4000000-7d89b416f000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.937448+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d89b416f000-7d89b8000000 ---p 00000000 00:00 0 2024-06-05T14:07:40.937504+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d89b8000000-7d89b8173000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.937554+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d89b8173000-7d89bc000000 ---p 00000000 00:00 0 2024-06-05T14:07:40.937610+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d89c0000000-7d89c0178000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.937983+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d89c0178000-7d89c4000000 ---p 00000000 00:00 0 2024-06-05T14:07:40.938046+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d89c4000000-7d89c417a000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.938134+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d89c417a000-7d89c8000000 ---p 00000000 00:00 0 2024-06-05T14:07:40.938191+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d89cc000000-7d89cc204000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.938255+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d89cc204000-7d89d0000000 ---p 00000000 00:00 0 2024-06-05T14:07:40.938306+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d89d0000000-7d89d0181000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.938363+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d89d0181000-7d89d4000000 ---p 00000000 00:00 0 2024-06-05T14:07:40.938409+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d89d8000000-7d89d8170000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.938459+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d89d8170000-7d89dc000000 ---p 00000000 00:00 0 2024-06-05T14:07:40.938504+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d89dc000000-7d89dc173000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.938556+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d89dc173000-7d89e0000000 ---p 00000000 00:00 0 2024-06-05T14:07:40.938602+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d89e2c00000-7d89e2c01000 ---p 00000000 00:00 0 2024-06-05T14:07:40.938670+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d89e2c01000-7d89e3401000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.938707+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d89e4000000-7d89e4178000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.938742+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d89e4178000-7d89e8000000 ---p 00000000 00:00 0 2024-06-05T14:07:40.938784+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d89e8000000-7d89e8173000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.938813+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d89e8173000-7d89ec000000 ---p 00000000 00:00 0 2024-06-05T14:07:40.938848+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d89ece00000-7d89ece01000 ---p 00000000 00:00 0 2024-06-05T14:07:40.938882+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d89ece01000-7d89ed601000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.938916+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d89ee200000-7d89ee201000 ---p 00000000 00:00 0 2024-06-05T14:07:40.938950+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d89ee201000-7d89eea01000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.938984+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d89eec00000-7d89eec01000 ---p 00000000 00:00 0 2024-06-05T14:07:40.939019+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d89eec01000-7d89ef401000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.939052+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d89ef600000-7d89ef601000 ---p 00000000 00:00 0 2024-06-05T14:07:40.939086+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d89ef601000-7d89efe01000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.939120+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d89f0000000-7d89f016b000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.939148+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d89f016b000-7d89f4000000 ---p 00000000 00:00 0 2024-06-05T14:07:40.939182+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d89f4000000-7d89f4177000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.939231+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d89f4177000-7d89f8000000 ---p 00000000 00:00 0 2024-06-05T14:07:40.939266+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d89f8e00000-7d89f8e01000 ---p 00000000 00:00 0 2024-06-05T14:07:40.939308+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d89f8e01000-7d89f9601000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.939346+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d89fa200000-7d89fa201000 ---p 00000000 00:00 0 2024-06-05T14:07:40.939387+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d89fa201000-7d89faa01000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.939421+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d89fc000000-7d89fc16c000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.939455+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d89fc16c000-7d8a00000000 ---p 00000000 00:00 0 2024-06-05T14:07:40.939490+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a00000000-7d8a00175000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.939533+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a00175000-7d8a04000000 ---p 00000000 00:00 0 2024-06-05T14:07:40.939567+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a04400000-7d8a04401000 ---p 00000000 00:00 0 2024-06-05T14:07:40.939596+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a04401000-7d8a04c01000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.939630+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a04e00000-7d8a04e01000 ---p 00000000 00:00 0 2024-06-05T14:07:40.939680+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a04e01000-7d8a05601000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.939716+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a05800000-7d8a05801000 ---p 00000000 00:00 0 2024-06-05T14:07:40.939762+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a05801000-7d8a06001000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.939824+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a06c00000-7d8a06c01000 ---p 00000000 00:00 0 2024-06-05T14:07:40.939867+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a06c01000-7d8a07401000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.939901+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a07600000-7d8a07601000 ---p 00000000 00:00 0 2024-06-05T14:07:40.939935+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a07601000-7d8a07e01000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.939975+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a08000000-7d8a08067000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.940029+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a08067000-7d8a0c000000 ---p 00000000 00:00 0 2024-06-05T14:07:40.940074+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a0c600000-7d8a0c601000 ---p 00000000 00:00 0 2024-06-05T14:07:40.940115+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a0c601000-7d8a0ce01000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.940149+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a0d000000-7d8a0d001000 ---p 00000000 00:00 0 2024-06-05T14:07:40.940185+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a0d001000-7d8a0d801000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.940236+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a0d9e3000-7d8a0da00000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.940265+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a0da00000-7d8a0da01000 ---p 00000000 00:00 0 2024-06-05T14:07:40.940306+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a0da01000-7d8a0e201000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.940340+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a0e201000-7d8a0e261000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.940376+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a0e261000-7d8a0e294000 r-xp 00000000 07:01 155 /snap/lxd/28463/lib/liblxcfs.so 2024-06-05T14:07:40.940414+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a0e294000-7d8a0e295000 r--p 00032000 07:01 155 /snap/lxd/28463/lib/liblxcfs.so 2024-06-05T14:07:40.940445+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a0e295000-7d8a0e296000 rw-p 00033000 07:01 155 /snap/lxd/28463/lib/liblxcfs.so 2024-06-05T14:07:40.940486+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a0e296000-7d8a0e2a4000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.940530+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a0e2a4000-7d8a0e2cc000 r--p 00000000 07:00 7471 /usr/lib/x86_64-linux-gnu/libc.so.6 2024-06-05T14:07:40.940566+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a0e2cc000-7d8a0e461000 r-xp 00028000 07:00 7471 /usr/lib/x86_64-linux-gnu/libc.so.6 2024-06-05T14:07:40.940601+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a0e461000-7d8a0e4b9000 r--p 001bd000 07:00 7471 /usr/lib/x86_64-linux-gnu/libc.so.6 2024-06-05T14:07:40.940635+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a0e4b9000-7d8a0e4ba000 ---p 00215000 07:00 7471 /usr/lib/x86_64-linux-gnu/libc.so.6 2024-06-05T14:07:40.940690+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a0e4ba000-7d8a0e4be000 r--p 00215000 07:00 7471 /usr/lib/x86_64-linux-gnu/libc.so.6 2024-06-05T14:07:40.940726+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a0e4be000-7d8a0e4c0000 rw-p 00219000 07:00 7471 /usr/lib/x86_64-linux-gnu/libc.so.6 2024-06-05T14:07:40.940759+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a0e4c0000-7d8a0e4cd000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.940800+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a0e4cd000-7d8a0e4d0000 r--p 00000000 07:00 7521 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 2024-06-05T14:07:40.940831+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a0e4d0000-7d8a0e4e7000 r-xp 00003000 07:00 7521 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 2024-06-05T14:07:40.940891+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a0e4e7000-7d8a0e4eb000 r--p 0001a000 07:00 7521 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 2024-06-05T14:07:40.940956+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a0e4eb000-7d8a0e4ec000 r--p 0001d000 07:00 7521 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 2024-06-05T14:07:40.940985+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a0e4ec000-7d8a0e4ed000 rw-p 0001e000 07:00 7521 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 2024-06-05T14:07:40.941020+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a0e4ed000-7d8a0e4f0000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.941055+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a0e4f0000-7d8a0e4f7000 r--p 00000000 07:01 565 /snap/lxd/28463/lib/x86_64-linux-gnu/libfuse3.so.3.10.5 2024-06-05T14:07:40.941090+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a0e4f7000-7d8a0e512000 r-xp 00007000 07:01 565 /snap/lxd/28463/lib/x86_64-linux-gnu/libfuse3.so.3.10.5 2024-06-05T14:07:40.941125+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a0e512000-7d8a0e51c000 r--p 00022000 07:01 565 /snap/lxd/28463/lib/x86_64-linux-gnu/libfuse3.so.3.10.5 2024-06-05T14:07:40.941349+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a0e51c000-7d8a0e52e000 r--p 0002b000 07:01 565 /snap/lxd/28463/lib/x86_64-linux-gnu/libfuse3.so.3.10.5 2024-06-05T14:07:40.941386+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a0e52e000-7d8a0e52f000 rw-p 0003d000 07:01 565 /snap/lxd/28463/lib/x86_64-linux-gnu/libfuse3.so.3.10.5 2024-06-05T14:07:40.941424+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a0e52f000-7d8a0e530000 r--p 00000000 07:01 538 /snap/lxd/28463/lib/x86_64-linux-gnu/libSegFault.so 2024-06-05T14:07:40.941469+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a0e530000-7d8a0e533000 r-xp 00001000 07:01 538 /snap/lxd/28463/lib/x86_64-linux-gnu/libSegFault.so 2024-06-05T14:07:40.941506+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a0e533000-7d8a0e534000 r--p 00004000 07:01 538 /snap/lxd/28463/lib/x86_64-linux-gnu/libSegFault.so 2024-06-05T14:07:40.941546+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a0e534000-7d8a0e535000 r--p 00005000 07:01 538 /snap/lxd/28463/lib/x86_64-linux-gnu/libSegFault.so 2024-06-05T14:07:40.941581+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a0e535000-7d8a0e536000 rw-p 00006000 07:01 538 /snap/lxd/28463/lib/x86_64-linux-gnu/libSegFault.so 2024-06-05T14:07:40.941626+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a0e536000-7d8a0e538000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.941687+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a0e538000-7d8a0e53a000 r--p 00000000 07:00 7442 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 2024-06-05T14:07:40.941724+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a0e53a000-7d8a0e564000 r-xp 00002000 07:00 7442 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 2024-06-05T14:07:40.941780+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a0e564000-7d8a0e56f000 r--p 0002c000 07:00 7442 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 2024-06-05T14:07:40.941817+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a0e56f000-7d8a0e570000 rw-p 00000000 00:00 0 2024-06-05T14:07:40.941852+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a0e570000-7d8a0e572000 r--p 00037000 07:00 7442 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 2024-06-05T14:07:40.941882+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7d8a0e572000-7d8a0e574000 rw-p 00039000 07:00 7442 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 2024-06-05T14:07:40.941975+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7ffd64ec8000-7ffd64ee9000 rw-p 00000000 00:00 0 [stack] 2024-06-05T14:07:40.942019+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7ffd64f4d000-7ffd64f51000 r--p 00000000 00:00 0 [vvar] 2024-06-05T14:07:40.942051+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: 7ffd64f51000-7ffd64f53000 r-xp 00000000 00:00 0 [vdso] 2024-06-05T14:07:40.942081+02:00 lxdvm2-x14-u17-r1 lxd.daemon[5415]: ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0 [vsyscall] ```
webdock-io commented 4 weeks ago

@mihalicyn Since you self-assigned I'm hoping this is something which can be looked at and resolved, as we are now seeing this happen on 2 additional systems. What the systems have in common:

  1. They do show some load, but nothing crazy compared to what we've been running in the past without issue
  2. It should be noted this is inside a VM
  3. LXCFS is running with --enable-loadavg

Here is a dump from another machine where this happened, not sure if you can tell if this is an identical crash or different. If it's different, let me know and I'll start collecting crash dumps from the various instances so you can get a complete picture:

Crash data ``` 2024-06-06T23:02:00.051688+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: *** signal 11 2024-06-06T23:02:00.051842+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: Register dump: 2024-06-06T23:02:00.051886+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: RAX: 0000000000008000 RBX: 0000585f1f1ad8c0 RCX: 0000000000000000 2024-06-06T23:02:00.052033+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: RDX: 0000000000004000 RSI: 00007c3774104d30 RDI: 00007c377410e430 2024-06-06T23:02:00.052079+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: RBP: 00007c38241ffae0 R8 : 00007c38241ffc90 R9 : 00007c3824d052fb 2024-06-06T23:02:00.052298+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: R10: 00007c3824ce42a0 R11: 0000000000000000 R12: 00007c377410e430 2024-06-06T23:02:00.052588+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: R13: 00007c38241ffc90 R14: 00007c38241ffbf8 R15: 0000000000000000 2024-06-06T23:02:00.052649+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: RSP: 00007c38241ffaa0 2024-06-06T23:02:00.052669+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: RIP: 00007c3824d0532b EFLAGS: 00010206 2024-06-06T23:02:00.052696+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: CS: 0033 FS: 0000 GS: 0000 2024-06-06T23:02:00.052756+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: Trap: 0000000e Error: 00000004 OldMask: 00004007 CR2: 00008018 2024-06-06T23:02:00.052779+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: FPUCW: 0000037f FPUSW: 00000000 TAG: 00000000 2024-06-06T23:02:00.052802+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: RIP: 00000000 RDP: 00000000 2024-06-06T23:02:00.052828+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: ST(0) 0000 0000000000000000 ST(1) 0000 0000000000000000 2024-06-06T23:02:00.052854+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: ST(2) 0000 0000000000000000 ST(3) 0000 0000000000000000 2024-06-06T23:02:00.052880+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: ST(4) 0000 0000000000000000 ST(5) 0000 0000000000000000 2024-06-06T23:02:00.052906+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: ST(6) 0000 0000000000000000 ST(7) 0000 0000000000000000 2024-06-06T23:02:00.052948+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: mxcsr: 1fa0 2024-06-06T23:02:00.052972+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: XMM0: 00000000000000000000000000000000 XMM1: 00000000000000000000000000000000 2024-06-06T23:02:00.052996+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: XMM2: 00000000000000000000000000000000 XMM3: 00000000000000000000000000000000 2024-06-06T23:02:00.053028+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: XMM4: 00000000000000000000000000000000 XMM5: 00000000000000000000000000000000 2024-06-06T23:02:00.053062+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: XMM6: 00000000000000000000000000000000 XMM7: 00000000000000000000000000000000 2024-06-06T23:02:00.053092+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: XMM8: 00000000000000000000000000000000 XMM9: 00000000000000000000000000000000 2024-06-06T23:02:00.053115+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: XMM10: 00000000000000000000000000000000 XMM11: 00000000000000000000000000000000 2024-06-06T23:02:00.053136+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: XMM12: 00000000000000000000000000000000 XMM13: 00000000000000000000000000000000 2024-06-06T23:02:00.053158+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: XMM14: 00000000000000000000000000000000 XMM15: 00000000000000000000000000000000 2024-06-06T23:02:00.053631+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: Backtrace: 2024-06-06T23:02:00.053715+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: /snap/lxd/current/lib/liblxcfs.so(proc_read+0x30)[0x7c3824d0532b] 2024-06-06T23:02:00.053743+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: lxcfs(+0x22c3)[0x585f1e32f2c3] 2024-06-06T23:02:00.053766+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: lxcfs(+0x3a56)[0x585f1e330a56] 2024-06-06T23:02:00.053792+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: /snap/lxd/current/lib/x86_64-linux-gnu/libfuse3.so.3(fuse_fs_read_buf+0x1be)[0x7c3824f7ff2e] 2024-06-06T23:02:00.053823+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: /snap/lxd/current/lib/x86_64-linux-gnu/libfuse3.so.3(+0x10d66)[0x7c3824f82d66] 2024-06-06T23:02:00.053854+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: /snap/lxd/current/lib/x86_64-linux-gnu/libfuse3.so.3(+0x114fa)[0x7c3824f834fa] 2024-06-06T23:02:00.053877+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: /snap/lxd/current/lib/x86_64-linux-gnu/libfuse3.so.3(+0x1e13f)[0x7c3824f9013f] 2024-06-06T23:02:00.053903+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: /snap/lxd/current/lib/x86_64-linux-gnu/libfuse3.so.3(+0x167a7)[0x7c3824f887a7] 2024-06-06T23:02:00.053936+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7c3824dbaac3] 2024-06-06T23:02:00.053962+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: /lib/x86_64-linux-gnu/libc.so.6(+0x126850)[0x7c3824e4c850] 2024-06-06T23:02:00.053983+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: Memory map: 2024-06-06T23:02:00.054010+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 585f1e32d000-585f1e335000 r-xp 00000000 07:02 44 /snap/lxd/28463/bin/lxcfs 2024-06-06T23:02:00.054052+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 585f1e335000-585f1e336000 r--p 00007000 07:02 44 /snap/lxd/28463/bin/lxcfs 2024-06-06T23:02:00.054078+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 585f1e336000-585f1e337000 rw-p 00008000 07:02 44 /snap/lxd/28463/bin/lxcfs 2024-06-06T23:02:00.054104+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 585f1f1a0000-585f1f1ef000 rw-p 00000000 00:00 0 [heap] 2024-06-06T23:02:00.054130+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c36e4000000-7c36e4207000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.054157+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c36e4207000-7c36e8000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.054182+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c36e8000000-7c36e8203000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.054208+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c36e8203000-7c36ec000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.054237+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c36f0000000-7c36f0203000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.054262+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c36f0203000-7c36f4000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.054288+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c36f4000000-7c36f4204000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.054314+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c36f4204000-7c36f8000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.054341+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c36fc000000-7c36fc208000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.054381+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c36fc208000-7c3700000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.054430+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3700000000-7c370020d000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.054457+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c370020d000-7c3704000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.054483+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3708000000-7c3708203000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.054508+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3708203000-7c370c000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.054534+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c370c000000-7c370c220000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.054559+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c370c220000-7c3710000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.054585+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3714000000-7c3714210000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.054610+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3714210000-7c3718000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.054635+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3718000000-7c3718208000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.054660+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3718208000-7c371c000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.054703+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3720000000-7c372021e000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.054733+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c372021e000-7c3724000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.055163+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3724000000-7c3724217000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.055210+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3724217000-7c3728000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.055321+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c372c000000-7c372c20c000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.055391+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c372c20c000-7c3730000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.056914+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3730000000-7c3730221000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.057246+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3730221000-7c3734000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.057276+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3738000000-7c373816a000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.057299+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c373816a000-7c373c000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.057322+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c373c000000-7c373c20d000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.057344+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c373c20d000-7c3740000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.057370+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3744000000-7c3744211000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.057392+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3744211000-7c3748000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.057447+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3748000000-7c374820f000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.057473+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c374820f000-7c374c000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.057499+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3750000000-7c3750211000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.057525+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3750211000-7c3754000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.057551+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3754000000-7c3754179000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.057572+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3754179000-7c3758000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.057598+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c375c000000-7c375c206000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.057623+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c375c206000-7c3760000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.057653+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3760000000-7c3760209000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.057678+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3760209000-7c3764000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.057703+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3768000000-7c376820b000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.057734+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c376820b000-7c376c000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.057760+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c376c000000-7c376c178000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.057785+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c376c178000-7c3770000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.057816+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3774000000-7c3774174000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.057842+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3774174000-7c3778000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.057867+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3778000000-7c377820f000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.057892+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c377820f000-7c377c000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.057917+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3780000000-7c378016c000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.057956+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c378016c000-7c3784000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.057982+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3784000000-7c3784211000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.058004+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3784211000-7c3788000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.058061+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c378c000000-7c378c20f000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.058092+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c378c20f000-7c3790000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.058114+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3790000000-7c3790184000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.058136+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3790184000-7c3794000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.058161+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3798000000-7c37981a3000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.058217+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c37981a3000-7c379c000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.058234+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c379c000000-7c379c212000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.058256+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c379c212000-7c37a0000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.058278+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c37a4000000-7c37a418c000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.058300+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c37a418c000-7c37a8000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.058317+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c37a8000000-7c37a8227000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.058350+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c37a8227000-7c37ac000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.058372+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c37b0000000-7c37b01a1000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.058393+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c37b01a1000-7c37b4000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.058439+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c37b4000000-7c37b418a000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.058533+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c37b418a000-7c37b8000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.058558+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c37bc000000-7c37bc198000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.058575+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c37bc198000-7c37c0000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.058597+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c37c0000000-7c37c018c000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.058615+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c37c018c000-7c37c4000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.058637+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c37c8000000-7c37c8222000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.058663+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c37c8222000-7c37cc000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.058685+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c37cc000000-7c37cc20e000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.058706+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c37cc20e000-7c37d0000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.058729+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c37d4000000-7c37d4195000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.058809+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c37d4195000-7c37d8000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.058836+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c37d8000000-7c37d818a000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.058861+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c37d818a000-7c37dc000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.058887+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c37e0000000-7c37e0172000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.058909+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c37e0172000-7c37e4000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.058940+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c37e4000000-7c37e4194000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.058967+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c37e4194000-7c37e8000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.058990+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c37ec000000-7c37ec185000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.059011+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c37ec185000-7c37f0000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.059112+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c37f0000000-7c37f01a8000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.059137+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c37f01a8000-7c37f4000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.059159+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c37f6200000-7c37f6201000 ---p 00000000 00:00 0 2024-06-06T23:02:00.059184+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c37f6201000-7c37f6a01000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.059210+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c37f6c00000-7c37f6c01000 ---p 00000000 00:00 0 2024-06-06T23:02:00.059232+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c37f6c01000-7c37f7401000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.059254+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c37f7600000-7c37f7601000 ---p 00000000 00:00 0 2024-06-06T23:02:00.059276+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c37f7601000-7c37f7e01000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.059297+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c37f8000000-7c37f8191000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.059319+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c37f8191000-7c37fc000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.059340+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c37fc000000-7c37fc18a000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.059366+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c37fc18a000-7c3800000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.059505+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3800400000-7c3800401000 ---p 00000000 00:00 0 2024-06-06T23:02:00.059537+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3800401000-7c3800c01000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.059564+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3800e00000-7c3800e01000 ---p 00000000 00:00 0 2024-06-06T23:02:00.059585+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3800e01000-7c3801601000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.059603+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3801800000-7c3801801000 ---p 00000000 00:00 0 2024-06-06T23:02:00.059624+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3801801000-7c3802001000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.059645+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3802200000-7c3802201000 ---p 00000000 00:00 0 2024-06-06T23:02:00.059667+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3802201000-7c3802a01000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.059688+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3802c00000-7c3802c01000 ---p 00000000 00:00 0 2024-06-06T23:02:00.059709+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3802c01000-7c3803401000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.059736+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3803600000-7c3803601000 ---p 00000000 00:00 0 2024-06-06T23:02:00.059757+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3803601000-7c3803e01000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.059775+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3804000000-7c3804190000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.059801+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3804190000-7c3808000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.059824+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3808000000-7c3808060000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.059846+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3808060000-7c380c000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.059868+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c380c400000-7c380c401000 ---p 00000000 00:00 0 2024-06-06T23:02:00.059890+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c380c401000-7c380cc01000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.059908+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c380ce00000-7c380ce01000 ---p 00000000 00:00 0 2024-06-06T23:02:00.059927+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c380ce01000-7c380d601000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.059946+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c380d800000-7c380d801000 ---p 00000000 00:00 0 2024-06-06T23:02:00.059968+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c380d801000-7c380e001000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.059993+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c380e200000-7c380e201000 ---p 00000000 00:00 0 2024-06-06T23:02:00.060023+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c380e201000-7c380ea01000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.060046+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c380ec00000-7c380ec01000 ---p 00000000 00:00 0 2024-06-06T23:02:00.060067+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c380ec01000-7c380f401000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.060163+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c380f600000-7c380f601000 ---p 00000000 00:00 0 2024-06-06T23:02:00.060189+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c380f601000-7c380fe01000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.060210+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3810000000-7c381018d000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.060228+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c381018d000-7c3814000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.060249+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3814000000-7c381418d000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.060271+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c381418d000-7c3818000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.060293+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3818400000-7c3818401000 ---p 00000000 00:00 0 2024-06-06T23:02:00.060318+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3818401000-7c3818c01000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.060340+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3818e00000-7c3818e01000 ---p 00000000 00:00 0 2024-06-06T23:02:00.060363+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3818e01000-7c3819601000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.061182+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3819800000-7c3819801000 ---p 00000000 00:00 0 2024-06-06T23:02:00.061218+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3819801000-7c381a001000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.061245+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c381a200000-7c381a201000 ---p 00000000 00:00 0 2024-06-06T23:02:00.061268+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c381a201000-7c381aa01000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.061290+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c381ac00000-7c381ac01000 ---p 00000000 00:00 0 2024-06-06T23:02:00.061350+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c381ac01000-7c381b401000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.061373+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c381b600000-7c381b601000 ---p 00000000 00:00 0 2024-06-06T23:02:00.061396+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c381b601000-7c381be01000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.061436+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c381c000000-7c381c182000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.061466+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c381c182000-7c3820000000 ---p 00000000 00:00 0 2024-06-06T23:02:00.061488+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3820800000-7c3820801000 ---p 00000000 00:00 0 2024-06-06T23:02:00.061510+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3820801000-7c3821001000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.061541+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3821200000-7c3821201000 ---p 00000000 00:00 0 2024-06-06T23:02:00.061567+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3821201000-7c3821a01000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.061586+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3821c00000-7c3821c01000 ---p 00000000 00:00 0 2024-06-06T23:02:00.061607+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3821c01000-7c3822401000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.061629+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3822600000-7c3822601000 ---p 00000000 00:00 0 2024-06-06T23:02:00.061651+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3822601000-7c3822e01000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.061672+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3823000000-7c3823001000 ---p 00000000 00:00 0 2024-06-06T23:02:00.061693+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3823001000-7c3823801000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.061715+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3823a00000-7c3823a01000 ---p 00000000 00:00 0 2024-06-06T23:02:00.061737+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3823a01000-7c3824201000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.061759+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3824400000-7c3824401000 ---p 00000000 00:00 0 2024-06-06T23:02:00.061784+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3824401000-7c3824c01000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.061806+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3824c66000-7c3824ce3000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.061828+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3824ce3000-7c3824d16000 r-xp 00000000 07:02 155 /snap/lxd/28463/lib/liblxcfs.so 2024-06-06T23:02:00.061848+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3824d16000-7c3824d17000 r--p 00032000 07:02 155 /snap/lxd/28463/lib/liblxcfs.so 2024-06-06T23:02:00.061870+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3824d17000-7c3824d18000 rw-p 00033000 07:02 155 /snap/lxd/28463/lib/liblxcfs.so 2024-06-06T23:02:00.061892+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3824d18000-7c3824d26000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.061913+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3824d26000-7c3824d4e000 r--p 00000000 07:00 7471 /usr/lib/x86_64-linux-gnu/libc.so.6 2024-06-06T23:02:00.061936+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3824d4e000-7c3824ee3000 r-xp 00028000 07:00 7471 /usr/lib/x86_64-linux-gnu/libc.so.6 2024-06-06T23:02:00.061959+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3824ee3000-7c3824f3b000 r--p 001bd000 07:00 7471 /usr/lib/x86_64-linux-gnu/libc.so.6 2024-06-06T23:02:00.061978+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3824f3b000-7c3824f3c000 ---p 00215000 07:00 7471 /usr/lib/x86_64-linux-gnu/libc.so.6 2024-06-06T23:02:00.062000+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3824f3c000-7c3824f40000 r--p 00215000 07:00 7471 /usr/lib/x86_64-linux-gnu/libc.so.6 2024-06-06T23:02:00.062032+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3824f40000-7c3824f42000 rw-p 00219000 07:00 7471 /usr/lib/x86_64-linux-gnu/libc.so.6 2024-06-06T23:02:00.062051+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3824f42000-7c3824f4f000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.062310+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3824f4f000-7c3824f52000 r--p 00000000 07:00 7521 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 2024-06-06T23:02:00.062338+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3824f52000-7c3824f69000 r-xp 00003000 07:00 7521 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 2024-06-06T23:02:00.062362+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3824f69000-7c3824f6d000 r--p 0001a000 07:00 7521 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 2024-06-06T23:02:00.062384+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3824f6d000-7c3824f6e000 r--p 0001d000 07:00 7521 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 2024-06-06T23:02:00.062414+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3824f6e000-7c3824f6f000 rw-p 0001e000 07:00 7521 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 2024-06-06T23:02:00.062446+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3824f6f000-7c3824f72000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.062464+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3824f72000-7c3824f79000 r--p 00000000 07:02 565 /snap/lxd/28463/lib/x86_64-linux-gnu/libfuse3.so.3.10.5 2024-06-06T23:02:00.062487+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3824f79000-7c3824f94000 r-xp 00007000 07:02 565 /snap/lxd/28463/lib/x86_64-linux-gnu/libfuse3.so.3.10.5 2024-06-06T23:02:00.062509+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3824f94000-7c3824f9e000 r--p 00022000 07:02 565 /snap/lxd/28463/lib/x86_64-linux-gnu/libfuse3.so.3.10.5 2024-06-06T23:02:00.062528+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3824f9e000-7c3824fb0000 r--p 0002b000 07:02 565 /snap/lxd/28463/lib/x86_64-linux-gnu/libfuse3.so.3.10.5 2024-06-06T23:02:00.062547+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3824fb0000-7c3824fb1000 rw-p 0003d000 07:02 565 /snap/lxd/28463/lib/x86_64-linux-gnu/libfuse3.so.3.10.5 2024-06-06T23:02:00.062569+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3824fb1000-7c3824fb2000 r--p 00000000 07:02 538 /snap/lxd/28463/lib/x86_64-linux-gnu/libSegFault.so 2024-06-06T23:02:00.062596+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3824fb2000-7c3824fb5000 r-xp 00001000 07:02 538 /snap/lxd/28463/lib/x86_64-linux-gnu/libSegFault.so 2024-06-06T23:02:00.062615+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3824fb5000-7c3824fb6000 r--p 00004000 07:02 538 /snap/lxd/28463/lib/x86_64-linux-gnu/libSegFault.so 2024-06-06T23:02:00.062642+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3824fb6000-7c3824fb7000 r--p 00005000 07:02 538 /snap/lxd/28463/lib/x86_64-linux-gnu/libSegFault.so 2024-06-06T23:02:00.062666+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3824fb7000-7c3824fb8000 rw-p 00006000 07:02 538 /snap/lxd/28463/lib/x86_64-linux-gnu/libSegFault.so 2024-06-06T23:02:00.062694+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3824fb8000-7c3824fba000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.062722+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3824fba000-7c3824fbc000 r--p 00000000 07:00 7442 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 2024-06-06T23:02:00.062762+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3824fbc000-7c3824fe6000 r-xp 00002000 07:00 7442 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 2024-06-06T23:02:00.062786+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3824fe6000-7c3824ff1000 r--p 0002c000 07:00 7442 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 2024-06-06T23:02:00.062811+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3824ff1000-7c3824ff2000 rw-p 00000000 00:00 0 2024-06-06T23:02:00.062835+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3824ff2000-7c3824ff4000 r--p 00037000 07:00 7442 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 2024-06-06T23:02:00.062860+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7c3824ff4000-7c3824ff6000 rw-p 00039000 07:00 7442 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 2024-06-06T23:02:00.062880+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7ffd7560f000-7ffd75630000 rw-p 00000000 00:00 0 [stack] 2024-06-06T23:02:00.062908+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7ffd757d0000-7ffd757d4000 r--p 00000000 00:00 0 [vvar] 2024-06-06T23:02:00.062949+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: 7ffd757d4000-7ffd757d6000 r-xp 00000000 00:00 0 [vdso] 2024-06-06T23:02:00.062972+02:00 lxdvm2-x14-u15-r1 lxd.daemon[2908]: ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0 [vsyscall] ```
mihalicyn commented 4 weeks ago

Hi @webdock-io

Thanks for your report! I'm working on your case right now.

Second one is interesting:

00000000000222fb <proc_read@@Base>:
   222fb:       f3 0f 1e fa             endbr64 
   222ff:       55                      push   %rbp
   22300:       48 89 e5                mov    %rsp,%rbp
   22303:       48 83 ec 40             sub    $0x40,%rsp

path:
   22307:       48 89 7d e8             mov    %rdi,-0x18(%rbp)

buf:
   2230b:       48 89 75 e0             mov    %rsi,-0x20(%rbp)

size:
   2230f:       48 89 55 d8             mov    %rdx,-0x28(%rbp)

offset:
   22313:       48 89 4d d0             mov    %rcx,-0x30(%rbp)

fi:
   22317:       4c 89 45 c8             mov    %r8,-0x38(%rbp)

rax = fi
   2231b:       48 8b 45 c8             mov    -0x38(%rbp),%rax

rax: fi
rax = *(rax + 16) = fi->fh
   2231f:       48 8b 40 10             mov    0x10(%rax),%rax

rax: fi->fh
*(rbp-8)=rax ( <=> ( f = fi->fh ) )
   22323:       48 89 45 f8             mov    %rax,-0x8(%rbp)

rbp: 7c38241ffae0
rax = *(rbp - 8)
   22327:       48 8b 45 f8             mov    -0x8(%rbp),%rax

rax: 0000000000008000 (invalid pointer!)
eax = *(rax+0x18)=*(rax+16+8) = f->type
   2232b:       8b 40 18                mov    0x18(%rax),%eax <=== BOOM!

offset 2232b <-> virtual address 7c3824d0532b

It looks like for some reason, fi->fh value is wrong (https://github.com/lxc/lxcfs/blob/62c723047ffc7784b097321dd6a7ce893b43a01a/src/proc_fuse.c#L1645).

I'm continuing investigation.

webdock-io commented 4 weeks ago

@mihalicyn That's amazing! Much appreciated.

I will collect crash dumps from other hosts where this has happened, I think we have seen this across 4 hosts thus far, and post them here when I have the chance - in case they are somehow different from what you've seen here already.

Cheers :)

webdock-io commented 4 weeks ago

Here are the remainder of crash dumps we've seen so far, so you can check if this is the same or different.

Crash data 1 ``` 2024-06-03T14:10:00.009431+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: *** signal 11 2024-06-03T14:10:00.009615+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: Register dump: 2024-06-03T14:10:00.009641+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: RAX: 00000007102b410e RBX: ffffffffffffff80 RCX: 0000710318568769 2024-06-03T14:10:00.009664+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: RDX: 0000710300bffc90 RSI: 00000007102b40fe RDI: 00000007102b410e 2024-06-03T14:10:00.009686+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: RBP: 0000710300bffb20 R8 : 000071031854d198 R9 : 000000000000000f 2024-06-03T14:10:00.009832+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: R10: 000071031854d198 R11: 0000000000000000 R12: 0000000000000004 2024-06-03T14:10:00.009922+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: R13: 00007101b410e050 R14: 00005f8313fe5718 R15: 0000000000000000 2024-06-03T14:10:00.009945+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: RSP: 0000710300bffad0 2024-06-03T14:10:00.009968+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: RIP: 00007103186343fe EFLAGS: 00010202 2024-06-03T14:10:00.009993+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: CS: 0033 FS: 0000 GS: 0000 2024-06-03T14:10:00.010018+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: Trap: 0000000e Error: 00000004 OldMask: 00004007 CR2: 102b4106 2024-06-03T14:10:00.010090+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: FPUCW: 0000037f FPUSW: 00000000 TAG: 00000000 2024-06-03T14:10:00.010116+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: RIP: 00000000 RDP: 00000000 2024-06-03T14:10:00.010138+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: ST(0) 0000 0000000000000000 ST(1) 0000 0000000000000000 2024-06-03T14:10:00.010159+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: ST(2) 0000 0000000000000000 ST(3) 0000 0000000000000000 2024-06-03T14:10:00.010181+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: ST(4) 0000 0000000000000000 ST(5) 0000 0000000000000000 2024-06-03T14:10:00.010206+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: ST(6) 0000 0000000000000000 ST(7) 0000 0000000000000000 2024-06-03T14:10:00.010228+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: mxcsr: 1fa0 2024-06-03T14:10:00.010252+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: XMM0: 00000000000000000000000000000000 XMM1: 00000000000000000000000000000000 2024-06-03T14:10:00.010277+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: XMM2: 00000000000000000000000000000000 XMM3: 00000000000000000000000000000000 2024-06-03T14:10:00.010307+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: XMM4: 00000000000000000000000000000000 XMM5: 00000000000000000000000000000000 2024-06-03T14:10:00.010332+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: XMM6: 00000000000000000000000000000000 XMM7: 00000000000000000000000000000000 2024-06-03T14:10:00.010361+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: XMM8: 00000000000000000000000000000000 XMM9: 00000000000000000000000000000000 2024-06-03T14:10:00.010384+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: XMM10: 00000000000000000000000000000000 XMM11: 00000000000000000000000000000000 2024-06-03T14:10:00.010419+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: XMM12: 00000000000000000000000000000000 XMM13: 00000000000000000000000000000000 2024-06-03T14:10:00.010455+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: XMM14: 00000000000000000000000000000000 XMM15: 00000000000000000000000000000000 2024-06-03T14:10:00.010479+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: Backtrace: 2024-06-03T14:10:00.012750+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: /lib/x86_64-linux-gnu/libc.so.6(free+0x1e)[0x7103186343fe] 2024-06-03T14:10:00.012828+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: /snap/lxd/current/lib/liblxcfs.so(do_release_file_info+0x42)[0x7103185730c8] 2024-06-03T14:10:00.012863+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: /snap/lxd/current/lib/liblxcfs.so(proc_release+0x20)[0x710318568789] 2024-06-03T14:10:00.012896+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: lxcfs(+0x2f2c)[0x5f8313dc9f2c] 2024-06-03T14:10:00.012933+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: lxcfs(+0x3c7f)[0x5f8313dcac7f] 2024-06-03T14:10:00.012966+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: /snap/lxd/current/lib/x86_64-linux-gnu/libfuse3.so.3(+0xbb6a)[0x7103187e6b6a] 2024-06-03T14:10:00.013042+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: /snap/lxd/current/lib/x86_64-linux-gnu/libfuse3.so.3(+0x104fa)[0x7103187eb4fa] 2024-06-03T14:10:00.013070+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: /snap/lxd/current/lib/x86_64-linux-gnu/libfuse3.so.3(+0x11738)[0x7103187ec738] 2024-06-03T14:10:00.013092+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: /snap/lxd/current/lib/x86_64-linux-gnu/libfuse3.so.3(+0x1e13f)[0x7103187f913f] 2024-06-03T14:10:00.013115+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: /snap/lxd/current/lib/x86_64-linux-gnu/libfuse3.so.3(+0x167a7)[0x7103187f17a7] 2024-06-03T14:10:00.013138+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x710318623ac3] 2024-06-03T14:10:00.013167+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: /lib/x86_64-linux-gnu/libc.so.6(+0x126850)[0x7103186b5850] 2024-06-03T14:10:00.013214+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: Memory map: 2024-06-03T14:10:00.013242+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 5f8313dc7000-5f8313dcf000 r-xp 00000000 07:01 44 /snap/lxd/28463/bin/lxcfs 2024-06-03T14:10:00.013264+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 5f8313dcf000-5f8313dd0000 r--p 00007000 07:01 44 /snap/lxd/28463/bin/lxcfs 2024-06-03T14:10:00.013289+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 5f8313dd0000-5f8313dd1000 rw-p 00008000 07:01 44 /snap/lxd/28463/bin/lxcfs 2024-06-03T14:10:00.013311+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 5f8313fd8000-5f8314027000 rw-p 00000000 00:00 0 [heap] 2024-06-03T14:10:00.013341+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7101b4000000-7101b4163000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.013434+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7101b4163000-7101b8000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.013466+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7101b8000000-7101b8156000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.013494+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7101b8156000-7101bc000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.013519+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7101c0000000-7101c015b000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.013541+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7101c015b000-7101c4000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.013565+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7101c4000000-7101c415d000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.013590+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7101c415d000-7101c8000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.013615+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7101cc000000-7101cc107000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.013639+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7101cc107000-7101d0000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.013664+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7101d0000000-7101d0153000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.013688+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7101d0153000-7101d4000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.013712+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7101d8000000-7101d811e000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.013732+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7101d811e000-7101dc000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.013755+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7101dc000000-7101dc14e000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.013779+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7101dc14e000-7101e0000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.013804+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7101e4000000-7101e4116000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.013832+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7101e4116000-7101e8000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.013857+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7101e8000000-7101e814f000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.013883+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7101e814f000-7101ec000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.013907+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7101f0000000-7101f011b000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.013932+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7101f011b000-7101f4000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.013956+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7101f4000000-7101f415d000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.013981+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7101f415d000-7101f8000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.014006+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7101fc000000-7101fc120000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.014042+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7101fc120000-710200000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.014067+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710200000000-71020014f000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.014088+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 71020014f000-710204000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.014112+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710208000000-710208166000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.014135+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710208166000-71020c000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.014160+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 71020c000000-71020c15b000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.014184+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 71020c15b000-710210000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.014213+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710214000000-71021413b000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.014256+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 71021413b000-710218000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.014282+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710218000000-71021815a000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.014302+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 71021815a000-71021c000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.014327+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710220000000-710220203000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.014350+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710220203000-710224000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.014375+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710224000000-710224135000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.014399+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710224135000-710228000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.014433+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 71022c000000-71022c15c000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.014460+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 71022c15c000-710230000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.014489+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710230000000-71023015a000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.014514+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 71023015a000-710234000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.014535+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710238000000-710238158000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.014586+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710238158000-71023c000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.014632+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 71023c000000-71023c156000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.014661+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 71023c156000-710240000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.014686+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710244000000-710244203000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.014707+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710244203000-710248000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.014732+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710248000000-710248205000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.014756+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710248205000-71024c000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.014780+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710250000000-710250151000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.014804+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710250151000-710254000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.014828+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710254000000-71025415e000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.014853+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 71025415e000-710258000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.014878+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 71025c000000-71025c204000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.014901+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 71025c204000-710260000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.014926+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710260000000-710260156000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.014951+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710260156000-710264000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.014976+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710268000000-710268228000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.015005+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710268228000-71026c000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.018810+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 71026c000000-71026c11d000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.018983+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 71026c11d000-710270000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.019016+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710274000000-710274150000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.019080+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710274150000-710278000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.019127+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710278000000-71027815d000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.019152+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 71027815d000-71027c000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.019177+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710280000000-71028015e000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.019202+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 71028015e000-710284000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.019227+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710284000000-710284166000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.019332+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710284166000-710288000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.019365+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 71028c000000-71028c162000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.019387+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 71028c162000-710290000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.019454+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710290000000-710290172000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.019495+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710290172000-710294000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.019516+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710298000000-710298156000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.019541+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710298156000-71029c000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.019576+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 71029c000000-71029c15e000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.019604+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 71029c15e000-7102a0000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.019626+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7102a4000000-7102a4154000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.019651+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7102a4154000-7102a8000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.019675+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7102a8000000-7102a8162000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.019726+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7102a8162000-7102ac000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.019751+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7102b0000000-7102b0206000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.019775+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7102b0206000-7102b4000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.019800+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7102b4000000-7102b415d000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.019829+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7102b415d000-7102b8000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.019854+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7102bc000000-7102bc15d000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.019878+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7102bc15d000-7102c0000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.019903+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7102c0000000-7102c015c000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.019927+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7102c015c000-7102c4000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.019948+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7102c8000000-7102c8160000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.019978+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7102c8160000-7102cc000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.020003+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7102cc000000-7102cc168000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.020043+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7102cc168000-7102d0000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.020068+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7102d4000000-7102d4154000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.020124+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7102d4154000-7102d8000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.020149+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7102d8000000-7102d8134000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.020171+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7102d8134000-7102dc000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.020196+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7102e0000000-7102e0204000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.020220+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7102e0204000-7102e4000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.020244+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7102e4000000-7102e414f000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.020270+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7102e414f000-7102e8000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.020301+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7102ec000000-7102ec160000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.020330+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7102ec160000-7102f0000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.020356+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7102f0000000-7102f0158000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.020395+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7102f0158000-7102f4000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.020443+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7102f4e00000-7102f4e01000 ---p 00000000 00:00 0 2024-06-03T14:10:00.020532+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7102f4e01000-7102f5601000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.020557+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7102f6200000-7102f6201000 ---p 00000000 00:00 0 2024-06-03T14:10:00.020581+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7102f6201000-7102f6a01000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.020609+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7102f6c00000-7102f6c01000 ---p 00000000 00:00 0 2024-06-03T14:10:00.020672+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7102f6c01000-7102f7401000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.020815+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7102f8000000-7102f8154000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.020864+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7102f8154000-7102fc000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.020888+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7102fc000000-7102fc062000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.020911+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7102fc062000-710300000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.020935+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710300400000-710300401000 ---p 00000000 00:00 0 2024-06-03T14:10:00.020959+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710300401000-710300c01000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.020983+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710301800000-710301801000 ---p 00000000 00:00 0 2024-06-03T14:10:00.021008+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710301801000-710302001000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.021049+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710302200000-710302201000 ---p 00000000 00:00 0 2024-06-03T14:10:00.021069+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710302201000-710302a01000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.021093+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710302c00000-710302c01000 ---p 00000000 00:00 0 2024-06-03T14:10:00.021116+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710302c01000-710303401000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.021140+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710303600000-710303601000 ---p 00000000 00:00 0 2024-06-03T14:10:00.021171+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710303601000-710303e01000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.021193+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710304000000-71030414f000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.021214+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 71030414f000-710308000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.021235+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710308000000-71030812f000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.021257+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 71030812f000-71030c000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.021278+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 71030ce00000-71030ce01000 ---p 00000000 00:00 0 2024-06-03T14:10:00.021299+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 71030ce01000-71030d601000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.021320+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 71030ec00000-71030ec01000 ---p 00000000 00:00 0 2024-06-03T14:10:00.021341+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 71030ec01000-71030f401000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.021362+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 71030f600000-71030f601000 ---p 00000000 00:00 0 2024-06-03T14:10:00.021387+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 71030f601000-71030fe01000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.021410+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710310000000-710310162000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.021447+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710310162000-710314000000 ---p 00000000 00:00 0 2024-06-03T14:10:00.021464+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710315400000-710315401000 ---p 00000000 00:00 0 2024-06-03T14:10:00.021485+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710315401000-710315c01000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.021507+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710316800000-710316801000 ---p 00000000 00:00 0 2024-06-03T14:10:00.021528+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710316801000-710317001000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.021550+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710317200000-710317201000 ---p 00000000 00:00 0 2024-06-03T14:10:00.021575+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710317201000-710317a01000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.021593+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710317c00000-710317c01000 ---p 00000000 00:00 0 2024-06-03T14:10:00.021614+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710317c01000-710318401000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.021638+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710318448000-71031844a000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.021660+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7103184d1000-71031854c000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.021681+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 71031854c000-71031857f000 r-xp 00000000 07:01 155 /snap/lxd/28463/lib/liblxcfs.so 2024-06-03T14:10:00.021713+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 71031857f000-710318580000 r--p 00032000 07:01 155 /snap/lxd/28463/lib/liblxcfs.so 2024-06-03T14:10:00.021736+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710318580000-710318581000 rw-p 00033000 07:01 155 /snap/lxd/28463/lib/liblxcfs.so 2024-06-03T14:10:00.021774+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710318581000-71031858f000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.021796+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 71031858f000-7103185b7000 r--p 00000000 07:00 7471 /usr/lib/x86_64-linux-gnu/libc.so.6 2024-06-03T14:10:00.021818+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7103185b7000-71031874c000 r-xp 00028000 07:00 7471 /usr/lib/x86_64-linux-gnu/libc.so.6 2024-06-03T14:10:00.021843+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 71031874c000-7103187a4000 r--p 001bd000 07:00 7471 /usr/lib/x86_64-linux-gnu/libc.so.6 2024-06-03T14:10:00.021861+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7103187a4000-7103187a5000 ---p 00215000 07:00 7471 /usr/lib/x86_64-linux-gnu/libc.so.6 2024-06-03T14:10:00.021882+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7103187a5000-7103187a9000 r--p 00215000 07:00 7471 /usr/lib/x86_64-linux-gnu/libc.so.6 2024-06-03T14:10:00.021904+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7103187a9000-7103187ab000 rw-p 00219000 07:00 7471 /usr/lib/x86_64-linux-gnu/libc.so.6 2024-06-03T14:10:00.021926+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7103187ab000-7103187b8000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.021947+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7103187b8000-7103187bb000 r--p 00000000 07:00 7521 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 2024-06-03T14:10:00.021969+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7103187bb000-7103187d2000 r-xp 00003000 07:00 7521 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 2024-06-03T14:10:00.021991+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7103187d2000-7103187d6000 r--p 0001a000 07:00 7521 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 2024-06-03T14:10:00.024456+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7103187d6000-7103187d7000 r--p 0001d000 07:00 7521 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 2024-06-03T14:10:00.024629+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7103187d7000-7103187d8000 rw-p 0001e000 07:00 7521 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 2024-06-03T14:10:00.024682+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7103187d8000-7103187db000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.024794+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7103187db000-7103187e2000 r--p 00000000 07:01 565 /snap/lxd/28463/lib/x86_64-linux-gnu/libfuse3.so.3.10.5 2024-06-03T14:10:00.024833+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7103187e2000-7103187fd000 r-xp 00007000 07:01 565 /snap/lxd/28463/lib/x86_64-linux-gnu/libfuse3.so.3.10.5 2024-06-03T14:10:00.024867+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7103187fd000-710318807000 r--p 00022000 07:01 565 /snap/lxd/28463/lib/x86_64-linux-gnu/libfuse3.so.3.10.5 2024-06-03T14:10:00.024901+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710318807000-710318819000 r--p 0002b000 07:01 565 /snap/lxd/28463/lib/x86_64-linux-gnu/libfuse3.so.3.10.5 2024-06-03T14:10:00.024964+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710318819000-71031881a000 rw-p 0003d000 07:01 565 /snap/lxd/28463/lib/x86_64-linux-gnu/libfuse3.so.3.10.5 2024-06-03T14:10:00.025067+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 71031881a000-71031881b000 r--p 00000000 07:01 538 /snap/lxd/28463/lib/x86_64-linux-gnu/libSegFault.so 2024-06-03T14:10:00.025101+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 71031881b000-71031881e000 r-xp 00001000 07:01 538 /snap/lxd/28463/lib/x86_64-linux-gnu/libSegFault.so 2024-06-03T14:10:00.025127+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 71031881e000-71031881f000 r--p 00004000 07:01 538 /snap/lxd/28463/lib/x86_64-linux-gnu/libSegFault.so 2024-06-03T14:10:00.025152+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 71031881f000-710318820000 r--p 00005000 07:01 538 /snap/lxd/28463/lib/x86_64-linux-gnu/libSegFault.so 2024-06-03T14:10:00.025178+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710318820000-710318821000 rw-p 00006000 07:01 538 /snap/lxd/28463/lib/x86_64-linux-gnu/libSegFault.so 2024-06-03T14:10:00.025203+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710318821000-710318823000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.025225+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710318823000-710318825000 r--p 00000000 07:00 7442 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 2024-06-03T14:10:00.025250+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 710318825000-71031884f000 r-xp 00002000 07:00 7442 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 2024-06-03T14:10:00.025280+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 71031884f000-71031885a000 r--p 0002c000 07:00 7442 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 2024-06-03T14:10:00.025301+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 71031885a000-71031885b000 rw-p 00000000 00:00 0 2024-06-03T14:10:00.025326+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 71031885b000-71031885d000 r--p 00037000 07:00 7442 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 2024-06-03T14:10:00.025344+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 71031885d000-71031885f000 rw-p 00039000 07:00 7442 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 2024-06-03T14:10:00.025366+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7ffe62a68000-7ffe62a89000 rw-p 00000000 00:00 0 [stack] 2024-06-03T14:10:00.025393+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7ffe62bc9000-7ffe62bcd000 r--p 00000000 00:00 0 [vvar] 2024-06-03T14:10:00.025423+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: 7ffe62bcd000-7ffe62bcf000 r-xp 00000000 00:00 0 [vdso] 2024-06-03T14:10:00.025448+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2883]: ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0 [vsyscall] ```
Crash data 2 ``` 2024-06-06T08:40:00.009928+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: *** signal 11 2024-06-06T08:40:00.010279+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: Register dump: 2024-06-06T08:40:00.010316+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: RAX: 00000007aba80103 RBX: ffffffffffffff80 RCX: 00007abb22b2e769 2024-06-06T08:40:00.010343+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: RDX: 00007abb215ffc90 RSI: 00000007aba800f3 RDI: 00000007aba80103 2024-06-06T08:40:00.010373+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: RBP: 00007abb215ffb20 R8 : 00007abb22b13198 R9 : 000000000000000f 2024-06-06T08:40:00.010422+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: R10: 00007abb22b13198 R11: 0000000000000000 R12: 0000000000000008 2024-06-06T08:40:00.010451+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: R13: 00007ababc106b70 R14: 00005c3e09e8c6f8 R15: 0000000000000000 2024-06-06T08:40:00.010479+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: RSP: 00007abb215ffad0 2024-06-06T08:40:00.010508+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: RIP: 00007abb22bfa3fe EFLAGS: 00010202 2024-06-06T08:40:00.010545+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: CS: 0033 FS: 0000 GS: 0000 2024-06-06T08:40:00.010578+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: Trap: 0000000e Error: 00000004 OldMask: 00004007 CR2: aba800fb 2024-06-06T08:40:00.010603+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: FPUCW: 0000037f FPUSW: 00000000 TAG: 00000000 2024-06-06T08:40:00.010630+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: RIP: 00000000 RDP: 00000000 2024-06-06T08:40:00.010658+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: ST(0) 0000 0000000000000000 ST(1) 0000 0000000000000000 2024-06-06T08:40:00.010685+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: ST(2) 0000 0000000000000000 ST(3) 0000 0000000000000000 2024-06-06T08:40:00.010712+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: ST(4) 0000 0000000000000000 ST(5) 0000 0000000000000000 2024-06-06T08:40:00.010738+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: ST(6) 0000 0000000000000000 ST(7) 0000 0000000000000000 2024-06-06T08:40:00.010765+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: mxcsr: 1fa0 2024-06-06T08:40:00.010793+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: XMM0: 00000000000000000000000000000000 XMM1: 00000000000000000000000000000000 2024-06-06T08:40:00.010821+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: XMM2: 00000000000000000000000000000000 XMM3: 00000000000000000000000000000000 2024-06-06T08:40:00.010849+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: XMM4: 00000000000000000000000000000000 XMM5: 00000000000000000000000000000000 2024-06-06T08:40:00.010876+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: XMM6: 00000000000000000000000000000000 XMM7: 00000000000000000000000000000000 2024-06-06T08:40:00.010904+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: XMM8: 00000000000000000000000000000000 XMM9: 00000000000000000000000000000000 2024-06-06T08:40:00.010936+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: XMM10: 00000000000000000000000000000000 XMM11: 00000000000000000000000000000000 2024-06-06T08:40:00.010970+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: XMM12: 00000000000000000000000000000000 XMM13: 00000000000000000000000000000000 2024-06-06T08:40:00.010997+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: XMM14: 00000000000000000000000000000000 XMM15: 00000000000000000000000000000000 2024-06-06T08:40:00.011024+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: Backtrace: 2024-06-06T08:40:00.011111+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: /lib/x86_64-linux-gnu/libc.so.6(free+0x1e)[0x7abb22bfa3fe] 2024-06-06T08:40:00.011149+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: /snap/lxd/current/lib/liblxcfs.so(do_release_file_info+0x42)[0x7abb22b390c8] 2024-06-06T08:40:00.011178+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: /snap/lxd/current/lib/liblxcfs.so(proc_release+0x20)[0x7abb22b2e789] 2024-06-06T08:40:00.011205+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: lxcfs(+0x2f2c)[0x5c3e0903bf2c] 2024-06-06T08:40:00.011229+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: lxcfs(+0x3c7f)[0x5c3e0903cc7f] 2024-06-06T08:40:00.011256+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: /snap/lxd/current/lib/x86_64-linux-gnu/libfuse3.so.3(+0xbb6a)[0x7abb22dacb6a] 2024-06-06T08:40:00.011284+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: /snap/lxd/current/lib/x86_64-linux-gnu/libfuse3.so.3(+0x104fa)[0x7abb22db14fa] 2024-06-06T08:40:00.011311+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: /snap/lxd/current/lib/x86_64-linux-gnu/libfuse3.so.3(+0x11738)[0x7abb22db2738] 2024-06-06T08:40:00.011339+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: /snap/lxd/current/lib/x86_64-linux-gnu/libfuse3.so.3(+0x1e13f)[0x7abb22dbf13f] 2024-06-06T08:40:00.011370+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: /snap/lxd/current/lib/x86_64-linux-gnu/libfuse3.so.3(+0x167a7)[0x7abb22db77a7] 2024-06-06T08:40:00.012460+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7abb22be9ac3] 2024-06-06T08:40:00.012534+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: /lib/x86_64-linux-gnu/libc.so.6(+0x126850)[0x7abb22c7b850] 2024-06-06T08:40:00.012573+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: Memory map: 2024-06-06T08:40:00.012616+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 5c3e09039000-5c3e09041000 r-xp 00000000 07:01 44 /snap/lxd/28463/bin/lxcfs 2024-06-06T08:40:00.012654+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 5c3e09041000-5c3e09042000 r--p 00007000 07:01 44 /snap/lxd/28463/bin/lxcfs 2024-06-06T08:40:00.012696+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 5c3e09042000-5c3e09043000 rw-p 00008000 07:01 44 /snap/lxd/28463/bin/lxcfs 2024-06-06T08:40:00.012733+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 5c3e09e7f000-5c3e09ece000 rw-p 00000000 00:00 0 [heap] 2024-06-06T08:40:00.012774+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7aba60000000-7aba6014b000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.012814+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7aba6014b000-7aba64000000 ---p 00000000 00:00 0 2024-06-06T08:40:00.012854+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7aba68000000-7aba68156000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.012895+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7aba68156000-7aba6c000000 ---p 00000000 00:00 0 2024-06-06T08:40:00.012934+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7aba6c000000-7aba6c158000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.012975+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7aba6c158000-7aba70000000 ---p 00000000 00:00 0 2024-06-06T08:40:00.013011+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7aba74000000-7aba74154000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.013050+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7aba74154000-7aba78000000 ---p 00000000 00:00 0 2024-06-06T08:40:00.013116+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7aba78000000-7aba78203000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.013158+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7aba78203000-7aba7c000000 ---p 00000000 00:00 0 2024-06-06T08:40:00.013199+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7aba80000000-7aba80161000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.013660+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7aba80161000-7aba84000000 ---p 00000000 00:00 0 2024-06-06T08:40:00.013720+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7aba84000000-7aba8415b000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.013763+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7aba8415b000-7aba88000000 ---p 00000000 00:00 0 2024-06-06T08:40:00.013800+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7aba8c000000-7aba8c203000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.013836+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7aba8c203000-7aba90000000 ---p 00000000 00:00 0 2024-06-06T08:40:00.013873+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7aba90000000-7aba9015f000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.013909+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7aba9015f000-7aba94000000 ---p 00000000 00:00 0 2024-06-06T08:40:00.013974+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7aba98000000-7aba9815a000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.014011+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7aba9815a000-7aba9c000000 ---p 00000000 00:00 0 2024-06-06T08:40:00.014052+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7aba9c000000-7aba9c157000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.014106+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7aba9c157000-7abaa0000000 ---p 00000000 00:00 0 2024-06-06T08:40:00.014143+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abaa4000000-7abaa415f000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.014185+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abaa415f000-7abaa8000000 ---p 00000000 00:00 0 2024-06-06T08:40:00.014222+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abaa8000000-7abaa8156000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.014345+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abaa8156000-7abaac000000 ---p 00000000 00:00 0 2024-06-06T08:40:00.014382+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abab0000000-7abab015a000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.014435+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abab015a000-7abab4000000 ---p 00000000 00:00 0 2024-06-06T08:40:00.014472+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abab4000000-7abab4203000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.014508+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abab4203000-7abab8000000 ---p 00000000 00:00 0 2024-06-06T08:40:00.014543+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7ababc000000-7ababc205000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.014580+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7ababc205000-7abac0000000 ---p 00000000 00:00 0 2024-06-06T08:40:00.014621+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abac0000000-7abac015b000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.014657+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abac015b000-7abac4000000 ---p 00000000 00:00 0 2024-06-06T08:40:00.014693+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abac8000000-7abac8230000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.014730+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abac8230000-7abacc000000 ---p 00000000 00:00 0 2024-06-06T08:40:00.014765+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abacc000000-7abacc15e000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.014807+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abacc15e000-7abad0000000 ---p 00000000 00:00 0 2024-06-06T08:40:00.014849+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abad4000000-7abad4166000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.014890+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abad4166000-7abad8000000 ---p 00000000 00:00 0 2024-06-06T08:40:00.014926+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abad8000000-7abad8151000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.014962+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abad8151000-7abadc000000 ---p 00000000 00:00 0 2024-06-06T08:40:00.014998+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abae0000000-7abae005f000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.015035+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abae005f000-7abae4000000 ---p 00000000 00:00 0 2024-06-06T08:40:00.015083+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abae4000000-7abae4154000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.015119+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abae4154000-7abae8000000 ---p 00000000 00:00 0 2024-06-06T08:40:00.015159+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abae9800000-7abae9801000 ---p 00000000 00:00 0 2024-06-06T08:40:00.015195+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abae9801000-7abaea001000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.015233+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abaec000000-7abaec156000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.015323+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abaec156000-7abaf0000000 ---p 00000000 00:00 0 2024-06-06T08:40:00.015360+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abaf0000000-7abaf0165000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.015409+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abaf0165000-7abaf4000000 ---p 00000000 00:00 0 2024-06-06T08:40:00.015447+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abaf8000000-7abaf8163000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.015489+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abaf8163000-7abafc000000 ---p 00000000 00:00 0 2024-06-06T08:40:00.015526+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abafc000000-7abafc166000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.015562+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abafc166000-7abb00000000 ---p 00000000 00:00 0 2024-06-06T08:40:00.015598+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb00e00000-7abb00e01000 ---p 00000000 00:00 0 2024-06-06T08:40:00.015634+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb00e01000-7abb01601000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.015671+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb01800000-7abb01801000 ---p 00000000 00:00 0 2024-06-06T08:40:00.015734+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb01801000-7abb02001000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.015777+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb02200000-7abb02201000 ---p 00000000 00:00 0 2024-06-06T08:40:00.015819+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb02201000-7abb02a01000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.015860+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb02c00000-7abb02c01000 ---p 00000000 00:00 0 2024-06-06T08:40:00.015903+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb02c01000-7abb03401000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.015948+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb03600000-7abb03601000 ---p 00000000 00:00 0 2024-06-06T08:40:00.015988+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb03601000-7abb03e01000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.016027+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb04000000-7abb04163000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.016091+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb04163000-7abb08000000 ---p 00000000 00:00 0 2024-06-06T08:40:00.016140+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb08000000-7abb08170000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.016178+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb08170000-7abb0c000000 ---p 00000000 00:00 0 2024-06-06T08:40:00.016218+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb0e200000-7abb0e201000 ---p 00000000 00:00 0 2024-06-06T08:40:00.016315+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb0e201000-7abb0ea01000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.016356+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb0f600000-7abb0f601000 ---p 00000000 00:00 0 2024-06-06T08:40:00.016413+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb0f601000-7abb0fe01000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.016461+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb10000000-7abb10159000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.016502+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb10159000-7abb14000000 ---p 00000000 00:00 0 2024-06-06T08:40:00.016541+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb14000000-7abb14155000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.016582+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb14155000-7abb18000000 ---p 00000000 00:00 0 2024-06-06T08:40:00.016618+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb18e00000-7abb18e01000 ---p 00000000 00:00 0 2024-06-06T08:40:00.016661+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb18e01000-7abb19601000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.016698+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb19800000-7abb19801000 ---p 00000000 00:00 0 2024-06-06T08:40:00.016739+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb19801000-7abb1a001000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.016775+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb1a200000-7abb1a201000 ---p 00000000 00:00 0 2024-06-06T08:40:00.016816+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb1a201000-7abb1aa01000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.016853+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb1b600000-7abb1b601000 ---p 00000000 00:00 0 2024-06-06T08:40:00.016888+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb1b601000-7abb1be01000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.016921+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb1c000000-7abb1c161000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.016956+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb1c161000-7abb20000000 ---p 00000000 00:00 0 2024-06-06T08:40:00.016992+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb20e00000-7abb20e01000 ---p 00000000 00:00 0 2024-06-06T08:40:00.017029+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb20e01000-7abb21601000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.017081+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb21800000-7abb21801000 ---p 00000000 00:00 0 2024-06-06T08:40:00.017120+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb21801000-7abb22001000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.017158+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb22200000-7abb22201000 ---p 00000000 00:00 0 2024-06-06T08:40:00.017190+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb22201000-7abb22a01000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.017222+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb22a0e000-7abb22a10000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.017317+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb22a97000-7abb22b12000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.017360+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb22b12000-7abb22b45000 r-xp 00000000 07:01 155 /snap/lxd/28463/lib/liblxcfs.so 2024-06-06T08:40:00.017407+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb22b45000-7abb22b46000 r--p 00032000 07:01 155 /snap/lxd/28463/lib/liblxcfs.so 2024-06-06T08:40:00.017477+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb22b46000-7abb22b47000 rw-p 00033000 07:01 155 /snap/lxd/28463/lib/liblxcfs.so 2024-06-06T08:40:00.017518+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb22b47000-7abb22b55000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.017556+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb22b55000-7abb22b7d000 r--p 00000000 07:00 7471 /usr/lib/x86_64-linux-gnu/libc.so.6 2024-06-06T08:40:00.017594+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb22b7d000-7abb22d12000 r-xp 00028000 07:00 7471 /usr/lib/x86_64-linux-gnu/libc.so.6 2024-06-06T08:40:00.017631+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb22d12000-7abb22d6a000 r--p 001bd000 07:00 7471 /usr/lib/x86_64-linux-gnu/libc.so.6 2024-06-06T08:40:00.017669+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb22d6a000-7abb22d6b000 ---p 00215000 07:00 7471 /usr/lib/x86_64-linux-gnu/libc.so.6 2024-06-06T08:40:00.017706+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb22d6b000-7abb22d6f000 r--p 00215000 07:00 7471 /usr/lib/x86_64-linux-gnu/libc.so.6 2024-06-06T08:40:00.017743+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb22d6f000-7abb22d71000 rw-p 00219000 07:00 7471 /usr/lib/x86_64-linux-gnu/libc.so.6 2024-06-06T08:40:00.017779+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb22d71000-7abb22d7e000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.017817+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb22d7e000-7abb22d81000 r--p 00000000 07:00 7521 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 2024-06-06T08:40:00.017855+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb22d81000-7abb22d98000 r-xp 00003000 07:00 7521 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 2024-06-06T08:40:00.017912+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb22d98000-7abb22d9c000 r--p 0001a000 07:00 7521 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 2024-06-06T08:40:00.018025+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb22d9c000-7abb22d9d000 r--p 0001d000 07:00 7521 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 2024-06-06T08:40:00.018090+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb22d9d000-7abb22d9e000 rw-p 0001e000 07:00 7521 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 2024-06-06T08:40:00.018134+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb22d9e000-7abb22da1000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.018176+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb22da1000-7abb22da8000 r--p 00000000 07:01 565 /snap/lxd/28463/lib/x86_64-linux-gnu/libfuse3.so.3.10.5 2024-06-06T08:40:00.018218+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb22da8000-7abb22dc3000 r-xp 00007000 07:01 565 /snap/lxd/28463/lib/x86_64-linux-gnu/libfuse3.so.3.10.5 2024-06-06T08:40:00.018319+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb22dc3000-7abb22dcd000 r--p 00022000 07:01 565 /snap/lxd/28463/lib/x86_64-linux-gnu/libfuse3.so.3.10.5 2024-06-06T08:40:00.018359+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb22dcd000-7abb22ddf000 r--p 0002b000 07:01 565 /snap/lxd/28463/lib/x86_64-linux-gnu/libfuse3.so.3.10.5 2024-06-06T08:40:00.018433+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb22ddf000-7abb22de0000 rw-p 0003d000 07:01 565 /snap/lxd/28463/lib/x86_64-linux-gnu/libfuse3.so.3.10.5 2024-06-06T08:40:00.018476+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb22de0000-7abb22de1000 r--p 00000000 07:01 538 /snap/lxd/28463/lib/x86_64-linux-gnu/libSegFault.so 2024-06-06T08:40:00.018518+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb22de1000-7abb22de4000 r-xp 00001000 07:01 538 /snap/lxd/28463/lib/x86_64-linux-gnu/libSegFault.so 2024-06-06T08:40:00.018559+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb22de4000-7abb22de5000 r--p 00004000 07:01 538 /snap/lxd/28463/lib/x86_64-linux-gnu/libSegFault.so 2024-06-06T08:40:00.018851+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb22de5000-7abb22de6000 r--p 00005000 07:01 538 /snap/lxd/28463/lib/x86_64-linux-gnu/libSegFault.so 2024-06-06T08:40:00.018912+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb22de6000-7abb22de7000 rw-p 00006000 07:01 538 /snap/lxd/28463/lib/x86_64-linux-gnu/libSegFault.so 2024-06-06T08:40:00.018953+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb22de7000-7abb22de9000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.018997+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb22de9000-7abb22deb000 r--p 00000000 07:00 7442 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 2024-06-06T08:40:00.019032+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb22deb000-7abb22e15000 r-xp 00002000 07:00 7442 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 2024-06-06T08:40:00.019084+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb22e15000-7abb22e20000 r--p 0002c000 07:00 7442 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 2024-06-06T08:40:00.019125+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb22e20000-7abb22e21000 rw-p 00000000 00:00 0 2024-06-06T08:40:00.019433+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb22e21000-7abb22e23000 r--p 00037000 07:00 7442 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 2024-06-06T08:40:00.019656+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7abb22e23000-7abb22e25000 rw-p 00039000 07:00 7442 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 2024-06-06T08:40:00.019692+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7ffc17715000-7ffc17736000 rw-p 00000000 00:00 0 [stack] 2024-06-06T08:40:00.019724+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7ffc1774a000-7ffc1774e000 r--p 00000000 00:00 0 [vvar] 2024-06-06T08:40:00.019749+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: 7ffc1774e000-7ffc17750000 r-xp 00000000 00:00 0 [vdso] 2024-06-06T08:40:00.019774+02:00 lxdvm2-x14-u26-r1 lxd.daemon[2878]: ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0 [vsyscall] ```
Crash data 3 ``` 2024-06-07T07:59:00.045421+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: *** signal 11 2024-06-07T07:59:00.045640+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: Register dump: 2024-06-07T07:59:00.045691+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: RAX: 00007320b70e8300 RBX: 0000000000000000 RCX: 0000000000000000 2024-06-07T07:59:00.045726+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: RDX: 0000000000000222 RSI: 0000000000000000 RDI: 00007320a15ff750 2024-06-07T07:59:00.045750+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: RBP: 0000000000000000 R8 : 0000000000000000 R9 : 00007320a15ff840 2024-06-07T07:59:00.045783+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: R10: 0000000000000003 R11: 000000000000003f R12: 0000000000000222 2024-06-07T07:59:00.045807+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: R13: 00007320a15ff750 R14: 00007320b6eba988 R15: 00007320a15ff8d0 2024-06-07T07:59:00.045830+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: RSP: 00007320a15ff740 2024-06-07T07:59:00.045853+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: RIP: 00007320b6f59481 EFLAGS: 00010206 2024-06-07T07:59:00.045877+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: CS: 0033 FS: 0000 GS: 0000 2024-06-07T07:59:00.045901+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: Trap: 0000000e Error: 00000006 OldMask: 00004007 CR2: 00000000 2024-06-07T07:59:00.045930+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: FPUCW: 0000037f FPUSW: 00000000 TAG: 00000000 2024-06-07T07:59:00.045958+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: RIP: 00000000 RDP: 00000000 2024-06-07T07:59:00.045981+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: ST(0) 0000 0000000000000000 ST(1) 0000 0000000000000000 2024-06-07T07:59:00.046010+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: ST(2) 0000 0000000000000000 ST(3) 0000 0000000000000000 2024-06-07T07:59:00.046034+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: ST(4) 0000 0000000000000000 ST(5) 0000 0000000000000000 2024-06-07T07:59:00.046062+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: ST(6) 0000 0000000000000000 ST(7) 0000 0000000000000000 2024-06-07T07:59:00.046090+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: mxcsr: 1fa0 2024-06-07T07:59:00.046113+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: XMM0: 00000000000000000000000000000000 XMM1: 00000000000000000000000000000000 2024-06-07T07:59:00.046142+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: XMM2: 00000000000000000000000000000000 XMM3: 00000000000000000000000000000000 2024-06-07T07:59:00.046166+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: XMM4: 00000000000000000000000000000000 XMM5: 00000000000000000000000000000000 2024-06-07T07:59:00.046191+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: XMM6: 00000000000000000000000000000000 XMM7: 00000000000000000000000000000000 2024-06-07T07:59:00.046219+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: XMM8: 00000000000000000000000000000000 XMM9: 00000000000000000000000000000000 2024-06-07T07:59:00.046242+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: XMM10: 00000000000000000000000000000000 XMM11: 00000000000000000000000000000000 2024-06-07T07:59:00.046291+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: XMM12: 00000000000000000000000000000000 XMM13: 00000000000000000000000000000000 2024-06-07T07:59:00.046319+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: XMM14: 00000000000000000000000000000000 XMM15: 00000000000000000000000000000000 2024-06-07T07:59:00.046343+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: Backtrace: 2024-06-07T07:59:00.046366+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: /lib/x86_64-linux-gnu/libc.so.6(+0x88481)[0x7320b6f59481] 2024-06-07T07:59:00.046390+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: /lib/x86_64-linux-gnu/libc.so.6(snprintf+0x96)[0x7320b6f31856] 2024-06-07T07:59:00.046418+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: /snap/lxd/current/lib/liblxcfs.so(proc_loadavg_read+0x49b)[0x7320b6eb0f4a] 2024-06-07T07:59:00.046442+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: /snap/lxd/current/lib/liblxcfs.so(proc_read+0x26d)[0x7320b6eb0568] 2024-06-07T07:59:00.046469+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: lxcfs(+0x22c3)[0x6346fe2a22c3] 2024-06-07T07:59:00.046492+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: lxcfs(+0x3a56)[0x6346fe2a3a56] 2024-06-07T07:59:00.046517+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: /snap/lxd/current/lib/x86_64-linux-gnu/libfuse3.so.3(fuse_fs_read_buf+0x1be)[0x7320b712af2e] 2024-06-07T07:59:00.046541+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: /snap/lxd/current/lib/x86_64-linux-gnu/libfuse3.so.3(+0x10d66)[0x7320b712dd66] 2024-06-07T07:59:00.046607+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: /snap/lxd/current/lib/x86_64-linux-gnu/libfuse3.so.3(+0x114fa)[0x7320b712e4fa] 2024-06-07T07:59:00.046632+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: /snap/lxd/current/lib/x86_64-linux-gnu/libfuse3.so.3(+0x1e13f)[0x7320b713b13f] 2024-06-07T07:59:00.046659+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: /snap/lxd/current/lib/x86_64-linux-gnu/libfuse3.so.3(+0x167a7)[0x7320b71337a7] 2024-06-07T07:59:00.046682+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7320b6f65ac3] 2024-06-07T07:59:00.046705+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: /lib/x86_64-linux-gnu/libc.so.6(+0x126850)[0x7320b6ff7850] 2024-06-07T07:59:00.046729+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: Memory map: 2024-06-07T07:59:00.046758+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 6346fe2a0000-6346fe2a8000 r-xp 00000000 07:01 44 /snap/lxd/28463/bin/lxcfs 2024-06-07T07:59:00.046782+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 6346fe2a8000-6346fe2a9000 r--p 00007000 07:01 44 /snap/lxd/28463/bin/lxcfs 2024-06-07T07:59:00.046805+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 6346fe2a9000-6346fe2aa000 rw-p 00008000 07:01 44 /snap/lxd/28463/bin/lxcfs 2024-06-07T07:59:00.046829+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 6346ff86f000-6346ff8be000 rw-p 00000000 00:00 0 [heap] 2024-06-07T07:59:00.046852+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 731fb4000000-731fb414c000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.046881+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 731fb414c000-731fb8000000 ---p 00000000 00:00 0 2024-06-07T07:59:00.046910+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 731fb8000000-731fb8111000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.046933+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 731fb8111000-731fbc000000 ---p 00000000 00:00 0 2024-06-07T07:59:00.046957+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 731fc0000000-731fc0113000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.046980+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 731fc0113000-731fc4000000 ---p 00000000 00:00 0 2024-06-07T07:59:00.047009+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 731fc4000000-731fc4143000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.047037+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 731fc4143000-731fc8000000 ---p 00000000 00:00 0 2024-06-07T07:59:00.047060+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 731fcc000000-731fcc158000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.047083+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 731fcc158000-731fd0000000 ---p 00000000 00:00 0 2024-06-07T07:59:00.047110+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 731fd0000000-731fd0156000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.047129+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 731fd0156000-731fd4000000 ---p 00000000 00:00 0 2024-06-07T07:59:00.047157+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 731fd8000000-731fd8131000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.047183+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 731fd8131000-731fdc000000 ---p 00000000 00:00 0 2024-06-07T07:59:00.047206+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 731fdc000000-731fdc12c000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.047229+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 731fdc12c000-731fe0000000 ---p 00000000 00:00 0 2024-06-07T07:59:00.047253+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 731fe4000000-731fe413c000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.056702+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 731fe413c000-731fe8000000 ---p 00000000 00:00 0 2024-06-07T07:59:00.056795+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 731fe8000000-731fe812b000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.056860+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 731fe812b000-731fec000000 ---p 00000000 00:00 0 2024-06-07T07:59:00.056923+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 731ff0000000-731ff0159000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.064509+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 731ff0159000-731ff4000000 ---p 00000000 00:00 0 2024-06-07T07:59:00.064629+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 731ff4000000-731ff4115000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.064699+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 731ff4115000-731ff8000000 ---p 00000000 00:00 0 2024-06-07T07:59:00.064769+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 731ffc000000-731ffc116000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.064832+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 731ffc116000-732000000000 ---p 00000000 00:00 0 2024-06-07T07:59:00.064926+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 732000000000-73200014d000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.064996+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 73200014d000-732004000000 ---p 00000000 00:00 0 2024-06-07T07:59:00.065059+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 732008000000-732008154000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.065121+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 732008154000-73200c000000 ---p 00000000 00:00 0 2024-06-07T07:59:00.067642+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 73200c000000-73200c203000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.067720+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 73200c203000-732010000000 ---p 00000000 00:00 0 2024-06-07T07:59:00.067790+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 732014000000-732014114000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.067835+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 732014114000-732018000000 ---p 00000000 00:00 0 2024-06-07T07:59:00.067867+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 732018000000-73201811f000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.067891+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 73201811f000-73201c000000 ---p 00000000 00:00 0 2024-06-07T07:59:00.067919+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 732020000000-732020150000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.067942+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 732020150000-732024000000 ---p 00000000 00:00 0 2024-06-07T07:59:00.067968+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 732024000000-73202411e000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.067987+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 73202411e000-732028000000 ---p 00000000 00:00 0 2024-06-07T07:59:00.068037+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 73202c000000-73202c147000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.068132+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 73202c147000-732030000000 ---p 00000000 00:00 0 2024-06-07T07:59:00.068161+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 732030000000-73203011a000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.068236+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 73203011a000-732034000000 ---p 00000000 00:00 0 2024-06-07T07:59:00.068359+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 732038000000-73203811c000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.068419+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 73203811c000-73203c000000 ---p 00000000 00:00 0 2024-06-07T07:59:00.068480+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 73203c000000-73203c124000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.068556+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 73203c124000-732040000000 ---p 00000000 00:00 0 2024-06-07T07:59:00.068619+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 732044000000-732044204000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.068687+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 732044204000-732048000000 ---p 00000000 00:00 0 2024-06-07T07:59:00.068748+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 732048000000-732048203000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.068804+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 732048203000-73204c000000 ---p 00000000 00:00 0 2024-06-07T07:59:00.068873+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 732050000000-73205011f000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.068929+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 73205011f000-732054000000 ---p 00000000 00:00 0 2024-06-07T07:59:00.068995+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 732054000000-732054142000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.069055+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 732054142000-732058000000 ---p 00000000 00:00 0 2024-06-07T07:59:00.069116+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 73205c000000-73205c21e000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.069177+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 73205c21e000-732060000000 ---p 00000000 00:00 0 2024-06-07T07:59:00.069240+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 732060000000-73206021a000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.071897+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 73206021a000-732064000000 ---p 00000000 00:00 0 2024-06-07T07:59:00.071935+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 732068000000-732068152000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.071956+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 732068152000-73206c000000 ---p 00000000 00:00 0 2024-06-07T07:59:00.071987+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 73206c000000-73206c147000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.072011+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 73206c147000-732070000000 ---p 00000000 00:00 0 2024-06-07T07:59:00.072038+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 732074000000-732074112000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.072071+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 732074112000-732078000000 ---p 00000000 00:00 0 2024-06-07T07:59:00.072098+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 732078000000-732078205000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.072161+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 732078205000-73207c000000 ---p 00000000 00:00 0 2024-06-07T07:59:00.072194+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 732080000000-732080116000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.072221+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 732080116000-732084000000 ---p 00000000 00:00 0 2024-06-07T07:59:00.072248+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 732084000000-73208405e000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.072315+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 73208405e000-732088000000 ---p 00000000 00:00 0 2024-06-07T07:59:00.072348+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 73208c000000-73208c222000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.072380+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 73208c222000-732090000000 ---p 00000000 00:00 0 2024-06-07T07:59:00.072403+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 732090000000-732090172000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.072423+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 732090172000-732094000000 ---p 00000000 00:00 0 2024-06-07T07:59:00.072454+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 732096c00000-732096c01000 ---p 00000000 00:00 0 2024-06-07T07:59:00.072487+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 732096c01000-732097401000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.072514+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 732097600000-732097601000 ---p 00000000 00:00 0 2024-06-07T07:59:00.072540+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 732097601000-732097e01000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.072573+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 732098000000-73209822c000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.072600+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 73209822c000-73209c000000 ---p 00000000 00:00 0 2024-06-07T07:59:00.072631+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 73209c000000-73209c11d000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.072658+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 73209c11d000-7320a0000000 ---p 00000000 00:00 0 2024-06-07T07:59:00.072685+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320a0400000-7320a0401000 ---p 00000000 00:00 0 2024-06-07T07:59:00.072711+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320a0401000-7320a0c01000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.072738+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320a0e00000-7320a0e01000 ---p 00000000 00:00 0 2024-06-07T07:59:00.072765+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320a0e01000-7320a1601000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.072796+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320a1800000-7320a1801000 ---p 00000000 00:00 0 2024-06-07T07:59:00.075589+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320a1801000-7320a2001000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.075653+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320a2200000-7320a2201000 ---p 00000000 00:00 0 2024-06-07T07:59:00.075692+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320a2201000-7320a2a01000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.075742+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320a2c00000-7320a2c01000 ---p 00000000 00:00 0 2024-06-07T07:59:00.075797+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320a2c01000-7320a3401000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.075848+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320a3600000-7320a3601000 ---p 00000000 00:00 0 2024-06-07T07:59:00.075894+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320a3601000-7320a3e01000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.075937+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320a4000000-7320a415a000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.075980+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320a415a000-7320a8000000 ---p 00000000 00:00 0 2024-06-07T07:59:00.076028+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320a8000000-7320a8140000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.076070+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320a8140000-7320ac000000 ---p 00000000 00:00 0 2024-06-07T07:59:00.076118+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320ac400000-7320ac401000 ---p 00000000 00:00 0 2024-06-07T07:59:00.076161+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320ac401000-7320acc01000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.076204+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320ace00000-7320ace01000 ---p 00000000 00:00 0 2024-06-07T07:59:00.076242+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320ace01000-7320ad601000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.076330+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320ad800000-7320ad801000 ---p 00000000 00:00 0 2024-06-07T07:59:00.076374+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320ad801000-7320ae001000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.076424+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320ae200000-7320ae201000 ---p 00000000 00:00 0 2024-06-07T07:59:00.076465+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320ae201000-7320aea01000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.076551+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320aec00000-7320aec01000 ---p 00000000 00:00 0 2024-06-07T07:59:00.076615+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320aec01000-7320af401000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.076658+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320af600000-7320af601000 ---p 00000000 00:00 0 2024-06-07T07:59:00.076704+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320af601000-7320afe01000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.076741+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320b0000000-7320b0160000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.076778+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320b0160000-7320b4000000 ---p 00000000 00:00 0 2024-06-07T07:59:00.076817+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320b4800000-7320b4801000 ---p 00000000 00:00 0 2024-06-07T07:59:00.076857+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320b4801000-7320b5001000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.076902+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320b5200000-7320b5201000 ---p 00000000 00:00 0 2024-06-07T07:59:00.076944+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320b5201000-7320b5a01000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.076985+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320b5c00000-7320b5c01000 ---p 00000000 00:00 0 2024-06-07T07:59:00.077035+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320b5c01000-7320b6401000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.077073+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320b6600000-7320b6601000 ---p 00000000 00:00 0 2024-06-07T07:59:00.078917+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320b6601000-7320b6e01000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.078952+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320b6e76000-7320b6e8e000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.079008+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320b6e8e000-7320b6ec1000 r-xp 00000000 07:01 155 /snap/lxd/28463/lib/liblxcfs.so 2024-06-07T07:59:00.079035+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320b6ec1000-7320b6ec2000 r--p 00032000 07:01 155 /snap/lxd/28463/lib/liblxcfs.so 2024-06-07T07:59:00.079059+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320b6ec2000-7320b6ec3000 rw-p 00033000 07:01 155 /snap/lxd/28463/lib/liblxcfs.so 2024-06-07T07:59:00.079083+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320b6ec3000-7320b6ed1000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.079107+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320b6ed1000-7320b6ef9000 r--p 00000000 07:00 7471 /usr/lib/x86_64-linux-gnu/libc.so.6 2024-06-07T07:59:00.079132+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320b6ef9000-7320b708e000 r-xp 00028000 07:00 7471 /usr/lib/x86_64-linux-gnu/libc.so.6 2024-06-07T07:59:00.079155+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320b708e000-7320b70e6000 r--p 001bd000 07:00 7471 /usr/lib/x86_64-linux-gnu/libc.so.6 2024-06-07T07:59:00.079184+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320b70e6000-7320b70e7000 ---p 00215000 07:00 7471 /usr/lib/x86_64-linux-gnu/libc.so.6 2024-06-07T07:59:00.079212+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320b70e7000-7320b70eb000 r--p 00215000 07:00 7471 /usr/lib/x86_64-linux-gnu/libc.so.6 2024-06-07T07:59:00.079241+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320b70eb000-7320b70ed000 rw-p 00219000 07:00 7471 /usr/lib/x86_64-linux-gnu/libc.so.6 2024-06-07T07:59:00.083428+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320b70ed000-7320b70fa000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.083492+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320b70fa000-7320b70fd000 r--p 00000000 07:00 7521 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 2024-06-07T07:59:00.083532+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320b70fd000-7320b7114000 r-xp 00003000 07:00 7521 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 2024-06-07T07:59:00.083587+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320b7114000-7320b7118000 r--p 0001a000 07:00 7521 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 2024-06-07T07:59:00.083615+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320b7118000-7320b7119000 r--p 0001d000 07:00 7521 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 2024-06-07T07:59:00.083646+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320b7119000-7320b711a000 rw-p 0001e000 07:00 7521 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 2024-06-07T07:59:00.083671+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320b711a000-7320b711d000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.083698+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320b711d000-7320b7124000 r--p 00000000 07:01 565 /snap/lxd/28463/lib/x86_64-linux-gnu/libfuse3.so.3.10.5 2024-06-07T07:59:00.083725+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320b7124000-7320b713f000 r-xp 00007000 07:01 565 /snap/lxd/28463/lib/x86_64-linux-gnu/libfuse3.so.3.10.5 2024-06-07T07:59:00.083756+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320b713f000-7320b7149000 r--p 00022000 07:01 565 /snap/lxd/28463/lib/x86_64-linux-gnu/libfuse3.so.3.10.5 2024-06-07T07:59:00.083781+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320b7149000-7320b715b000 r--p 0002b000 07:01 565 /snap/lxd/28463/lib/x86_64-linux-gnu/libfuse3.so.3.10.5 2024-06-07T07:59:00.083807+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320b715b000-7320b715c000 rw-p 0003d000 07:01 565 /snap/lxd/28463/lib/x86_64-linux-gnu/libfuse3.so.3.10.5 2024-06-07T07:59:00.083837+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320b715c000-7320b715d000 r--p 00000000 07:01 538 /snap/lxd/28463/lib/x86_64-linux-gnu/libSegFault.so 2024-06-07T07:59:00.083867+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320b715d000-7320b7160000 r-xp 00001000 07:01 538 /snap/lxd/28463/lib/x86_64-linux-gnu/libSegFault.so 2024-06-07T07:59:00.083899+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320b7160000-7320b7161000 r--p 00004000 07:01 538 /snap/lxd/28463/lib/x86_64-linux-gnu/libSegFault.so 2024-06-07T07:59:00.083927+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320b7161000-7320b7162000 r--p 00005000 07:01 538 /snap/lxd/28463/lib/x86_64-linux-gnu/libSegFault.so 2024-06-07T07:59:00.083952+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320b7162000-7320b7163000 rw-p 00006000 07:01 538 /snap/lxd/28463/lib/x86_64-linux-gnu/libSegFault.so 2024-06-07T07:59:00.083978+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320b7163000-7320b7165000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.084000+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320b7165000-7320b7167000 r--p 00000000 07:00 7442 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 2024-06-07T07:59:00.084031+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320b7167000-7320b7191000 r-xp 00002000 07:00 7442 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 2024-06-07T07:59:00.084072+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320b7191000-7320b719c000 r--p 0002c000 07:00 7442 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 2024-06-07T07:59:00.084097+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320b719c000-7320b719d000 rw-p 00000000 00:00 0 2024-06-07T07:59:00.084122+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320b719d000-7320b719f000 r--p 00037000 07:00 7442 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 2024-06-07T07:59:00.084152+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7320b719f000-7320b71a1000 rw-p 00039000 07:00 7442 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 2024-06-07T07:59:00.084176+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7ffd2a921000-7ffd2a942000 rw-p 00000000 00:00 0 [stack] 2024-06-07T07:59:00.084202+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7ffd2a9d3000-7ffd2a9d7000 r--p 00000000 00:00 0 [vvar] 2024-06-07T07:59:00.084227+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: 7ffd2a9d7000-7ffd2a9d9000 r-xp 00000000 00:00 0 [vdso] 2024-06-07T07:59:00.084375+02:00 lxdvm1-x14-u19-r1 lxd.daemon[3333]: ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0 [vsyscall] ```
mihalicyn commented 4 weeks ago

Hm, interesting. All 3 cases are a bit different, but root cause is the same. Invalid value of INTTYPE_TO_PTR(fi->fh).

mihalicyn commented 4 weeks ago

@webdock-io can you please show uname -a? Have you recently updated your kernel?

webdock-io commented 4 weeks ago

Sure - all our instances with the issue are identical to this:

Linux lxdvm2-x14-u26-r1 6.8.0-31-generic #31-Ubuntu SMP PREEMPT_DYNAMIC Sat Apr 20 00:40:06 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

mihalicyn commented 4 weeks ago

all our instances with the issue are identical to this:

Did this problem appear recently? After a kernel upgrade? Or after LXD upgrade? (or both?..)

webdock-io commented 4 weeks ago

Well yes and no... These are not old / upgraded systems.. they are all brand new. All our newly deployed systems are on Noble and the latest kernel and latest stable lxd version.

These are also systems where we are running containers inside VMs which we haven't done before

So this is all new and the first time we run significant number of containers and load on these new systems

We are not seeing this on any of our old systems

We did do a lot of load testing before deployment and didn't see this happen. But we may have missed it as we weren't actively looking at lxcfs, rather at system stability

mihalicyn commented 4 weeks ago

@webdock-io thanks for providing an extra context on this ;-)

Can you try to collect a crash dump from the LXCFS?

All you need to do is (as root):

echo '|/bin/sh -c $@ -- eval exec cat > /var/crash/core-%e.%p' > /proc/sys/kernel/core_pattern

and wait until a next crash. Then collect a crashdump file in /var/crash/ directory and share it with me. (you can GPG encrypt it if you want with my key gpg --encrypt your_core_dump --recipient alexander@mihalicyn.com)

webdock-io commented 3 weeks ago

Hello. We have now run the command on all thus-far affected systems. If it happens again in the same location(s) we will have a dump for you

webdock-io commented 3 weeks ago

Small update: I'm really not sure we will see this type of crash again. We identified a config issue on our systems where our ZFS ARC cache limit was set way too low. This was causing abnormally high IOWAIT on our systems which seems to have been the root cause of our problems (we were also seeing system halts/crashes). After we upped our ARC cache limit to something more reasonable, we see really good behavior everywhere and thus far no lxcfs or system crashes.

This may point to something on the systems or in the kernel, due to the io bottleneck, getting messed up. It's only been a couple of days since we made this change, so the issue may not be completely solved - but so far it's looking good. We will be keeping the crash dump collection flag turned on and will of course revert here if we see it happening again.

Thank you for your time

mihalicyn commented 3 weeks ago

We identified a config issue on our systems where our ZFS ARC cache limit was set way too low. This was causing abnormally high IOWAIT on our systems which seems to have been the root cause of our problems (we were also seeing system halts/crashes).

Even when IOWAIT times are high, crashes are still crashes and we should fix it. So, you have a workaround but issue is still there. So I would not ignore it, and I would try to revert this workaround and try to get a reproduction and a crashdump so we can try to figure out what is wrong from the LXCFS/libfuse/kernel side which leads to a crashes like you've seen before.

we were also seeing system halts/crashes

so, not only LXCFS was crashing? What else was crashing then?

webdock-io commented 3 weeks ago

Hello. Yes I agree, crashes are still crashes. I think it may be difficult to get a reproducer on this, but we will have a system free soon where I can attempt to recreate the issue for you, but no guarantees.

so, not only LXCFS was crashing? What else was crashing then?

I do not want to confuse the matter. This was more of a statement that we saw other things misbehaving, namely our virtual machines. As mentioned earlier we are running these workloads inside LXD VMs - so, nested LXD if you will. Here we saw the VMs hang - no crash output, nothing in syslog no explanation that we could find, the VM would just go into a halt state and we would need to restart it to get it operational again. This did not coincide with the lxcfs crashes we saw, so they are not related per se but the root cause was the same, we think.

After we made the ARC cache happy and alleviated the bottleneck, we have not seen such a crash in 48+ hours now, where before we saw 1-2 crashes every 24 hours. So we are hopeful we have mitigated that issue as well. Although, a full VM crash is worrying to say the least, and as is the case with the lxcfs crash we haven't really fixed the issue, we've just resolved what was provoking the issue (we think/hope)

webdock-io commented 3 weeks ago

Hello

We have now seen this happen on a completely new host, which had our "workaround" implemented. So this crash is still happening and is still real. However, we have had to remove your crash dump reporting flag from the hosts where we did have it activated as it was filling up our disk! We are a public cloud hosting lxd container instances, and whatever crashes were happening in customer containers were being dumped to /var/crash. Not only did this create a lot of crash dumps on some hosts, worse it was quickly filling up disk on some systems. We caught it in time and averted disaster - but we simply cannot allow this crash dumping to be active on production systems.

So... In order to try and procure a crash dump for you, we have dedicated an instance to this purpose where we are stressing it a lot. We hope this will provoke a crash sometime in the next 24-48 hours - if not, then I'm not sure what we can do as then it seems synthetic workloads will not do the trick... We will see.

Anyway, here is the latest crash on an otherwise healthy system, so you can see if it's the same (I'm thinking it is)

Crash data ``` 2024-06-11T19:38:46.020228+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: *** signal 11 2024-06-11T19:38:46.020421+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: Register dump: 2024-06-11T19:38:46.020454+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: RAX: 00007c9e3011ef80 RBX: 000008e5000008e0 RCX: 0000008e5000008c 2024-06-11T19:38:46.020480+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: RDX: 0000000000000001 RSI: 00007c9eac104980 RDI: 00007c9e30000030 2024-06-11T19:38:46.020508+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: RBP: 00007c9e30105610 R8 : 00007c9ef721d198 R9 : 000000000000000f 2024-06-11T19:38:46.020542+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: R10: 00007c9ef721d198 R11: 0000000000000000 R12: 00007c9e30000030 2024-06-11T19:38:46.020564+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: R13: 0000000000000000 R14: 0000858330105ef0 R15: 0000000000000000 2024-06-11T19:38:46.020585+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: RSP: 00007c9eee9ffa60 2024-06-11T19:38:46.020608+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: RIP: 00007c9ef7301857 EFLAGS: 00010202 2024-06-11T19:38:46.020629+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: CS: 0033 FS: 0000 GS: 0000 2024-06-11T19:38:46.020656+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: Trap: 0000000d Error: 00000000 OldMask: 00004007 CR2: 00000000 2024-06-11T19:38:46.020695+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: FPUCW: 0000037f FPUSW: 00000000 TAG: 00000000 2024-06-11T19:38:46.020722+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: RIP: 00000000 RDP: 00000000 2024-06-11T19:38:46.020745+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: ST(0) 0000 0000000000000000 ST(1) 0000 0000000000000000 2024-06-11T19:38:46.020775+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: ST(2) 0000 0000000000000000 ST(3) 0000 0000000000000000 2024-06-11T19:38:46.020804+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: ST(4) 0000 0000000000000000 ST(5) 0000 0000000000000000 2024-06-11T19:38:46.020828+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: ST(6) 0000 0000000000000000 ST(7) 0000 0000000000000000 2024-06-11T19:38:46.020855+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: mxcsr: 1fa0 2024-06-11T19:38:46.020899+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: XMM0: 00000000000000000000000000000000 XMM1: 00000000000000000000000000000000 2024-06-11T19:38:46.020927+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: XMM2: 00000000000000000000000000000000 XMM3: 00000000000000000000000000000000 2024-06-11T19:38:46.020959+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: XMM4: 00000000000000000000000000000000 XMM5: 00000000000000000000000000000000 2024-06-11T19:38:46.020980+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: XMM6: 00000000000000000000000000000000 XMM7: 00000000000000000000000000000000 2024-06-11T19:38:46.021002+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: XMM8: 00000000000000000000000000000000 XMM9: 00000000000000000000000000000000 2024-06-11T19:38:46.021023+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: XMM10: 00000000000000000000000000000000 XMM11: 00000000000000000000000000000000 2024-06-11T19:38:46.021044+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: XMM12: 00000000000000000000000000000000 XMM13: 00000000000000000000000000000000 2024-06-11T19:38:46.021071+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: XMM14: 00000000000000000000000000000000 XMM15: 00000000000000000000000000000000 2024-06-11T19:38:46.021097+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: Backtrace: 2024-06-11T19:38:46.021156+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: /lib/x86_64-linux-gnu/libc.so.6(+0xa2857)[0x7c9ef7301857] 2024-06-11T19:38:46.021198+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: /lib/x86_64-linux-gnu/libc.so.6(free+0x73)[0x7c9ef7304453] 2024-06-11T19:38:46.021220+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: /snap/lxd/current/lib/liblxcfs.so(do_release_file_info+0x42)[0x7c9ef72430c8] 2024-06-11T19:38:46.021247+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: /snap/lxd/current/lib/liblxcfs.so(proc_release+0x20)[0x7c9ef7238789] 2024-06-11T19:38:46.021273+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: lxcfs(+0x2f2c)[0x63e917b83f2c] 2024-06-11T19:38:46.021299+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: lxcfs(+0x3c7f)[0x63e917b84c7f] 2024-06-11T19:38:46.021345+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: /snap/lxd/current/lib/x86_64-linux-gnu/libfuse3.so.3(+0xbb6a)[0x7c9ef74b6b6a] 2024-06-11T19:38:46.021372+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: /snap/lxd/current/lib/x86_64-linux-gnu/libfuse3.so.3(+0x104fa)[0x7c9ef74bb4fa] 2024-06-11T19:38:46.021403+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: /snap/lxd/current/lib/x86_64-linux-gnu/libfuse3.so.3(+0x11738)[0x7c9ef74bc738] 2024-06-11T19:38:46.021425+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: /snap/lxd/current/lib/x86_64-linux-gnu/libfuse3.so.3(+0x1e13f)[0x7c9ef74c913f] 2024-06-11T19:38:46.021450+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: /snap/lxd/current/lib/x86_64-linux-gnu/libfuse3.so.3(+0x167a7)[0x7c9ef74c17a7] 2024-06-11T19:38:46.021476+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x7c9ef72f3ac3] 2024-06-11T19:38:46.021504+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: /lib/x86_64-linux-gnu/libc.so.6(+0x126850)[0x7c9ef7385850] 2024-06-11T19:38:46.021525+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: Memory map: 2024-06-11T19:38:46.021551+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 63e917b81000-63e917b89000 r-xp 00000000 07:02 44 /snap/lxd/28463/bin/lxcfs 2024-06-11T19:38:46.021578+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 63e917b89000-63e917b8a000 r--p 00007000 07:02 44 /snap/lxd/28463/bin/lxcfs 2024-06-11T19:38:46.021604+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 63e917b8a000-63e917b8b000 rw-p 00008000 07:02 44 /snap/lxd/28463/bin/lxcfs 2024-06-11T19:38:46.021630+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 63e91820d000-63e91825c000 rw-p 00000000 00:00 0 [heap] 2024-06-11T19:38:46.021658+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9d64000000-7c9d64203000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.021687+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9d64203000-7c9d68000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.021709+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9d68000000-7c9d68223000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.021739+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9d68223000-7c9d6c000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.021759+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9d70000000-7c9d70175000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.021784+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9d70175000-7c9d74000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.021805+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9d74000000-7c9d74204000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.021826+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9d74204000-7c9d78000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.021856+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9d7c000000-7c9d7c229000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.021877+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9d7c229000-7c9d80000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.021907+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9d80000000-7c9d80204000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.021929+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9d80204000-7c9d84000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.021955+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9d88000000-7c9d88215000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.021984+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9d88215000-7c9d8c000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.022006+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9d8c000000-7c9d8c172000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.022029+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9d8c172000-7c9d90000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.022073+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9d94000000-7c9d94119000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.022099+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9d94119000-7c9d98000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.026498+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9d98000000-7c9d9816f000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.026549+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9d9816f000-7c9d9c000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.026579+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9da0000000-7c9da016d000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.026606+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9da016d000-7c9da4000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.026629+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9da4000000-7c9da4204000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.026652+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9da4204000-7c9da8000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.026681+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9dac000000-7c9dac177000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.026703+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9dac177000-7c9db0000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.026734+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9db0000000-7c9db0203000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.026768+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9db0203000-7c9db4000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.026797+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9db8000000-7c9db8168000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.026819+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9db8168000-7c9dbc000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.026850+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9dbc000000-7c9dbc204000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.026876+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9dbc204000-7c9dc0000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.026903+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9dc4000000-7c9dc4204000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.026930+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9dc4204000-7c9dc8000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.026952+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9dc8000000-7c9dc8177000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.026974+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9dc8177000-7c9dcc000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.027000+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9dd0000000-7c9dd0113000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.027022+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9dd0113000-7c9dd4000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.027048+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9dd4000000-7c9dd416e000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.027073+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9dd416e000-7c9dd8000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.027095+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ddc000000-7c9ddc204000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.027122+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ddc204000-7c9de0000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.027143+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9de0000000-7c9de010f000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.027169+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9de010f000-7c9de4000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.027207+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9de8000000-7c9de8203000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.027230+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9de8203000-7c9dec000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.027257+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9dec000000-7c9dec179000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.027279+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9dec179000-7c9df0000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.027320+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9df4000000-7c9df4203000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.027348+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9df4203000-7c9df8000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.027374+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9df8000000-7c9df8114000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.027399+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9df8114000-7c9dfc000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.027420+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e00000000-7c9e00213000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.027448+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e00213000-7c9e04000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.027474+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e04000000-7c9e04203000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.027504+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e04203000-7c9e08000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.027534+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e0c000000-7c9e0c209000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.027591+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e0c209000-7c9e10000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.027617+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e10000000-7c9e10203000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.027640+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e10203000-7c9e14000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.027666+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e18000000-7c9e1816b000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.027693+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e1816b000-7c9e1c000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.027716+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e1c000000-7c9e1c174000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.027742+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e1c174000-7c9e20000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.027769+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e24000000-7c9e24173000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.027795+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e24173000-7c9e28000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.027814+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e28000000-7c9e28203000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.027869+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e28203000-7c9e2c000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.027896+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e30000000-7c9e30204000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.027918+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e30204000-7c9e34000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.027944+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e34000000-7c9e34145000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.027967+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e34145000-7c9e38000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.027993+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e3c000000-7c9e3c204000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.028020+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e3c204000-7c9e40000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.028046+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e40000000-7c9e40174000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.028068+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e40174000-7c9e44000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.028090+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e48000000-7c9e48113000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.028111+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e48113000-7c9e4c000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.028138+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e4c000000-7c9e4c17e000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.028165+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e4c17e000-7c9e50000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.030279+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e54000000-7c9e5416c000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.030350+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e5416c000-7c9e58000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.030375+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e58000000-7c9e58164000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.030402+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e58164000-7c9e5c000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.030423+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e60000000-7c9e6016d000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.030468+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e6016d000-7c9e64000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.030491+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e64000000-7c9e64203000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.030514+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e64203000-7c9e68000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.030541+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e6c000000-7c9e6c205000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.030571+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e6c205000-7c9e70000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.030592+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e70000000-7c9e70172000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.030627+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e70172000-7c9e74000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.030658+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e78000000-7c9e78113000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.030693+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e78113000-7c9e7c000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.030715+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e7c000000-7c9e7c171000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.030738+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e7c171000-7c9e80000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.030765+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e84000000-7c9e84205000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.030786+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e84205000-7c9e88000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.030847+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e88000000-7c9e88203000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.030878+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e88203000-7c9e8c000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.030899+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e90000000-7c9e90203000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.030926+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e90203000-7c9e94000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.030947+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e94000000-7c9e94204000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.030969+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e94204000-7c9e98000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.030996+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e9c000000-7c9e9c235000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.031019+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9e9c235000-7c9ea0000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.031041+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ea0000000-7c9ea0180000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.031066+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ea0180000-7c9ea4000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.031097+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ea8000000-7c9ea817c000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.031124+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ea817c000-7c9eac000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.031150+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9eac000000-7c9eac179000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.031172+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9eac179000-7c9eb0000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.031207+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9eb4000000-7c9eb4179000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.031231+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9eb4179000-7c9eb8000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.031253+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9eb8000000-7c9eb8061000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.031278+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9eb8061000-7c9ebc000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.031300+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ec0000000-7c9ec0236000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.031338+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ec0236000-7c9ec4000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.031360+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ec4000000-7c9ec4179000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.031390+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ec4179000-7c9ec8000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.031423+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ecc000000-7c9ecc16b000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.031454+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ecc16b000-7c9ed0000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.031482+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ed0000000-7c9ed016b000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.031505+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ed016b000-7c9ed4000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.031531+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ed7600000-7c9ed7601000 ---p 00000000 00:00 0 2024-06-11T19:38:46.031558+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ed7601000-7c9ed7e01000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.031580+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ed8000000-7c9ed8203000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.031602+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ed8203000-7c9edc000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.031624+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9edc000000-7c9edc17b000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.031646+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9edc17b000-7c9ee0000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.031676+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ee0400000-7c9ee0401000 ---p 00000000 00:00 0 2024-06-11T19:38:46.031700+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ee0401000-7c9ee0c01000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.031721+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ee0e00000-7c9ee0e01000 ---p 00000000 00:00 0 2024-06-11T19:38:46.031747+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ee0e01000-7c9ee1601000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.031768+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ee1800000-7c9ee1801000 ---p 00000000 00:00 0 2024-06-11T19:38:46.031790+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ee1801000-7c9ee2001000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.031816+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ee2200000-7c9ee2201000 ---p 00000000 00:00 0 2024-06-11T19:38:46.031838+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ee2201000-7c9ee2a01000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.031860+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ee2c00000-7c9ee2c01000 ---p 00000000 00:00 0 2024-06-11T19:38:46.031901+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ee2c01000-7c9ee3401000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.031933+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ee3600000-7c9ee3601000 ---p 00000000 00:00 0 2024-06-11T19:38:46.031959+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ee3601000-7c9ee3e01000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.031985+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ee4000000-7c9ee4177000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.032004+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ee4177000-7c9ee8000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.032026+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ee8000000-7c9ee8203000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.032052+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ee8203000-7c9eec000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.032082+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9eed800000-7c9eed801000 ---p 00000000 00:00 0 2024-06-11T19:38:46.032109+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9eed801000-7c9eee001000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.032135+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9eee200000-7c9eee201000 ---p 00000000 00:00 0 2024-06-11T19:38:46.032157+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9eee201000-7c9eeea01000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.032198+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9eeec00000-7c9eeec01000 ---p 00000000 00:00 0 2024-06-11T19:38:46.032225+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9eeec01000-7c9eef401000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.032249+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9eef600000-7c9eef601000 ---p 00000000 00:00 0 2024-06-11T19:38:46.032280+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9eef601000-7c9eefe01000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.032326+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ef0000000-7c9ef0172000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.032352+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ef0172000-7c9ef4000000 ---p 00000000 00:00 0 2024-06-11T19:38:46.032374+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ef4c00000-7c9ef4c01000 ---p 00000000 00:00 0 2024-06-11T19:38:46.032397+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ef4c01000-7c9ef5401000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.032423+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ef5600000-7c9ef5601000 ---p 00000000 00:00 0 2024-06-11T19:38:46.032450+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ef5601000-7c9ef5e01000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.032473+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ef6000000-7c9ef6001000 ---p 00000000 00:00 0 2024-06-11T19:38:46.032495+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ef6001000-7c9ef6801000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.032522+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ef699e000-7c9ef6a00000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.032578+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ef6a00000-7c9ef6a01000 ---p 00000000 00:00 0 2024-06-11T19:38:46.032601+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ef6a01000-7c9ef7201000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.032623+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ef7201000-7c9ef721c000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.032645+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ef721c000-7c9ef724f000 r-xp 00000000 07:02 155 /snap/lxd/28463/lib/liblxcfs.so 2024-06-11T19:38:46.032669+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ef724f000-7c9ef7250000 r--p 00032000 07:02 155 /snap/lxd/28463/lib/liblxcfs.so 2024-06-11T19:38:46.032694+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ef7250000-7c9ef7251000 rw-p 00033000 07:02 155 /snap/lxd/28463/lib/liblxcfs.so 2024-06-11T19:38:46.032716+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ef7251000-7c9ef725f000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.032738+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ef725f000-7c9ef7287000 r--p 00000000 07:00 7471 /usr/lib/x86_64-linux-gnu/libc.so.6 2024-06-11T19:38:46.032766+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ef7287000-7c9ef741c000 r-xp 00028000 07:00 7471 /usr/lib/x86_64-linux-gnu/libc.so.6 2024-06-11T19:38:46.032787+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ef741c000-7c9ef7474000 r--p 001bd000 07:00 7471 /usr/lib/x86_64-linux-gnu/libc.so.6 2024-06-11T19:38:46.032831+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ef7474000-7c9ef7475000 ---p 00215000 07:00 7471 /usr/lib/x86_64-linux-gnu/libc.so.6 2024-06-11T19:38:46.032858+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ef7475000-7c9ef7479000 r--p 00215000 07:00 7471 /usr/lib/x86_64-linux-gnu/libc.so.6 2024-06-11T19:38:46.032884+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ef7479000-7c9ef747b000 rw-p 00219000 07:00 7471 /usr/lib/x86_64-linux-gnu/libc.so.6 2024-06-11T19:38:46.032907+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ef747b000-7c9ef7488000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.032933+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ef7488000-7c9ef748b000 r--p 00000000 07:00 7521 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 2024-06-11T19:38:46.032957+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ef748b000-7c9ef74a2000 r-xp 00003000 07:00 7521 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 2024-06-11T19:38:46.032984+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ef74a2000-7c9ef74a6000 r--p 0001a000 07:00 7521 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 2024-06-11T19:38:46.033006+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ef74a6000-7c9ef74a7000 r--p 0001d000 07:00 7521 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 2024-06-11T19:38:46.033033+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ef74a7000-7c9ef74a8000 rw-p 0001e000 07:00 7521 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 2024-06-11T19:38:46.033057+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ef74a8000-7c9ef74ab000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.033079+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ef74ab000-7c9ef74b2000 r--p 00000000 07:02 565 /snap/lxd/28463/lib/x86_64-linux-gnu/libfuse3.so.3.10.5 2024-06-11T19:38:46.033109+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ef74b2000-7c9ef74cd000 r-xp 00007000 07:02 565 /snap/lxd/28463/lib/x86_64-linux-gnu/libfuse3.so.3.10.5 2024-06-11T19:38:46.033132+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ef74cd000-7c9ef74d7000 r--p 00022000 07:02 565 /snap/lxd/28463/lib/x86_64-linux-gnu/libfuse3.so.3.10.5 2024-06-11T19:38:46.033154+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ef74d7000-7c9ef74e9000 r--p 0002b000 07:02 565 /snap/lxd/28463/lib/x86_64-linux-gnu/libfuse3.so.3.10.5 2024-06-11T19:38:46.033188+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ef74e9000-7c9ef74ea000 rw-p 0003d000 07:02 565 /snap/lxd/28463/lib/x86_64-linux-gnu/libfuse3.so.3.10.5 2024-06-11T19:38:46.033217+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ef74ea000-7c9ef74eb000 r--p 00000000 07:02 538 /snap/lxd/28463/lib/x86_64-linux-gnu/libSegFault.so 2024-06-11T19:38:46.033241+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ef74eb000-7c9ef74ee000 r-xp 00001000 07:02 538 /snap/lxd/28463/lib/x86_64-linux-gnu/libSegFault.so 2024-06-11T19:38:46.033263+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ef74ee000-7c9ef74ef000 r--p 00004000 07:02 538 /snap/lxd/28463/lib/x86_64-linux-gnu/libSegFault.so 2024-06-11T19:38:46.033317+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ef74ef000-7c9ef74f0000 r--p 00005000 07:02 538 /snap/lxd/28463/lib/x86_64-linux-gnu/libSegFault.so 2024-06-11T19:38:46.033347+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ef74f0000-7c9ef74f1000 rw-p 00006000 07:02 538 /snap/lxd/28463/lib/x86_64-linux-gnu/libSegFault.so 2024-06-11T19:38:46.033370+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ef74f1000-7c9ef74f3000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.033396+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ef74f3000-7c9ef74f5000 r--p 00000000 07:00 7442 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 2024-06-11T19:38:46.033418+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ef74f5000-7c9ef751f000 r-xp 00002000 07:00 7442 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 2024-06-11T19:38:46.033441+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ef751f000-7c9ef752a000 r--p 0002c000 07:00 7442 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 2024-06-11T19:38:46.033463+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ef752a000-7c9ef752b000 rw-p 00000000 00:00 0 2024-06-11T19:38:46.033484+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ef752b000-7c9ef752d000 r--p 00037000 07:00 7442 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 2024-06-11T19:38:46.033504+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7c9ef752d000-7c9ef752f000 rw-p 00039000 07:00 7442 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 2024-06-11T19:38:46.033526+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7ffe895f7000-7ffe89618000 rw-p 00000000 00:00 0 [stack] 2024-06-11T19:38:46.033549+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7ffe89658000-7ffe8965c000 r--p 00000000 00:00 0 [vvar] 2024-06-11T19:38:46.033571+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: 7ffe8965c000-7ffe8965e000 r-xp 00000000 00:00 0 [vdso] 2024-06-11T19:38:46.033593+02:00 lxdvm2-x14-u16-r1 lxd.daemon[3114]: ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0 [vsyscall] ```
webdock-io commented 3 weeks ago

@mihalicyn Here is another case for you. I dont want to be reporting the same crash over and over, but I'm reporting this one as it's different somehow. So, we have "detection" of crashes which just consists of us checking if the lxcfs process is running with pgrep lxcfs. Now, we were made aware by a customer something was wrong just now, and checking we see lxcfs is not functioning properly and we do see the below crash in the syslog. However, we not only see lxcfs running in our process list, but we have two instances running. I have no idea if that is something that is possible/expected to happen and I find it strange lxcfs hasn't completely died/gone away like with the other crashes we've seen. This is what ps aux shows:

$ ps aux | grep lxcfs
root     2142417  0.0  0.0   7076  2016 pts/1    S+   13:52   0:00 grep --color=auto lxcfs
root     2599964  0.2  0.0 451664  2304 ?        Sl   Jun11   3:43 lxcfs --enable-loadavg /var/snap/lxd/common/var/lib/lxcfs -p /var/snap/lxd/common/lxcfs.pid
root     2733049  0.6  0.0 675464  2592 ?        Sl   Jun11   8:48 lxcfs --enable-loadavg /var/snap/lxd/common/var/lib/lxcfs -p /var/snap/lxd/common/lxcfs.pid

In my mind, very odd behavior in this case. We checked on further hosts, and we actually have one more host which has two instances of lxcfs running, but there it seems OK and doing its job. We are not sure what to make of this. Here is the relevant crash dump:

Crash data ``` 2024-06-12T05:53:38.366230+02:00 lxdvm1-x14-u15-r1 kernel: tasks_rcu_exit_srcu_stall: rcu_tasks grace period number 669 (since boot) gp_state: RTGS_POST_SCAN_TASKLIST is 21156334 jiffies old. 2024-06-12T05:53:38.366269+02:00 lxdvm1-x14-u15-r1 kernel: Please check any exiting tasks stuck between calls to exit_tasks_rcu_start() and exit_tasks_rcu_finish() 2024-06-12T05:53:38.475974+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: *** signal 11 2024-06-12T05:53:38.476325+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: Register dump: 2024-06-12T05:53:38.476400+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: RAX: 0000788497de0300 RBX: 0000000000000000 RCX: 0000000000000100 2024-06-12T05:53:38.476464+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: RDX: 0000000000001687 RSI: 0000000000000100 RDI: 00007884833ff540 2024-06-12T05:53:38.476530+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: RBP: 0000000000000100 R8 : 0000000000000000 R9 : 00007884833ff630 2024-06-12T05:53:38.476605+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: R10: 0000788497d87ac0 R11: 0000788497d883c0 R12: 0000000000001687 2024-06-12T05:53:38.476679+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: R13: 00007884833ff540 R14: 0000788497bb2030 R15: 00007884833ff6c0 2024-06-12T05:53:38.476739+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: RSP: 00007884833ff530 2024-06-12T05:53:38.476799+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: RIP: 0000788497c51481 EFLAGS: 00010202 2024-06-12T05:53:38.476859+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: CS: 0033 FS: 0000 GS: 0000 2024-06-12T05:53:38.476926+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: Trap: 0000000e Error: 00000006 OldMask: 00004007 CR2: 00000100 2024-06-12T05:53:38.476989+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: FPUCW: 0000037f FPUSW: 00000000 TAG: 00000000 2024-06-12T05:53:38.477070+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: RIP: 00000000 RDP: 00000000 2024-06-12T05:53:38.477139+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: ST(0) 0000 0000000000000000 ST(1) 0000 0000000000000000 2024-06-12T05:53:38.477199+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: ST(2) 0000 0000000000000000 ST(3) 0000 0000000000000000 2024-06-12T05:53:38.477267+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: ST(4) 0000 0000000000000000 ST(5) 0000 0000000000000000 2024-06-12T05:53:38.477320+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: ST(6) 0000 0000000000000000 ST(7) 0000 0000000000000000 2024-06-12T05:53:38.477370+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: mxcsr: 1fa0 2024-06-12T05:53:38.477422+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: XMM0: 00000000000000000000000000000000 XMM1: 00000000000000000000000000000000 2024-06-12T05:53:38.477474+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: XMM2: 00000000000000000000000000000000 XMM3: 00000000000000000000000000000000 2024-06-12T05:53:38.477537+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: XMM4: 00000000000000000000000000000000 XMM5: 00000000000000000000000000000000 2024-06-12T05:53:38.477588+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: XMM6: 00000000000000000000000000000000 XMM7: 00000000000000000000000000000000 2024-06-12T05:53:38.477639+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: XMM8: 00000000000000000000000000000000 XMM9: 00000000000000000000000000000000 2024-06-12T05:53:38.477711+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: XMM10: 00000000000000000000000000000000 XMM11: 00000000000000000000000000000000 2024-06-12T05:53:38.477769+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: XMM12: 00000000000000000000000000000000 XMM13: 00000000000000000000000000000000 2024-06-12T05:53:38.477821+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: XMM14: 00000000000000000000000000000000 XMM15: 00000000000000000000000000000000 2024-06-12T05:53:38.477882+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: Backtrace: 2024-06-12T05:53:38.478000+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: /lib/x86_64-linux-gnu/libc.so.6(+0x88481)[0x788497c51481] 2024-06-12T05:53:38.478097+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: /lib/x86_64-linux-gnu/libc.so.6(snprintf+0x96)[0x788497c29856] 2024-06-12T05:53:38.478156+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: /snap/lxd/current/lib/liblxcfs.so(+0x1facc)[0x788497ba5acc] 2024-06-12T05:53:38.478219+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: /snap/lxd/current/lib/liblxcfs.so(proc_read+0x17d)[0x788497ba8478] 2024-06-12T05:53:38.478297+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: lxcfs(+0x22c3)[0x615ab2cfd2c3] 2024-06-12T05:53:38.478357+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: lxcfs(+0x3a56)[0x615ab2cfea56] 2024-06-12T05:53:38.478437+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: /snap/lxd/current/lib/x86_64-linux-gnu/libfuse3.so.3(fuse_fs_read_buf+0x1be)[0x788497e22f2e] 2024-06-12T05:53:38.478497+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: /snap/lxd/current/lib/x86_64-linux-gnu/libfuse3.so.3(+0x10d66)[0x788497e25d66] 2024-06-12T05:53:38.478557+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: /snap/lxd/current/lib/x86_64-linux-gnu/libfuse3.so.3(+0x114fa)[0x788497e264fa] 2024-06-12T05:53:38.478616+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: /snap/lxd/current/lib/x86_64-linux-gnu/libfuse3.so.3(+0x1e13f)[0x788497e3313f] 2024-06-12T05:53:38.478668+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: /snap/lxd/current/lib/x86_64-linux-gnu/libfuse3.so.3(+0x167a7)[0x788497e2b7a7] 2024-06-12T05:53:38.478720+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: /lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)[0x788497c5dac3] 2024-06-12T05:53:38.478790+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: /lib/x86_64-linux-gnu/libc.so.6(+0x126850)[0x788497cef850] 2024-06-12T05:53:38.478860+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: Memory map: 2024-06-12T05:53:38.478920+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 615ab2cfb000-615ab2d03000 r-xp 00000000 07:02 44 /snap/lxd/28463/bin/lxcfs 2024-06-12T05:53:38.478974+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 615ab2d03000-615ab2d04000 r--p 00007000 07:02 44 /snap/lxd/28463/bin/lxcfs 2024-06-12T05:53:38.479057+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 615ab2d04000-615ab2d05000 rw-p 00008000 07:02 44 /snap/lxd/28463/bin/lxcfs 2024-06-12T05:53:38.479157+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 615ab4b83000-615ab4bd2000 rw-p 00000000 00:00 0 [heap] 2024-06-12T05:53:38.479245+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 78835c000000-78835c185000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.479321+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 78835c185000-788360000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.479407+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788364000000-788364182000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.479481+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788364182000-788368000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.479536+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788368000000-788368188000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.479601+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788368188000-78836c000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.479657+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788370000000-788370189000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.479722+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788370189000-788374000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.479800+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788374000000-788374112000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.479874+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788374112000-788378000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.479947+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 78837c000000-78837c184000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.480022+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 78837c184000-788380000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.480109+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788380000000-78838017e000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.480172+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 78838017e000-788384000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.480228+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788388000000-788388112000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.480300+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788388112000-78838c000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.480392+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 78838c000000-78838c203000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.480460+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 78838c203000-788390000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.480538+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788394000000-788394184000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.480949+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788394184000-788398000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.481084+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788398000000-788398177000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.481193+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788398177000-78839c000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.481369+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 7883a0000000-7883a0203000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.481452+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 7883a0203000-7883a4000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.481529+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 7883a4000000-7883a4203000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.481621+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 7883a4203000-7883a8000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.481679+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 7883ac000000-7883ac180000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.481753+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 7883ac180000-7883b0000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.481818+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 7883b0000000-7883b0184000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.481882+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 7883b0184000-7883b4000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.481947+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 7883b8000000-7883b816a000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.482011+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 7883b816a000-7883bc000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.482110+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 7883bc000000-7883bc17e000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.482184+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 7883bc17e000-7883c0000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.482269+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 7883c4000000-7883c4176000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.482337+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 7883c4176000-7883c8000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.482403+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 7883c8000000-7883c8183000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.482477+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 7883c8183000-7883cc000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.482551+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 7883d0000000-7883d0183000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.482607+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 7883d0183000-7883d4000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.482677+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 7883d4000000-7883d418a000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.482724+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 7883d418a000-7883d8000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.482780+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 7883dc000000-7883dc203000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.482837+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 7883dc203000-7883e0000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.482883+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 7883e0000000-7883e018b000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.482949+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 7883e018b000-7883e4000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.483024+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 7883e8000000-7883e8191000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.483111+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 7883e8191000-7883ec000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.483184+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 7883ec000000-7883ec16d000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.483267+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 7883ec16d000-7883f0000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.483367+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 7883f4000000-7883f4187000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.483432+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 7883f4187000-7883f8000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.483496+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 7883f8000000-7883f8175000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.483561+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 7883f8175000-7883fc000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.483650+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788400000000-788400187000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.483719+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788400187000-788404000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.483805+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788404000000-788404188000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.483913+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788404188000-788408000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.483990+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 78840c000000-78840c16e000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.484075+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 78840c16e000-788410000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.484147+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788410000000-78841018b000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.484212+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 78841018b000-788414000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.484297+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788418000000-78841816d000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.484363+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 78841816d000-78841c000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.484428+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 78841c000000-78841c189000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.484492+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 78841c189000-788420000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.484556+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788424000000-788424174000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.484629+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788424174000-788428000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.484686+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788428000000-78842818a000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.484750+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 78842818a000-78842c000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.484815+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788430000000-788430182000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.484879+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788430182000-788434000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.485513+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788434000000-788434203000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.485611+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788434203000-788438000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.485659+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 78843c000000-78843c18a000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.485728+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 78843c18a000-788440000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.485805+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788440000000-788440187000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.485892+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788440187000-788444000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.485957+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788448000000-78844817b000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.486021+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 78844817b000-78844c000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.486115+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 78844c000000-78844c186000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.486187+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 78844c186000-788450000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.486273+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788454000000-788454183000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.486339+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788454183000-788458000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.486404+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788458000000-78845818c000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.486459+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 78845818c000-78845c000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.486524+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788460000000-78846017c000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.486579+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 78846017c000-788464000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.486634+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788464000000-788464178000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.486700+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788464178000-788468000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.486769+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 78846b600000-78846b601000 ---p 00000000 00:00 0 2024-06-12T05:53:38.486834+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 78846b601000-78846be01000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.486903+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 78846c000000-78846c198000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.486958+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 78846c198000-788470000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.487014+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788470000000-78847018c000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.487095+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 78847018c000-788474000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.487169+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788476200000-788476201000 ---p 00000000 00:00 0 2024-06-12T05:53:38.487254+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788476201000-788476a01000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.487363+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788478000000-78847818a000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.487442+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 78847818a000-78847c000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.487508+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 78847c000000-78847c068000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.487583+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 78847c068000-788480000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.487639+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788481800000-788481801000 ---p 00000000 00:00 0 2024-06-12T05:53:38.487695+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788481801000-788482001000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.487761+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788482200000-788482201000 ---p 00000000 00:00 0 2024-06-12T05:53:38.487817+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788482201000-788482a01000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.487888+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788482c00000-788482c01000 ---p 00000000 00:00 0 2024-06-12T05:53:38.487936+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788482c01000-788483401000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.488001+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788483600000-788483601000 ---p 00000000 00:00 0 2024-06-12T05:53:38.488089+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788483601000-788483e01000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.488475+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788484000000-78848418b000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.488564+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 78848418b000-788488000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.488642+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788488000000-788488174000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.488700+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788488174000-78848c000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.488761+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 78848c400000-78848c401000 ---p 00000000 00:00 0 2024-06-12T05:53:38.488826+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 78848c401000-78848cc01000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.488891+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 78848ce00000-78848ce01000 ---p 00000000 00:00 0 2024-06-12T05:53:38.488956+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 78848ce01000-78848d601000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.489000+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 78848e200000-78848e201000 ---p 00000000 00:00 0 2024-06-12T05:53:38.489026+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 78848e201000-78848ea01000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.489049+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 78848ec00000-78848ec01000 ---p 00000000 00:00 0 2024-06-12T05:53:38.489097+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 78848ec01000-78848f401000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.489125+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 78848f600000-78848f601000 ---p 00000000 00:00 0 2024-06-12T05:53:38.489152+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 78848f601000-78848fe01000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.489178+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788490000000-78849018b000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.489201+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 78849018b000-788494000000 ---p 00000000 00:00 0 2024-06-12T05:53:38.489224+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788494a00000-788494a01000 ---p 00000000 00:00 0 2024-06-12T05:53:38.489264+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788494a01000-788495201000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.489284+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788495400000-788495401000 ---p 00000000 00:00 0 2024-06-12T05:53:38.489311+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788495401000-788495c01000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.489337+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788496800000-788496801000 ---p 00000000 00:00 0 2024-06-12T05:53:38.489364+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788496801000-788497001000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.489395+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788497200000-788497201000 ---p 00000000 00:00 0 2024-06-12T05:53:38.489422+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788497201000-788497a01000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.489448+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788497a82000-788497a84000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.489468+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788497b0b000-788497b86000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.489500+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788497b86000-788497bb9000 r-xp 00000000 07:02 155 /snap/lxd/28463/lib/liblxcfs.so 2024-06-12T05:53:38.489547+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788497bb9000-788497bba000 r--p 00032000 07:02 155 /snap/lxd/28463/lib/liblxcfs.so 2024-06-12T05:53:38.489575+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788497bba000-788497bbb000 rw-p 00033000 07:02 155 /snap/lxd/28463/lib/liblxcfs.so 2024-06-12T05:53:38.489595+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788497bbb000-788497bc9000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.489614+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788497bc9000-788497bf1000 r--p 00000000 07:00 7471 /usr/lib/x86_64-linux-gnu/libc.so.6 2024-06-12T05:53:38.489637+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788497bf1000-788497d86000 r-xp 00028000 07:00 7471 /usr/lib/x86_64-linux-gnu/libc.so.6 2024-06-12T05:53:38.489664+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788497d86000-788497dde000 r--p 001bd000 07:00 7471 /usr/lib/x86_64-linux-gnu/libc.so.6 2024-06-12T05:53:38.489691+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788497dde000-788497ddf000 ---p 00215000 07:00 7471 /usr/lib/x86_64-linux-gnu/libc.so.6 2024-06-12T05:53:38.489718+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788497ddf000-788497de3000 r--p 00215000 07:00 7471 /usr/lib/x86_64-linux-gnu/libc.so.6 2024-06-12T05:53:38.489745+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788497de3000-788497de5000 rw-p 00219000 07:00 7471 /usr/lib/x86_64-linux-gnu/libc.so.6 2024-06-12T05:53:38.489772+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788497de5000-788497df2000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.489804+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788497df2000-788497df5000 r--p 00000000 07:00 7521 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 2024-06-12T05:53:38.489841+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788497df5000-788497e0c000 r-xp 00003000 07:00 7521 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 2024-06-12T05:53:38.489869+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788497e0c000-788497e10000 r--p 0001a000 07:00 7521 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 2024-06-12T05:53:38.489896+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788497e10000-788497e11000 r--p 0001d000 07:00 7521 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 2024-06-12T05:53:38.489919+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788497e11000-788497e12000 rw-p 0001e000 07:00 7521 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1 2024-06-12T05:53:38.489942+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788497e12000-788497e15000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.489968+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788497e15000-788497e1c000 r--p 00000000 07:02 565 /snap/lxd/28463/lib/x86_64-linux-gnu/libfuse3.so.3.10.5 2024-06-12T05:53:38.489996+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788497e1c000-788497e37000 r-xp 00007000 07:02 565 /snap/lxd/28463/lib/x86_64-linux-gnu/libfuse3.so.3.10.5 2024-06-12T05:53:38.490019+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788497e37000-788497e41000 r--p 00022000 07:02 565 /snap/lxd/28463/lib/x86_64-linux-gnu/libfuse3.so.3.10.5 2024-06-12T05:53:38.490076+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788497e41000-788497e53000 r--p 0002b000 07:02 565 /snap/lxd/28463/lib/x86_64-linux-gnu/libfuse3.so.3.10.5 2024-06-12T05:53:38.490108+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788497e53000-788497e54000 rw-p 0003d000 07:02 565 /snap/lxd/28463/lib/x86_64-linux-gnu/libfuse3.so.3.10.5 2024-06-12T05:53:38.490136+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788497e54000-788497e55000 r--p 00000000 07:02 538 /snap/lxd/28463/lib/x86_64-linux-gnu/libSegFault.so 2024-06-12T05:53:38.490163+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788497e55000-788497e58000 r-xp 00001000 07:02 538 /snap/lxd/28463/lib/x86_64-linux-gnu/libSegFault.so 2024-06-12T05:53:38.490273+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788497e58000-788497e59000 r--p 00004000 07:02 538 /snap/lxd/28463/lib/x86_64-linux-gnu/libSegFault.so 2024-06-12T05:53:38.490301+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788497e59000-788497e5a000 r--p 00005000 07:02 538 /snap/lxd/28463/lib/x86_64-linux-gnu/libSegFault.so 2024-06-12T05:53:38.490333+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788497e5a000-788497e5b000 rw-p 00006000 07:02 538 /snap/lxd/28463/lib/x86_64-linux-gnu/libSegFault.so 2024-06-12T05:53:38.490357+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788497e5b000-788497e5d000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.490384+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788497e5d000-788497e5f000 r--p 00000000 07:00 7442 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 2024-06-12T05:53:38.490411+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788497e5f000-788497e89000 r-xp 00002000 07:00 7442 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 2024-06-12T05:53:38.490438+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788497e89000-788497e94000 r--p 0002c000 07:00 7442 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 2024-06-12T05:53:38.490465+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788497e94000-788497e95000 rw-p 00000000 00:00 0 2024-06-12T05:53:38.490485+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788497e95000-788497e97000 r--p 00037000 07:00 7442 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 2024-06-12T05:53:38.490508+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 788497e97000-788497e99000 rw-p 00039000 07:00 7442 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2 2024-06-12T05:53:38.490540+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 7ffe743d3000-7ffe743f4000 rw-p 00000000 00:00 0 [stack] 2024-06-12T05:53:38.490571+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 7ffe743f5000-7ffe743f9000 r--p 00000000 00:00 0 [vvar] 2024-06-12T05:53:38.490599+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: 7ffe743f9000-7ffe743fb000 r-xp 00000000 00:00 0 [vdso] 2024-06-12T05:53:38.490623+02:00 lxdvm1-x14-u15-r1 lxd.daemon[2901]: ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0 [vsyscall] ```
webdock-io commented 3 weeks ago

One last addendum here @mihalicyn we are also the guys running everything under cgroupv1, because of this: https://github.com/lxc/lxcfs/issues/538

I dont know if that has any bearing on the issue, but we thought it was relevant to mention this.

Edit: We can give you access to a test environment if you want to inspect some things or test something out. Just send us your public key at support@webdock.io and we will give you access. Thanks very much for your time. This issue is unfortunately becoming rather painful for us, so we really hope you can find time to check this further if you can. Fingers crossed :)

webdock-io commented 3 weeks ago

@webdock-io thanks for providing an extra context on this ;-)

Can you try to collect a crash dump from the LXCFS?

All you need to do is (as root):

echo '|/bin/sh -c $@ -- eval exec cat > /var/crash/core-%e.%p' > /proc/sys/kernel/core_pattern

and wait until a next crash. Then collect a crashdump file in /var/crash/ directory and share it with me. (you can GPG encrypt it if you want with my key gpg --encrypt your_core_dump --recipient alexander@mihalicyn.com)

@mihalicyn is there a way where we can limit core dumps to disk to only happen if lxcfs crashes and not other stuff? If so we would be able to enable this infrastructure-wide and catch this for sure... We tried looking around for info on this, but it's unclear how to achieve this exactly..if you know how, that would help

webdock-io commented 3 weeks ago

So... In order to try and procure a crash dump for you, we have dedicated an instance to this purpose where we are stressing it a lot. We hope this will provoke a crash sometime in the next 24-48 hours - if not, then I'm not sure what we can do as then it seems synthetic workloads will not do the trick... We will see.

We have now absolutely hammered a system for more than 24 hours with high load on cpu, memory, io and sirq. We nerfed our Arc cache and had 3 container instances doing a lot of activity. And.... Nothing. lxcfs runs fine, no crash of our VM

So, we are unable to reproduce this synthetically it seems. Maybe we just need a lot more containers... An idea could be to spin up 100+ containers and have them all do... something.

@mihalicyn Maybe to help us get a reproducer here, can you tell us if the crashing codepath relates to some specific virtualization in lxcfs? For example, if the code is tied to CPU or memory or whichever specific subsystem? Because then we could focus on creating diverse workloads against just that system, instead of hammering everything all at once.

Anyway, we await your feedback here :)

mihalicyn commented 3 weeks ago

Hi, @webdock-io

@mihalicyn is there a way where we can limit core dumps to disk to only happen if lxcfs crashes and not other stuff? If so we would be able to enable this infrastructure-wide and catch this for sure... We tried looking around for info on this, but it's unclear how to achieve this exactly..if you know how, that would help

You can try something like that:

  1. Create a /root/core_dump.sh file with the following content: 
    
# cat /root/core_dump.sh
#!/bin/sh

PROC_NAME=$1 PROC_PID=$2

[ "${PROC_NAME}" != "lxcfs" ] && exit 0

cat /dev/stdin | gzip --fast > /var/crash/core-${PROC_NAME}.${PROC_PID}.gz


2. `chmod +x /root/core_dump.sh`

3.

save an old core_pattern setting value:

cat /proc/sys/kernel/core_pattern > /root/core_pattern.old_value.bak

set the new core dump handler

echo '|/root/core_dump.sh %e %p' > /proc/sys/kernel/core_pattern

mihalicyn commented 3 weeks ago

Edit: We can give you access to a test environment if you want to inspect some things or test something out. Just send us your public key at support@webdock.io and we will give you access. Thanks very much for your time. This issue is unfortunately becoming rather painful for us, so we really hope you can find time to check this further if you can. Fingers crossed :)

This is a good option. Let's try to get some core dump with the method I've described above (with filtering) and if it doesn't work then giving me an access would be a last hope weapon.

mihalicyn commented 3 weeks ago

One last addendum here @mihalicyn we are also the guys running everything under cgroupv1, because of this: #538

Ugh, that's painful. Because at some point cgroupv1 become deprecated in upstream, or even earlier, in systemd and you will have no chance to stay on it. I believe we should sort out this blocker that prevents you from the migration to cgroup-v2. It's extremely important. Let's discuss this in #538

mihalicyn commented 3 weeks ago

@mihalicyn Maybe to help us get a reproducer here, can you tell us if the crashing codepath relates to some specific virtualization in lxcfs? For example, if the code is tied to CPU or memory or whichever specific subsystem? Because then we could focus on creating diverse workloads against just that system, instead of hammering everything all at once.

That's a good question. Actually, I tend to believe that crash happens not because of an issue in LXCFS, but likely it's a stack corruption somewhere in libfuse or even a kernel problem. And to investigate this deeper I need to have a memory dump of the LXCFS process (ideally, a few different ones).

mihalicyn commented 3 weeks ago

@webdock-io I have edited your messages a bit and put all the crash data content inside a special markup:

<details><summary>Crash data</summary>
...
</details>

this makes this issue thread a bit more readable for others :)

webdock-io commented 3 weeks ago

@mihalicyn Thank you for the markup tip

Ugh, that's painful. Because at some point cgroupv1 become deprecated in upstream, or even earlier, in systemd and you will have no chance to stay on it. I believe we should sort out this blocker that prevents you from the migration to cgroup-v2. It's extremely important. Let's discuss this in https://github.com/lxc/lxcfs/issues/538

I am very happy you see this as important, because it is! We were starting to have the discussion internally whether we should migrate over to KVM entirely and completely drop container VPS support, as this issue would be a complete blocker for proper container support in the near-future. Now that we know that this issue is being worked on, we will hold off on pressing the panic button anytime soon :)

We are unfortunately still seeing occasional crashes as you know, and had one last night. Unfortunately we had not had the time to implement your new crash dump code. This will be done today on all our hosts, and I am confident we will have one or more crash dumps for you within the next few days.

Thank you for your continued assistance and we will revert here once we have more material for you.

webdock-io commented 2 weeks ago

@mihalicyn We have a crash dump for you! However...

you can GPG encrypt it if you want with my key gpg --encrypt your_core_dump --recipient alexander@mihalicyn.com

This command looks a bit wrong and I tried a few things but I am unable to encrypt the dump for you. It looks to me you expect me to find your pubkey somewhere? I really tried my best reading the man page for gpg and looking around to see if I could find your pubkey somewhere but I am unable. Please provide more idiot-proof steps for me here, and I'll get this encrypted and placed somewhere you can grab it asap :)

Thanks!

webdock-io commented 2 weeks ago

@mihalicyn After much much faffing around I finally found your gpg key at keyserver.ubuntu.com and imported it with gpg --keyserver keyserver.ubuntu.com --recv-keys 0x801a2056d5cf7ecf6679a79bf9fe8fed56c36d56

Anyway, here is the encrypted crash dump attached. I'll post further crash dumps as they come in :) I really hope this helps you find the issue! core-lxcfs.1499342.gpg.gz

Edit: Github would not accept .gpg exension for uploads, so I just renamed the file and reversed the extension so github would eat it. You should rename it to .gz.gpg before decrypting and then decompressing

webdock-io commented 2 weeks ago

@mihalicyn And here is new one from another host

core-lxcfs.1147289.gpg.gz

mihalicyn commented 2 weeks ago

Hi @webdock-io !

After much much faffing around I finally found your gpg key at keyserver.ubuntu.com and imported it with

Sorry about not providing you with a more detailed instruction. It was an optional step to encrypt it, but you did it right!

Thanks for providing me with your coredumps, I'm on it to investigate this issue.

webdock-io commented 4 days ago

Hello @mihalicyn

We now have two more crash dumps for you:

core-lxcfs.2897.gpg.gz core-lxcfs.3396.gpg.gz

Any progress on this issue so far?

capriciousduck commented 2 days ago

@webdock-io @mihalicyn

I'm facing this too I think as I noticed all the 4 cores of my host visible in one of my containers but for some reason I restarted my host and all looks good now. I too placed the logging mechanism, will let you guys know if I see a crash again. And let me know if you guys need any information from me.

Thanks.

mihalicyn commented 1 day ago

Hi @webdock-io

I posted analysis (for those who interested in these details) at the end of this message.

First of all, thanks for helping us and providing core dumps. This is crucial.

I think that we need to go even further with this and provide you with some special debug builds of LXCFS which can allow us to catch the issue. I'll think about a proper way to do so, of course, I can't just provide you with the binary and ask you to run it. It's just not how we do things. :) Likely, I'll prepare some special branch of LXCFS and instructions how to build it on your infrastructure and deploy.

Also, please, can you tell me if you have any special or unusual software in containers on your infrastructure that can affect LXCFS? May be, you are using some monitoring system agent like node_exporter or something like that. Because you have a stable reproducer, if I would have such a stable reproducer things would be much-much easier to solve.

Analysis of core-lxcfs.1147289 ``` (gdb) p *(struct fuse*)((struct fuse_session*)0x5c47d1218a30)->userdata $17 = {se = 0x5c47d1218a30, name_table = {array = 0x5c47d121a850, use = 3334, size = 8192, split = 0}, id_table = { array = 0x5c47d122a860, use = 3335, size = 8192, split = 0}, lru_table = {next = 0x5c47d12186e8, prev = 0x5c47d12186e8}, ctr = 3335, generation = 0, hidectr = 0, lock = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' , __align = 0}, conf = {set_gid = 0, gid = 0, set_uid = 0, uid = 0, set_mode = 0, umask = 0, entry_timeout = 0.5, negative_timeout = 0, attr_timeout = 0.5, intr = 1, intr_signal = 22, remember = 0, hard_remove = 0, use_ino = 0, readdir_ino = 0, direct_io = 1, kernel_cache = 0, auto_cache = 0, ac_attr_timeout_set = 0, ac_attr_timeout = 0.5, nullpath_ok = 0, show_help = 0, modules = 0x0, debug = 0}, intr_installed = 0, fs = 0x5c47d12188c0, lockq = 0x0, pagesize = 4096, partial_slabs = { next = 0x78c609595000, prev = 0x78c609595000}, full_slabs = {next = 0x78c60a121000, prev = 0x78c609596000}, prune_thread = 0} (gdb) p *(struct fuse_file_info*)($r8) $18 = {flags = 32768, writepage = 0, direct_io = 0, keep_cache = 0, flush = 0, nonseekable = 0, flock_release = 0, cache_readdir = 0, padding = 0, padding2 = 0, fh = 111524889521775, lock_owner = 11505417049478139234, poll_events = 0} (gdb) p *(struct fuse_buf*)0x78c4c410e918 $27 = {size = 80, flags = 0, mem = 0x78c4fc0008e0, fd = 0, pos = 0} (gdb) p ((struct fuse_buf*)0x78c4c410e918)->mem $28 = (void *) 0x78c4fc0008e0 (gdb) p *(struct fuse_in_header*) $28 $29 = {len = 80, opcode = 15, unique = 818687616, nodeid = 6, uid = 1000000, gid = 1000000, pid = 2057454, padding = 0} (gdb) p/x *(struct fuse_read_in*) ($28+sizeof(struct fuse_in_header)) $32 = {fh = 0x656e696c6e6f, offset = 0xd05, size = 0x1000, read_flags = 0x2, lock_owner = 0x9fab73d922de0d62, flags = 0x8000, padding = 0x0} (gdb) p *0x656e696c6e6f Cannot access memory at address 0x656e696c6e6f ``` This specific coredump is extremely useful and interesting. We have invalid value of `fi->fh = 0x656e696c6e6f`. What I did is manually recalculated this value based on the libfuse implementation and memory dump. I was able to find original datastructure which fuse request (which comes from the kernel) and decode it. And.. I've got the same value! Which means that either: - kernel sends a wrong `fi->fh` value to us (very unlikely) - memory corruption happens somewhere in libfuse and read garbage
Analysis of core-lxcfs.1499342 ``` (gdb) bt #0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=134107568801344) at ./nptl/pthread_kill.c:44 #1 __pthread_kill_internal (signo=6, threadid=134107568801344) at ./nptl/pthread_kill.c:78 #2 __GI___pthread_kill (threadid=134107568801344, signo=signo@entry=6) at ./nptl/pthread_kill.c:89 #3 0x000079f891e94476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26 #4 0x000079f891e7a7f3 in __GI_abort () at ./stdlib/abort.c:79 #5 0x000079f891edb676 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x79f89202db77 "%s\n") at ../sysdeps/posix/libc_fatal.c:155 #6 0x000079f891ef2cfc in malloc_printerr (str=str@entry=0x79f892030790 "double free or corruption (out)") at ./malloc/malloc.c:5664 #7 0x000079f891ef4e70 in _int_free (av=0x79f89206cc80 , p=0x79f85c0000b0, have_lock=) at ./malloc/malloc.c:4588 #8 0x000079f891ef7453 in __GI___libc_free (mem=) at ./malloc/malloc.c:3391 #9 0x000079f891e360c8 in do_release_file_info () from /snap/lxd/current/lib/liblxcfs.so #10 0x000079f891e2b789 in proc_release () from /snap/lxd/current/lib/liblxcfs.so #11 0x000061d1e19a9f2c in ?? () #12 0x000061d1e19aac7f in ?? () #13 0x000079f8920a9b6a in fuse_do_release (f=0x61d1e28856b0, ino=8, path=0x79f7e8103030 "/proc/stat", fi=) at ../lib/fuse.c:3142 #14 0x000079f8920ae4fa in fuse_lib_release (req=0x79f7e8107ac0, ino=8, fi=0x79f859fffc90) at ../lib/fuse.c:4126 #15 0x000079f8920af738 in do_release (req=, nodeid=, inarg=) at ../lib/fuse_lowlevel.c:1455 #16 0x000079f8920bc13f in fuse_session_process_buf_int (se=0x61d1e28811b0, buf=0x79f734104378, ch=) at ../lib/fuse_lowlevel.c:2666 #17 0x000079f8920b47a7 in fuse_do_work (data=0x79f734104360) at ../lib/fuse_loop_mt.c:162 #18 0x000079f891ee6ac3 in start_thread (arg=) at ./nptl/pthread_create.c:442 #19 0x000079f891f78850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 ``` glibc detected double-free during releasing. We can't say anything more from that, because from coredump we see a final picture, we don't know who and when freeed this memory the first time.
Analysis of core-lxcfs.2897 ``` (gdb) bt #0 __vsnprintf_internal (string=0x0, maxlen=, format=0x7b77901cdf73 "%s", args=args@entry=0x7b777b3ff5a0, mode_flags=mode_flags@entry=0) at ./libio/vsnprintf.c:112 #1 0x00007b7790244856 in __GI___snprintf (s=, maxlen=, format=) at ./stdio-common/snprintf.c:31 #2 0x00007b77901c8a73 in read_file_fuse () from /snap/lxd/current/lib/liblxcfs.so #3 0x00007b77901beabb in ?? () from /snap/lxd/current/lib/liblxcfs.so #4 0x00007b77901c34c8 in proc_read () from /snap/lxd/current/lib/liblxcfs.so #5 0x0000587e7118e2c3 in ?? () #6 0x0000587e7118fa56 in ?? () #7 0x00007b779043df2e in fuse_fs_read_buf (fs=0x587e725208c0, path=0x7b73ec113f50 "/proc/diskstats", bufp=bufp@entry=0x7b777b3ffbf8, size=size@entry=16384, off=off@entry=0, fi=fi@entry=0x7b777b3ffc90) at ../lib/fuse.c:1820 #8 0x00007b7790440d66 in fuse_lib_read (req=0x7b73ec11a600, ino=4, size=16384, off=0, fi=0x7b777b3ffc90) at ../lib/fuse.c:3306 #9 0x00007b77904414fa in do_read (req=, nodeid=, inarg=) at ../lib/fuse_lowlevel.c:1346 #10 0x00007b779044e13f in fuse_session_process_buf_int (se=0x587e7251c1b0, buf=0x7b746410d288, ch=) at ../lib/fuse_lowlevel.c:2666 #11 0x00007b77904467a7 in fuse_do_work (data=0x7b746410d270) at ../lib/fuse_loop_mt.c:162 #12 0x00007b7790278ac3 in start_thread (arg=) at ./nptl/pthread_create.c:442 #13 0x00007b779030a850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 ``` In this case, problem is here: ``` struct file_info *d = INTTYPE_TO_PTR(fi->fh); ... char *cache = d->buf; ... l = snprintf(cache, cache_size, "%s", lbuf); if (l < 0) return log_error(0, "Failed to write cache"); ``` `d->buf` must be not NULL, but it is. Looks like somebody released a file while we are in read which should be not possible.
Analysis of core-lxcfs.3396 ``` (gdb) bt #0 __vsnprintf_internal (string=0x100 , maxlen=, format=0x7fd6aab64030 "cpu%d %lu 0 %lu %lu 0 0 0 0 0 0\n", args=args@entry=0x7fd6973ff6c0, mode_flags=mode_flags@entry=0) at ./libio/vsnprintf.c:112 #1 0x00007fd6aabdb856 in __GI___snprintf (s=, maxlen=, format=) at ./stdio-common/snprintf.c:31 #2 0x00007fd6aab57acc in ?? () from /snap/lxd/current/lib/liblxcfs.so #3 0x00007fd6aab5a478 in proc_read () from /snap/lxd/current/lib/liblxcfs.so #4 0x000057e77a58b2c3 in ?? () #5 0x000057e77a58ca56 in ?? () #6 0x00007fd6aadd4f2e in fuse_fs_read_buf (fs=0x57e77b5668c0, path=0x7fd464111910 "/proc/stat", bufp=bufp@entry=0x7fd6973ffbf8, size=size@entry=16384, off=off@entry=0, fi=fi@entry=0x7fd6973ffc90) at ../lib/fuse.c:1820 #7 0x00007fd6aadd7d66 in fuse_lib_read (req=0x7fd464112ca0, ino=8, size=16384, off=0, fi=0x7fd6973ffc90) at ../lib/fuse.c:3306 #8 0x00007fd6aadd84fa in do_read (req=, nodeid=, inarg=) at ../lib/fuse_lowlevel.c:1346 #9 0x00007fd6aade513f in fuse_session_process_buf_int (se=0x57e77b5621b0, buf=0x7fd48c128a18, ch=) at ../lib/fuse_lowlevel.c:2666 #10 0x00007fd6aaddd7a7 in fuse_do_work (data=0x7fd48c128a00) at ../lib/fuse_loop_mt.c:162 #11 0x00007fd6aac0fac3 in start_thread (arg=) at ./nptl/pthread_create.c:442 #12 0x00007fd6aaca1850 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 ``` in proc_stat_read(): ``` char *cache = d->buf + CPUALL_MAX_SIZE; ... l = snprintf(cache, cache_size, "cpu%d %" PRIu64 " 0 %" PRIu64 " %" PRIu64 " 0 0 0 0 0 0\n", cpu_to_render, cg_cpu_usage[physcpu].user, cg_cpu_usage[physcpu].system, new_idle); ``` again, very weird, `cache = 0x100` which is, obviously invalid address. `CPUALL_MAX_SIZE = (BUF_RESERVE_SIZE / 2) = 256 = 0x100`, which means that `d->buf` was NULL again!
webdock-io commented 1 day ago

Hi @mihalicyn

Amazing work so far, but concerning that it's outside of lxcfs seemingly and thus potentially has deeper roots. Any wholesale lxcfs debug deployment would only be feasible if we can do so without a reboot of the hosts, as otherwise we'd have to deploy it whenever a host crashes anyway (which is unfortunately not rare these days, explanation to follow) - but let's see what you come up with and how we can best achieve this.

Some information for you to go on:

apt install -y --install-recommends linux-generic-hwe-24.04
apt install -y zfsutils-linux apparmor ubuntu-standard bridge-utils htop nano lftp fail2ban git ntp ntpdate pigz php7.4-cli php7.4-curl php7.4-mbstring pwgen build-essential libboost-all-dev cmake iotop mbuffer snapd molly-guard tcptrack smartmontools net-tools iftop screen stress cpu-checker numactl iptables-persistent

We install lxd using snap, of course.

Now, from what we can tell this crash happens on hosts which are using some amount of disk i/o - without fully understanding your analysis it seems the crash is related to disk i/o or am i mistaken?

If it is, then what we're also seeing these days after migrating to latest kernel/Noble is occasional kernel crashes with no output in syslog or kern log other than null bytes - we have tracked this down to be very strongly correlated with the zfs arc cache and heavy read workloads. Essentially speaking, as long as the ARC RAM cache is not full and zfs can cache reads in memory things are fine - as soon as the ARC cache is filled and if the heavy read workloads continue, thus causing zfs to hit disk a lot, we get a kernel panic.

This is admittedly a more serious issue for us and we are doing our best to mitigate by spreading workloads across our infrastructure and allowing zfs to use obscene amounts of ARC cache. I thought I'd mention this issue as maybe it's related. However, we are seeing these kernel crashes on bare metal as well as vm's and these kernel crashes do not coincide with lxcfs crashes in any meaningful way, which happen on relatively calm systems also. So yeah, not directly related but maybe relevant not sure, as we have a feeling that lxcfs is crashing a lot less now that we've increased our ARC cache across our entire infrastructure.

Please let me know if there is any further, more specific information, you need. I would not say that we have a reliable reproducer per se, other than "this is happening fairly frequently" :)

mihalicyn commented 1 day ago

All the crashing lxcfs thus far has been on systems where we are running lxd containers inside LXD KVM virtual machines, so nested containers so to speak. We have yet to see lxcfs crash on systems where it's running on bare metal.

Ow! This is very-very important fact. Just to ensure that I understood you right. All LXCFS crashes happened only inside the VM instances. You have not seen any crashed on bare metal hosts yet. Right?

Now, from what we can tell this crash happens on hosts which are using some amount of disk i/o - without fully understanding your analysis it seems the crash is related to disk i/o or am i mistaken?

One of crashes happened on the read of /proc/diskstats file, but it does not mean that crash itself somehow related to I/O activity.

Interesting thing here is that all these crashes are somehow similar and at the same time different. There is nothing in common from the LXCFS perspective. Different files like /proc/stat, /proc/diskstats. Different operations: release, read. At the same time, in all cases we see an inconsistent memory state. What is the most weird is that each time traces go to a fuse file handle...

If it is, then what we're also seeing these days after migrating to latest kernel/Noble is occasional kernel crashes

This is painful. And, for sure, this is a priority 1 to fix. I would suggest to enable kdump to collect a coredump files for the kernel so you can report these crashes to the Ubuntu kernel maintainers. Please, refer to https://ubuntu.com/server/docs/kernel-crash-dump (feel free to ask me if you have any questions).

Essentially speaking, as long as the ARC RAM cache is not full and zfs can cache reads in memory things are fine - as soon as the ARC cache is filled and if the heavy read workloads continue, thus causing zfs to hit disk a lot, we get a kernel panic.

Hm, just in case, of course, you have tested your RAM modules with memtest+, disk SMARTs are fine too.

I would say that this can be a sign of a very serious problem in the kernel. ZFS, while being a piece of art in software engineering, is an out-of-tree Linux kernel module, which brings additional risks because OpenZFS developers have to adapt it to each kernel version and errors are not excluded there. It can be a misuse of API because of API changes in the upstream kernel and so on. Which can lead to memory leaks, use-after-free errors, OOB r/w and so on. As a consequence, it may corrupt your kernel memory state and make you system to behave in extremely unexplainable ways. Even things which are not directly connected to ZFS or filesystems or disk I/O can start to break due to memory corruption (when a buggy code writes to a memory region which does not belong to it).

However, we are seeing these kernel crashes on bare metal as well as vm's and these kernel crashes do not coincide with lxcfs crashes in any meaningful way, which happen on relatively calm systems also.

Memory corruption is a random thing. For example, one time it can override a memory structure of a "not important" (there is no unimportant things in the kernel ;-)) thing and you won't see any visible effects. At the same time, next time memory corruption can shot in a more important memory area which will be noticed immediately, because of crash of some process or VM or even the entire system.

So yeah, not directly related but maybe relevant not sure, as we have a feeling that lxcfs is crashing a lot less now that we've increased our ARC cache across our entire infrastructure.

And this is a very important observation. It can be that ZFS somehow corrupts memory and when you decreased a memory corruption rate LXCFS started to crash less often too.

To conclude, I would suggest to:

May be related to https://github.com/openzfs/zfs/issues/16187

mihalicyn commented 1 day ago

Hi @capriciousduck !

Thanks for reporting this!

I'm facing this too I think as I noticed all the 4 cores of my host visible in one of my containers but for some reason I restarted my host and all looks good now.

Please, can you tell me the following info from your system:

cat /etc/os-release
uname -a
lsmod

also, are you using just LXC+LXCFS or LXD or Incus?

webdock-io commented 1 day ago

Ow! This is very-very important fact. Just to ensure that I understood you right. All LXCFS crashes happened only inside the VM instances. You have not seen any crashed on bare metal hosts yet. Right?

That's correct. As of yet lxcfs crashes have only happened inside VMs and not on bare metal yet

This is painful. And, for sure, this is a priority 1 to fix. I would suggest to enable kdump to collect a coredump files for the kernel so you can report these crashes to the Ubuntu kernel maintainers. Please, refer to https://ubuntu.com/server/docs/kernel-crash-dump (feel free to ask me if you have any questions).

We looked into this, and the more we looked the more daunting this became. It is by no means as simple as shown in the Ubuntu docs it seems, and we ended up studying this guide in detail: https://hackmd.io/@0xff07/S1ASmzgun

And we concluded that the difficulty in setting this up, and difficulty in reading the dumps (although, maybe we wouldn't actually need to read these ourselves, but rather pass this on to more knowledgeable persons) really discouraged us to even try. Not to mention this requires a reboot before it's even active, which is nothing we can/will do (unless a system has already crashed and we need to reboot anyway....)

I guess we could attempt this, so that this is "ready" so that once a system needs a reboot anyway, from then on it will be active and then... We would see. If you can illuminate this for us / give us some pointers on how to get this set up I'd be willing to give it a shot.

Hm, just in case, of course, you have tested your RAM modules with memtest+, disk SMARTs are fine too.

This is all brand new systems with ECC RAM throughout and new drives with no SMART errors. We did run 24-48 hour high load testing where we stressed memory before deploying into production, which has always been enough/succesful in detecting bad RAM in our past experience.

Not to mention these issues have happened across a wide range of hosts (and even platforms, Ryzen vs. Intel Xeon) so we are pretty confident this is not hardware related in any way.

I would say that this can be a sign of a very serious problem in the kernel. ZFS, while being a piece of art in software engineering, is an out-of-tree Linux kernel module, which brings additional risks because OpenZFS developers have to adapt it to each kernel version and errors are not excluded there. It can be a misuse of API because of API changes in the upstream kernel and so on. Which can lead to memory leaks, use-after-free errors, OOB r/w and so on. As a consequence, it may corrupt your kernel memory state and make you system to behave in extremely unexplainable ways. Even things which are not directly connected to ZFS or filesystems or disk I/O can start to break due to memory corruption (when a buggy code writes to a memory region which does not belong to it).

Yes I agree from the information you have given us here and from what we've seen, this really smells like memory corruption caused by ZFS.

To conclude, I would suggest to:

  • enable kdump (to collect kernel crashes)
  • try to experiment with different kernel versions (pick an older one on some nodes to see if situation changes)

Maybe you could save us some research time and/or mistakes, if you can show us the steps on how to safely revert the kernel version? We have never done this, we have only ever upgraded the kernel :D

May be related to openzfs/zfs#16187

We found this issue as well in our research and felt it was very similar to our case! Although, we have been running with pretty recent kernels and zfs versions newer than the one reported in that issue, in production for a long time, and saw none of these crashes before - so we really don't know if this is the same or not. It's really only when we deployed systems with the latest Noble kernel and the zfs that comes with it, zfs-2.2.2-0ubuntu9, that we started seeing this.

mihalicyn commented 1 day ago

And we concluded that the difficulty in setting this up, and difficulty in reading the dumps (although, maybe we wouldn't actually need to read these ourselves, but rather pass this on to more knowledgeable persons) really discouraged us to even try.

It is definitely worth it. As having a coredump will allow us to at least identify a callstack and report the issue properly (who knows, may be it's a known issue). As for reading a dump, don't worry, you can share it with me I'll help with that.

Maybe you could save us some research time and/or mistakes, if you can show us the steps on how to safely revert the kernel version? We have never done this, we have only ever upgraded the kernel :D

If you are using Noble, it looks like there is no choice. As 6.8 is a main kernel Noble released with. In this case you can try to deploy upstream kernel version + upstream ZFS 2.2.4 (https://openzfs.github.io/openzfs-docs/Developer%20Resources/Custom%20Packages.html#debian-and-ubuntu). ZFS update from 2.2.2 -> 2.2.4 should be pretty safe as it's minor and bugfix release. I don't know why zfs package in Ubuntu Noble is still on 2.2.2...

Maybe, it's better to try to build newer ZFS and deploy it to see if it helps. And if not, then go for an upstream kernel versions (for example, https://kernel.ubuntu.com/mainline/v6.8.12/).

stgraber commented 1 day ago

You could try my mainline packages to get a clean upstream kernel (no Ubuntu changes) but still built with a config that's very close to the Ubuntu one.

To do so, you'll need to:

webdock-io commented 13 hours ago

It is definitely worth it. As having a coredump will allow us to at least identify a callstack and report the issue properly (who knows, may be it's a known issue). As for reading a dump, don't worry, you can share it with me I'll help with that.

Thanks so much @mihalicyn that is amazing. We will proceed to, at least perform the intial steps before the reboot, on all our hosts today, so they are somewhat ready to collect crash dumps. We have a single system which crashed again last night (after having crashed two times in the previous 12 hours, so a system in a pretty flaky state) where we fully enabled this, but there we also took some mitigation steps such as upgrading to the latest kernel (from 6.8.0-31 to 6.8.0-36), moving some high i/o users away and increasing the ARC cache even further. So it's not certain this system will crash again and provide a dump for us.

I will of course ping you as soon as such a dump is produced.

As to the suggestion from @mihalicyn to try zfs-dkms - we've for other reasons installed zfs-dkms in the past and now we have some systems on a legacy part of our infrastructure which has this issue: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/2044630

As it's really not recommended to install zfs-dkms and seemingly if left for "too long" it can cause problems down the line where removing it is seemingly fraught with potential danger and would involve downtime for customers for sure, we are really hesitant to do something like this. Ideally we'd stay on official Ubuntu kernels and do things "the recommended way"

Which leads me to the suggestion from @stgraber - this also strikes me as potentially dangerous in a couple of ways, firstly it is unclear to me whether we could easily revert to an official kernel with the built-in zfs support down the line. What if Stephane gets hit by a bus and no longer able to maintain the Zabbly builds? When would we know for sure official Ubuntu kernel is safe to switch back to? I really do appreciate the suggestion @stgraber but without further insight (and a proper plan) I am currently very hesitant to do something like this. With that said, I have reached out to you directly by email and I hope we can have a further chat about this in the coming days if you have time.

So, for right now what we are doing is preparation and mitigation steps where we:

  1. Enable kernel crash dump collection on all affected hosts
  2. Increase the ARC caches to the maximum feasable
  3. Moving users doing high read i/o around to decrease the chances of any one system being loaded to the point of triggering this fault (too much)
  4. Doing system upgrades everywhere so if/when they are rebooted not only will crash collection be active but the latest kernel available (in the slim hope that the issue is resolved already, somehow)

Anything beyond that will have to be the subject of further analysis, as we can't just willy-nilly go to "experimental" kernels with associated reboots and downtime for customers time and again :(

I have a followup question for you guys

We are deploying a number of new host machines next week on our infrastructure. This gives us the opportunity to do things differently, and hopefully deploy unaffected systems, which could be "safe havens" to migrate customers to which are triggering the issue on the unstable part of our infrastructure. Would we benefit from basing these systems on Ubuntu Jammy instead of Noble? Or would we just be getting the latest kernel/zfs modules, the same as for Noble, anyway and be in the same situation?

I guess Jammy would give us better opportunity to run with an older kernel, say the latest kernel version we know was succesful in production before. I guess what I am asking is, what would you do in this situation? Any input / knowledge here would be appreciated :)