samywee
commented
3 years ago Here is few more additional info when this is happenning
my pid of bees process is 2176 At this time bees is not logging anything and CPU is 100% by the bees process.
head /proc/2176/task/*/stack
==> /proc/2176/task/2176/stack <==
[<0>] do_sigtimedwait.isra.43+0x19b/0x230
[<0>] __do_sys_rt_sigtimedwait+0x85/0xf0
[<0>] __x64_sys_rt_sigtimedwait+0x1e/0x20
[<0>] do_syscall_64+0x5a/0x110
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[<0>] 0xffffffffffffffff
==> /proc/2176/task/2177/stack <==
[<0>] futex_wait_queue_me+0xc4/0x120
[<0>] futex_wait+0x15b/0x250
[<0>] do_futex+0x2ec/0xb10
[<0>] __x64_sys_futex+0x14c/0x188
[<0>] do_syscall_64+0x5a/0x110
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[<0>] 0xffffffffffffffff
==> /proc/2176/task/2178/stack <==
[<0>] futex_wait_queue_me+0xc4/0x120
[<0>] futex_wait+0x15b/0x250
[<0>] do_futex+0x2ec/0xb10
[<0>] __x64_sys_futex+0x14c/0x188
[<0>] do_syscall_64+0x5a/0x110
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[<0>] 0xffffffffffffffff
==> /proc/2176/task/2181/stack <==
[<0>] hrtimer_nanosleep+0xd6/0x1f0
[<0>] __x64_sys_nanosleep+0x7b/0xa0
[<0>] do_syscall_64+0x5a/0x110
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[<0>] 0xffffffffffffffff
==> /proc/2176/task/2182/stack <==
[<0>] futex_wait_queue_me+0xc4/0x120
[<0>] futex_wait+0x15b/0x250
[<0>] do_futex+0x2ec/0xb10
[<0>] __x64_sys_futex+0x14c/0x188
[<0>] do_syscall_64+0x5a/0x110
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[<0>] 0xffffffffffffffff
==> /proc/2176/task/2183/stack <==
[<0>] balance_dirty_pages+0x42c/0xd80
[<0>] balance_dirty_pages_ratelimited+0x2c6/0x3c0
[<0>] btrfs_buffered_write.isra.28+0x4c7/0x760 [btrfs]
[<0>] btrfs_file_write_iter+0x3f6/0x590 [btrfs]
[<0>] __vfs_write+0x131/0x190
[<0>] vfs_write+0xab/0x1b0
[<0>] ksys_pwrite64+0x66/0xa0
[<0>] __x64_sys_pwrite64+0x1e/0x20
[<0>] do_syscall_64+0x5a/0x110
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
==> /proc/2176/task/2184/stack <==
[<0>] futex_wait_queue_me+0xc4/0x120
[<0>] futex_wait+0x15b/0x250
[<0>] do_futex+0x2ec/0xb10
[<0>] __x64_sys_futex+0x14c/0x188
[<0>] do_syscall_64+0x5a/0x110
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[<0>] 0xffffffffffffffff
==> /proc/2176/task/2185/stack <==
[<0>] wait_current_trans+0xd5/0x110 [btrfs]
[<0>] start_transaction+0x330/0x410 [btrfs]
[<0>] btrfs_start_transaction+0x1e/0x20 [btrfs]
[<0>] btrfs_create+0x59/0x200 [btrfs]
[<0>] lookup_open+0x512/0x760
[<0>] path_openat+0x54d/0xe70
[<0>] do_filp_open+0x93/0x100
[<0>] do_sys_open+0x184/0x210
[<0>] __x64_sys_openat+0x20/0x30
[<0>] do_syscall_64+0x5a/0x110
==> /proc/2176/task/4880/stack <==
[<0>] wait_current_trans+0xd5/0x110 [btrfs]
[<0>] start_transaction+0x1e2/0x410 [btrfs]
[<0>] btrfs_join_transaction+0x1d/0x20 [btrfs]
[<0>] iterate_extent_inodes+0x271/0x320 [btrfs]
[<0>] iterate_inodes_from_logical+0x9c/0xd0 [btrfs]
[<0>] btrfs_ioctl_logical_to_ino+0x10e/0x1b0 [btrfs]
[<0>] btrfs_ioctl+0xd3b/0x2ce0 [btrfs]
[<0>] do_vfs_ioctl+0xa8/0x620
[<0>] ksys_ioctl+0x67/0x90
[<0>] __x64_sys_ioctl+0x1a/0x20
==> /proc/2176/task/4881/stack <==
[<0>] 0xffffffffffffffff
strace -p 2176
strace: Process 2176 attached
gettid() = 2176
write(2, "<7>bees[2176]: Restarting sigwai"..., 38) = 38
rt_sigtimedwait([INT TERM],
Command line options (Started from systemctl)
/usr/lib/bees/bees --no-timestamps --strip-paths --no-timestamps --verbose=8 --loadavg-target=6 /run/bees/mnt/fe151c71-0f9c-4205-b47f-3b1c91b5f678
kakra
Zygo
I have been trying to run bees for few days on a backup data set of size ~500GB on a 1800GB btrfs filesystem. In testing on a small data set bees seems to achive what I wanted it to do. But on this larger fileset bees go into hang often.
Here are my settings. Ubuntu 18.01 with kernel version 4.19.0 (upgraded manually to this version to run bees) , btrfs-progs v4.15.1
Have a 384MB - DB_SIZE
3CPU / 6GB memory on server where I run bees (VM - intel )
I have a combination avarage files as well as few .vhd files (vhd => microsoft backup imagefiles)
After about 2 hrs or so, server start to increase load, bees process at 100% cpu and hangs at this stage. When I try to stop at this stage bees become a defunt process and //.beeshome - not accessessible (cannot do a ls).
Thre are multiple messages in the log. I am pasting here last parts of it (I was running with Verbose=6). Note that since 02:40:44 no logs till I tried to stop. Any precautions that I can take to avoid this? Can I get bees to work for me or do I have to give up due to some kernel incompatibility as many have discussed? Have I hit a issue with a large files trying to de-dupe?
Not sure if this is related to the old thread (https://github.com/Zygo/bees/issues/63 - I thought I will start a new thread as it was old.)
I can provide additional info at your request. Your help is appreciated.