search

F-Stack / f-stack

F-Stack is an user space network development kit with high performance based on DPDK, FreeBSD TCP/IP stack and coroutine API.
http://www.f-stack.org
Other
3.82k stars 891 forks source link

Blocked on fget_unlocked #732

Open uvletter opened 1 year ago

uvletter commented 1 year ago

My redis program is blocking at the startup, and the CPU runs to 100%, it seems like a busy loop.

When I gstack the process the stacktrace looks like

Thread 1 (Thread 0x7f1f6caae340 (LWP 800)):
#0  0x000000000044839a in fget_unlocked ()
#1  0x0000000000449255 in fget ()
#2  0x0000000000466dfb in kern_ioctl ()
#3  0x000000000043f6d3 in ff_ioctl ()
jfb8856606 commented 1 year ago

I debug redis-6.2.6 and it work correctly, may have some state not cleaned up, you can try reboot redis or OS, and retry it.

If the issue still exists, you can gdb redis and show me the full stack trace.

uvletter commented 1 year ago

Thx, it's not a stable issues, the busy loop seems to happen in refcount_acquire_if_gt. If I re-compile the whole fstack as well as freebsd, the problem seems to disappear, but sometimes after a compilation it appears again. I didn't dive into this problem, keep it here for a record.

rajeshwarp2002 commented 1 year ago

I keep getting this often at higher rates, sometime comes too many times and sometimes doesnt come. same backtrace while setting NONBLOCK for socket options did you find any solution to this? @uvletter @jfb8856606

uvletter commented 1 year ago

Nope, I have no idea about how it occurs. I find trigger a freebsd recompilation by mutating the redcount.h can forbid it.