cloudflare / pingora

A library for building fast, reliable and evolvable network services.
Apache License 2.0
21.39k stars 1.18k forks source link

one thread in the service cpu load reach the 99% #378

Open suningzh opened 2 weeks ago

suningzh commented 2 weeks ago

Describe the bug

there are four threads in my service and i closed the work_stealing,When it has been running for two days i found one thread in the service cpu load reach the 99%

Pingora info

Please include the following information about your environment:

Pingora version: 0.3.0 Rust version: cargo 1.80.0 (376290515 2024-07-16) Operating system version: Debian 12.4

the perf top message: 25.82% pingora-demo [.] <tokio::future::poll_fn::PollFn as core::future::future::Future>::poll 17.01% pingora-demo [.] pingora_proxy::proxyh1::$LT$impl$u20$pingora_proxy..HttpProxy$LT$SV$GT$$GT$::proxy_handledownstream::$u7b$$u7b$closure$u7d$$u7d$::h68 9.24% libc-2.31.so [.] malloc 5.81% pingora-demo [.] pingora_error::Error::create 5.58% pingora-demo [.] ::poll 5.25% pingora-demo [.] tokio::macros::support::thread_rng_n 4.17% libc-2.31.so [.] cfree 2.42% pingora-demo [.] core::ptr::drop_in_place 1.90% pingora-demo [.] core::ptr::drop_in_place$LT$$LP$pingora_core..protocols..http..server..Session..read_body_or_idle..$u7b$$u7b$closure$u7d$$u7d$$C$tokio..s 1.76% pingora-demo [.] core::ptr::drop_in_place$LT$pingora_proxy..proxy_h1..$LT$impl$u20$pingora_proxy..HttpProxy$LT$SV$GT$$GT$..proxy_handle_downstream..$u7b$$ 1.46% pingora-demo [.] __rdl_dealloc 1.33% pingora-demo [.] rdl_alloc 1.18% pingora-demo [.] ::drop 1.04% pingora-demo [.] tokio::sync::batch_semaphore::Semaphore::try_acquire 0.94% libc-2.31.so [.] 0x000000000008262f 0.83% pingora-demo [.] core::ptr::drop_in_place$LT$pingora_proxy..proxy_cache..ServeFromCache..next_http_task..$u7b$$u7b$closure$u7d$$u7d$$GT$::hbebc4c258c82f58 0.83% libc-2.31.so [.] 0x0000000000082606 0.72% pingora-demo [.] rust_dealloc 0.67% libc-2.31.so [.] 0x00000000000825a9 0.67% libc-2.31.so [.] 0x00000000000825a0 0.63% libc-2.31.so [.] 0x00000000000825c3 0.62% libc-2.31.so [.] 0x00000000000825d0 0.61% libc-2.31.so [.] 0x00000000000825b1 0.61% libc-2.31.so [.] 0x0000000000082875 0.58% libc-2.31.so [.] 0x000000000008261a 0.56% libc-2.31.so [.] 0x00000000000826f2 0.56% libc-2.31.so [.] 0x00000000000825f0 0.54% libc-2.31.so [.] 0x0000000000082864 0.54% libc-2.31.so [.] 0x00000000000826d5 0.53% libc-2.31.so [.] 0x00000000000826ea

drcaramelsyrup commented 2 weeks ago

This is curious, do you have an example project demonstrating this?