search

iggy-rs / iggy

Iggy is the persistent message streaming platform written in Rust, supporting QUIC, TCP and HTTP transport protocols, capable of processing millions of messages per second.
https://iggy.rs
MIT License
1.75k stars 81 forks source link

`thread 'tokio-runtime-worker' panicked #986

Closed Liberxue closed 1 month ago

Liberxue commented 1 month ago

rustc --version

rustc 1.80.0-nightly (8387315ab 2024-05-14)

panic logs

2024-06-04T11:26:51.528305Z ERROR ThreadId(02) server::channels::commands::print_sysinfo: Failed to send SysInfoPrintCommand. Error: sending on a closed chan
nel
2024-06-04T11:26:59.842590Z  INFO ThreadId(02) server::tcp::tcp_listener: Accepted new TCP connection: 127.0.0.1:53629
thread 'tokio-runtime-worker' panicked at /Users/liber/.cargo/registry/src/index.crates.io-6f17d22bba15001f/keepcalm-0.3.5/src/implementation.rs:114:21:
This lock was poisoned by a panic elsewhere in the code.
2024-06-04T11:26:59.842971Z  INFO ThreadId(02) server::tcp::tcp_listener: Accepted new TCP connection: 127.0.0.1:53630
thread 'tokio-runtime-worker' panicked at /Users/liber/.cargo/registry/src/index.crates.io-6f17d22bba15001f/keepcalm-0.3.5/src/implementation.rs:114:21:
This lock was poisoned by a panic elsewhere in the code.
2024-06-04T11:27:01.527824Z ERROR ThreadId(02) server::channels::commands::print_sysinfo: Failed to send SysInfoPrintCommand. Error: sending on a closed chan
nel
2024-06-04T11:27:11.527541Z ERROR ThreadId(02) server::channels::commands::print_sysinfo: Failed to send SysInfoPrintCommand. Error: sending on a closed chan
nel
2024-06-04T11:27:11.528361Z ERROR ThreadId(02) server::channels::commands::save_messages: Failed to send SaveMessagesCommand. Error: sending on a closed chan
nel
2024-06-04T11:27:21.528694Z ERROR ThreadId(02) server::channels::commands::print_sysinfo: Failed to send SysInfoPrintCommand. Error: sending on a closed chan
nel
2024-06-04T11:27:31.527756Z ERROR ThreadId(02) server::channels::commands::print_sysinfo: Failed to send SysInfoPrintCommand. Error: sending on a closed chan
nel
2024-06-04T11:27:41.528889Z ERROR ThreadId(02) server::channels::commands::print_sysinfo: Failed to send SysInfoPrintCommand. Error: sending on a closed chan
nel

COMMAND     PID  USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
iggy-serv 95799 liber   14u  IPv4 0x5ee4cd6944e3df69      0t0  TCP *:8090 (LISTEN)
telnet  localhost 8090
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Connection closed by foreign host.
hubcio commented 1 month ago

@Liberxue could you please provide full log (directory logs)? preferably along with default_config.toml, you can both it in local_data directory

hubcio commented 1 month ago

also, what led to the reproduction? what exactly were done to trigger this?

Liberxue commented 1 month ago

@hubcio Thank you very much for your response 🙏 ~~~ I apologize, I now understand the issue. It occurred because I was conducting a benchmark, which resulted in insufficient disk space for my Docker, hence the problem arose.

Caused by: 0: IO error 1: No space left on device (os error 28)) thread 'tokio-runtime-worker' panicked at server/src/streaming/partitions/messages.rs:490:55: calledResult::unwrap()on anErrvalue: CannotSaveMessagesToSegment(Failed to save messages to segment: