Closed realuptime closed 9 months ago
Another over-night crash:
rtpQueueDelay 0.232422 too large 1 13919.3 RTP queue 118 packets discarded for SSRC 1 hiSeqTx 58159 hiSeqAckendl 0 seqNrOfNextRtp 58160 seqNrOfLastRtp 58277 diff 118 imp.rs: force_idr rc true enabled 1 rtpQueueDelay 0.253906 too large 1 Warning: Unexpected item 0x7f85e402b7e0 dequeued from queue queue1 (refcounting problem?) 13919.6 RTP queue 133 packets discarded for SSRC 1 hiSeqTx 58283 hiSeqAckendl 0 seqNrOfNextRtp 58284 seqNrOfLastRtp 58416 diff 133 3:52:01.238217250 ESC[36m130979ESC[00m 0x25fb8c0 ESC[31;01mERROR ESC[00m ESC[00m queue_dataflow gstqueue.c:1486:gst_queue_push_one:
ESC[00m exit because we have no item in the queue thread ' ' panicked at 'Screamtx callback srcpad.push failed: Error', src/screamtx/imp.rs:360:22 note: run with RUST_BACKTRACE=1
environment variable to display a backtrace fatal runtime error: failed to initiate panic, error 5 Caught fatal signal - signo:6 code:-6 errno:0 addr:0x3e80001ffa3 Obtained 17 stack frames.0 0x007f87d066ecf0 in __sigaction
1 0x007f87d06c526b in pthread_kill
2 0x007f87d066ec46 in gsignal
3 0x007f87d06557fc in abort
4 0x007f864c990327 in std::sys::unix::abort_internal::hf940f5a54d8a80b4
5 0x007f864c98ea42 in rust_panic
6 0x007f864c98e8ca in std::panicking::rust_panic_with_hook::h5cafdc4b3bfd5528
7 0x007f864c98e609 in std::panicking::begin_panichandler::$u7b$$u7b$closure$u7d$$u7d$::hf31c60f40775892c
8 0x007f864c98d3dc in std::sys_common::backtrace::__rust_end_short_backtrace::h28a5c7be595826cd
9 0x007f864c98e312 in rust_begin_unwind
10 0x007f864c8d0ee3 in core::panicking::panic_fmt::h8fa27a0b37dd98b7
11 0x007f864c8d1313 in core::result::unwrap_failed::h62abf61e411aaa08
12 0x007f864c8ddb18 in core::result::Result$LT$T$C$E$GT$::expect::h54fe755c33896c5a
13 0x007f864c8e594e in gstscream::screamtx::imp::callback::h11dff52b2a2873a4
14 0x007f86900b8297 in transmitRtpThread(void*)
15 0x007f87d06c3402 in pthread_condattr_setpshared
16 0x007f87d0752590 in __xmknodat
and another:
summary 1628.9 Transmit rate = 8367kbps, PLR = 0.00%( 0.03%), RTT = 0.049s, Queue delay = 0.015s encoder_rate 8000 kbbps InputServer: Received Heartbeat InputServer: Received Heartbeat summary 1630.9 Transmit rate = 8264kbps, PLR = 0.00%( 0.03%), RTT = 0.049s, Queue delay = 0.015s encoder_rate 8000 kbbps refresh 1631 RTP queue 1 packets discarded for SSRC 1 refresh 1631 RTP queue 1 packets discarded for SSRC 1 refresh 1631 RTP queue 1 packets discarded for SSRC 1 refresh 1631 RTP queue 1 packets discarded for SSRC 1 refresh 1631 RTP queue 1 packets discarded for SSRC 1 refresh 1631 RTP queue 1 packets discarded for SSRC 1 refresh 1631 RTP queue 1 packets discarded for SSRC 1 refresh 1631 RTP queue 1 packets discarded for SSRC 1 refresh 1631 RTP queue 1 packets discarded for SSRC 1 refresh 1631 RTP queue 1 packets discarded for SSRC 1 refresh 1631 RTP queue 1 packets discarded for SSRC 1 refresh 1631 RTP queue 1 packets discarded for SSRC 1 refresh 1631 RTP queue 1 packets discarded for SSRC 1 refresh 1631 RTP queue 1 packets discarded for SSRC 1 Error: gst_pad_push: assertion 'GST_IS_BUFFER (buffer)' failed thread '
' panicked at 'Screamtx callback srcpad.push failed: Error', src/screamtx/imp.rs:360:22 note: run with RUST_BACKTRACE=1
environment variable to display a backtrace fatal runtime error: failed to initiate panic, error 5 Caught fatal signal - signo:6 code:-6 errno:0 addr:0x3e800005ba0 refresh 1631.03 RTP queue 1 packets discarded for SSRC 1 refresh 1631.03 RTP queue 1 packets discarded for SSRC 1 refresh 1631.03 RTP queue 1 packets discarded for SSRC 1 refresh 1631.03 RTP queue 1 packets discarded for SSRC 1 refresh 1631.03 RTP queue 1 packets discarded for SSRC 1 refresh 1631.03 RTP queue 1 packets discarded for SSRC 1 refresh 1631.03 RTP queue 1 packets discarded for SSRC 1 refresh 1631.03 RTP queue 1 packets discarded for SSRC 1 refresh 1631.03 RTP queue 1 packets discarded for SSRC 1 refresh 1631.03 RTP queue 1 packets discarded for SSRC 1 refresh 1631.03 RTP queue 1 packets discarded for SSRC 1 refresh 1631.03 RTP queue 1 packets discarded for SSRC 1 refresh 1631.03 RTP queue 1 packets discarded for SSRC 1 refresh 1631.03 RTP queue 1 packets discarded for SSRC 1 refresh 1631.03 RTP queue 1 packets discarded for SSRC 1 refresh 1631.03 RTP queue 1 packets discarded for SSRC 1 refresh 1631.03 RTP queue 1 packets discarded for SSRC 1 refresh 1631.03 RTP queue 1 packets discarded for SSRC 1 refresh 1631.03 RTP queue 1 packets discarded for SSRC 1 refresh 1631.03 RTP queue 1 packets discarded for SSRC 1 refresh 1631.05 RTP queue 1 packets discarded for SSRC 1 refresh 1631.05 RTP queue 1 packets discarded for SSRC 1 refresh 1631.05 RTP queue 1 packets discarded for SSRC 1 refresh 1631.05 RTP queue 1 packets discarded for SSRC 1 refresh 1631.05 RTP queue 1 packets discarded for SSRC 1 refresh 1631.05 RTP queue 1 packets discarded for SSRC 1 refresh 1631.05 RTP queue 1 packets discarded for SSRC 1 refresh 1631.05 RTP queue 1 packets discarded for SSRC 1 refresh 1631.05 RTP queue 1 packets discarded for SSRC 1 refresh 1631.05 RTP queue 1 packets discarded for SSRC 1 refresh 1631.05 RTP queue 1 packets discarded for SSRC 1 refresh 1631.05 RTP queue 1 packets discarded for SSRC 1 refresh 1631.05 RTP queue 1 packets discarded for SSRC 1 refresh 1631.05 RTP queue 1 packets discarded for SSRC 1 refresh 1631.05 RTP queue 1 packets discarded for SSRC 1 refresh 1631.05 RTP queue 1 packets discarded for SSRC 1 refresh 1631.07 RTP queue 1 packets discarded for SSRC 1 refresh 1631.07 RTP queue 1 packets discarded for SSRC 1 refresh 1631.07 RTP queue 1 packets discarded for SSRC 1 refresh 1631.07 RTP queue 1 packets discarded for SSRC 1 refresh 1631.07 RTP queue 1 packets discarded for SSRC 1 refresh 1631.07 RTP queue 1 packets discarded for SSRC 1 refresh 1631.07 RTP queue 1 packets discarded for SSRC 1 refresh 1631.07 RTP queue 1 packets discarded for SSRC 1 refresh 1631.07 RTP queue 1 packets discarded for SSRC 1 refresh 1631.07 RTP queue 1 packets discarded for SSRC 1 refresh 1631.07 RTP queue 1 packets discarded for SSRC 1 refresh 1631.07 RTP queue 1 packets discarded for SSRC 1 refresh 1631.07 RTP queue 1 packets discarded for SSRC 1 refresh 1631.07 RTP queue 1 packets discarded for SSRC 1 refresh 1631.07 RTP queue 1 packets discarded for SSRC 1 refresh 1631.07 RTP queue 1 packets discarded for SSRC 1 refresh 1631.09 RTP queue 1 packets discarded for SSRC 1 refresh 1631.09 RTP queue 1 packets discarded for SSRC 1 refresh 1631.09 RTP queue 1 packets discarded for SSRC 1 refresh 1631.09 RTP queue 1 packets discarded for SSRC 1 refresh 1631.09 RTP queue 1 packets discarded for SSRC 1 Obtained 17 stack frames.0 0x007fa46c7d2cf0 in __sigaction
1 0x007fa46c82926b in pthread_kill
2 0x007fa46c7d2c46 in gsignal
3 0x007fa46c7b97fc in abort
4 0x007fa2f9390327 in std::sys::unix::abort_internal::hf940f5a54d8a80b4
5 0x007fa2f938ea42 in rust_panic
6 0x007fa2f938e8ca in std::panicking::rust_panic_with_hook::h5cafdc4b3bfd5528
7 0x007fa2f938e609 in std::panicking::begin_panichandler::$u7b$$u7b$closure$u7d$$u7d$::hf31c60f40775892c
8 0x007fa2f938d3dc in std::sys_common::backtrace::__rust_end_short_backtrace::h28a5c7be595826cd
9 0x007fa2f938e312 in rust_begin_unwind
10 0x007fa2f92d0ee3 in core::panicking::panic_fmt::h8fa27a0b37dd98b7
11 0x007fa2f92d1313 in core::result::unwrap_failed::h62abf61e411aaa08
12 0x007fa2f92ddb18 in core::result::Result$LT$T$C$E$GT$::expect::h54fe755c33896c5a
13 0x007fa2f92e594e in gstscream::screamtx::imp::callback::h11dff52b2a2873a4
14 0x007fa30c04c297 in transmitRtpThread(void*)
15 0x007fa46c827402 in pthread_condattr_setpshared
16 0x007fa46c8b6590 in __xmknodat
Pushed the fix
Rust code does not like null pointers. RtpQueue.pop() sometimes extracts null items (when the queue is empty?)
Here are three callstacks when it happened:
1:
2:
3, exactly when a refresh happened: