EricssonResearch / scream

SCReAM - Mobile optimised congestion control algorithm
BSD 2-Clause "Simplified" License
174 stars 53 forks source link

Rust screamtx crash caused by a null buffer passed to rust callback #56

Closed realuptime closed 9 months ago

realuptime commented 9 months ago

Rust code does not like null pointers. RtpQueue.pop() sometimes extracts null items (when the queue is empty?)

Here are three callstacks when it happened:

1:

rtpQueueDelay 0.237793 too large 1 6693.13 RTP queue 123 packets discarded for SSRC 1 hiSeqTx 27367 hiSeqAckendl 0 seqNrOfNextRtp 27368 seqNrOfLastRtp 27490 diff 123 imp.rs: force_idr rc true enabled 1 Error: gst_pad_push: assertion 'GST_IS_BUFFER (buffer)' failed rtpQueueDelay 0.250488 too large 1 6693.39 RTP queue 133 packets discarded for SSRC 1 hiSeqTx 27500 hiSeqAckendl 0 seqNrOfNextRtp 27501 seqNrOfLastRtp 27633 diff 133 thread '' panicked at 'Screamtx callback srcpad.push failed: Error', src/screamtx/imp.rs:358:22 note: run with RUST_BACKTRACE=1 environment variable to display a backtrace fatal runtime error: failed to initiate panic, error 5 Caught fatal signal - signo:6 code:-6 errno:0 addr:0x3e80001c894 Obtained 15 stack frames.

0 0x007f521fd86420 in funlockfile

1 0x007f521fbc100b in gsignal

2 0x007f521fba0859 in abort

3 0x007f50e0082b37 in std::sys::unix::abort_internal::hbf9c556da2861ce5

4 0x007f50e0081252 in rust_panic

5 0x007f50e00810da in std::panicking::rust_panic_with_hook::hdb4da1ae79c845a5

6 0x007f50e0080e19 in std::panicking::begin_panichandler::$u7b$$u7b$closure$u7d$$u7d$::h02b5b35b126d5cf2

7 0x007f50e007fbec in std::sys_common::backtrace::__rust_end_short_backtrace::h6c6853376cf416d1

8 0x007f50e0080b22 in rust_begin_unwind

9 0x007f50e0049f23 in core::panicking::panic_fmt::hfd9e949092070b66

10 0x007f50e004a353 in core::result::unwrap_failed::h4d34d8346233eb49

11 0x007f50e004d2b4 in gstscream::screamtx::imp::callback::hdc9f203da2c8fa8e

12 0x007f50ec04f8b5 in transmitRtpThread(void*)

13 0x007f521fd7a609 in start_thread

14 0x007f521fc9d133 in clone

2:

Warning: Unexpected item 0x7f21040c0ea0 dequeued from queue queue1 (refcounting problem?) 0:46:24.098985438 52690 0x2a33cc0 ERROR queue_dataflow gstqueue.c:1486:gst_queue_push_one: exit because we have no item in the queue rtpQueueDelay 0.497314 too large 1 2782.61 RTP queue 836 packets discarded for SSRC 1 hiSeqTx 58122 hiSeqAckendl 11191 seqNrOfNextRtp 58123 seqNrOfLastRtp 60763 diff 2641 thread '' panicked at 'Screamtx callback srcpad.push failed: Error', src/screamtx/imp.rs:360:22 note: run with RUST_BACKTRACE=1 environment variable to display a backtrace fatal runtime error: failed to initiate panic, error 5 Caught fatal signal - signo:6 code:-6 errno:0 addr:0x3e80000cdd2 Obtained 17 stack frames.

0 0x007f22d7c0fcf0 in __sigaction

1 0x007f22d7c6626b in pthread_kill

2 0x007f22d7c0fc46 in gsignal

3 0x007f22d7bf67fc in abort

4 0x007f2164590327 in std::sys::unix::abort_internal::hf940f5a54d8a80b4

5 0x007f216458ea42 in rust_panic

6 0x007f216458e8ca in std::panicking::rust_panic_with_hook::h5cafdc4b3bfd5528

7 0x007f216458e609 in std::panicking::begin_panichandler::$u7b$$u7b$closure$u7d$$u7d$::hf31c60f40775892c

8 0x007f216458d3dc in std::sys_common::backtrace::__rust_end_short_backtrace::h28a5c7be595826cd

9 0x007f216458e312 in rust_begin_unwind

10 0x007f21644d0ee3 in core::panicking::panic_fmt::h8fa27a0b37dd98b7

11 0x007f21644d1313 in core::result::unwrap_failed::h62abf61e411aaa08

12 0x007f21644ddb18 in core::result::Result$LT$T$C$E$GT$::expect::h54fe755c33896c5a

13 0x007f21644e594e in gstscream::screamtx::imp::callback::h11dff52b2a2873a4

14 0x007f216493d297 in transmitRtpThread(void*)

15 0x007f22d7c64402 in pthread_condattr_setpshared

16 0x007f22d7cf3590 in __xmknodat

3, exactly when a refresh happened:

refresh 1035.84 RTP queue 1 packets discarded for SSRC 1 refresh 1035.84 RTP queue 1 packets discarded for SSRC 1 refresh 1035.84 RTP queue 1 packets discarded for SSRC 1 refresh 1035.84 RTP queue 1 packets discarded for SSRC 1 refresh 1035.84 RTP queue 1 packets discarded for SSRC 1 refresh 1035.84 RTP queue 1 packets discarded for SSRC 1 refresh 1035.84 RTP queue 1 packets discarded for SSRC 1 refresh 1035.84 RTP queue 1 packets discarded for SSRC 1 Error: gst_pad_push: assertion 'GST_IS_BUFFER (buffer)' failed thread '' panicked at 'Screamtx callback srcpad.push failed: Error', src/screamtx/imp.rs:360:22 note: run with RUST_BACKTRACE=1 environment variable to display a backtrace fatal runtime error: failed to initiate panic, error 5 Caught fatal signal - signo:6 code:-6 errno:0 addr:0x3e8000068f4 refresh 1035.86 RTP queue 1 packets discarded for SSRC 1 refresh 1035.86 RTP queue 1 packets discarded for SSRC 1 refresh 1035.86 RTP queue 1 packets discarded for SSRC 1 refresh 1035.86 RTP queue 1 packets discarded for SSRC 1 refresh 1035.86 RTP queue 1 packets discarded for SSRC 1 refresh 1035.86 RTP queue 1 packets discarded for SSRC 1 refresh 1035.86 RTP queue 1 packets discarded for SSRC 1 refresh 1035.86 RTP queue 1 packets discarded for SSRC 1 refresh 1035.86 RTP queue 1 packets discarded for SSRC 1 refresh 1035.86 RTP queue 1 packets discarded for SSRC 1 refresh 1035.86 RTP queue 1 packets discarded for SSRC 1 refresh 1035.86 RTP queue 1 packets discarded for SSRC 1 refresh 1035.86 RTP queue 1 packets discarded for SSRC 1 refresh 1035.86 RTP queue 1 packets discarded for SSRC 1 refresh 1035.86 RTP queue 1 packets discarded for SSRC 1 refresh 1035.86 RTP queue 1 packets discarded for SSRC 1 refresh 1035.86 RTP queue 1 packets discarded for SSRC 1 Obtained 17 stack frames.

0 0x007f3c0f4e5cf0 in __sigaction

1 0x007f3c0f53c26b in pthread_kill

2 0x007f3c0f4e5c46 in gsignal

3 0x007f3c0f4cc7fc in abort

4 0x007f3aa81fe327 in std::sys::unix::abort_internal::hf940f5a54d8a80b4

5 0x007f3aa81fca42 in rust_panic

6 0x007f3aa81fc8ca in std::panicking::rust_panic_with_hook::h5cafdc4b3bfd5528

7 0x007f3aa81fc609 in std::panicking::begin_panichandler::$u7b$$u7b$closure$u7d$$u7d$::hf31c60f40775892c

8 0x007f3aa81fb3dc in std::sys_common::backtrace::__rust_end_short_backtrace::h28a5c7be595826cd

9 0x007f3aa81fc312 in rust_begin_unwind

10 0x007f3aa813eee3 in core::panicking::panic_fmt::h8fa27a0b37dd98b7

11 0x007f3aa813f313 in core::result::unwrap_failed::h62abf61e411aaa08

12 0x007f3aa814bb18 in core::result::Result$LT$T$C$E$GT$::expect::h54fe755c33896c5a

13 0x007f3aa815394e in gstscream::screamtx::imp::callback::h11dff52b2a2873a4

14 0x007f3b0803a28e in transmitRtpThread(void*)

15 0x007f3c0f53a402 in pthread_condattr_setpshared

16 0x007f3c0f5c9590 in __xmknodat

realuptime commented 9 months ago

Another over-night crash:

rtpQueueDelay 0.232422 too large 1 13919.3 RTP queue 118 packets discarded for SSRC 1 hiSeqTx 58159 hiSeqAckendl 0 seqNrOfNextRtp 58160 seqNrOfLastRtp 58277 diff 118 imp.rs: force_idr rc true enabled 1 rtpQueueDelay 0.253906 too large 1 Warning: Unexpected item 0x7f85e402b7e0 dequeued from queue queue1 (refcounting problem?) 13919.6 RTP queue 133 packets discarded for SSRC 1 hiSeqTx 58283 hiSeqAckendl 0 seqNrOfNextRtp 58284 seqNrOfLastRtp 58416 diff 133 3:52:01.238217250 ESC[36m130979ESC[00m 0x25fb8c0 ESC[31;01mERROR ESC[00m ESC[00m queue_dataflow gstqueue.c:1486:gst_queue_push_one:ESC[00m exit because we have no item in the queue thread '' panicked at 'Screamtx callback srcpad.push failed: Error', src/screamtx/imp.rs:360:22 note: run with RUST_BACKTRACE=1 environment variable to display a backtrace fatal runtime error: failed to initiate panic, error 5 Caught fatal signal - signo:6 code:-6 errno:0 addr:0x3e80001ffa3 Obtained 17 stack frames.

0 0x007f87d066ecf0 in __sigaction

1 0x007f87d06c526b in pthread_kill

2 0x007f87d066ec46 in gsignal

3 0x007f87d06557fc in abort

4 0x007f864c990327 in std::sys::unix::abort_internal::hf940f5a54d8a80b4

5 0x007f864c98ea42 in rust_panic

6 0x007f864c98e8ca in std::panicking::rust_panic_with_hook::h5cafdc4b3bfd5528

7 0x007f864c98e609 in std::panicking::begin_panichandler::$u7b$$u7b$closure$u7d$$u7d$::hf31c60f40775892c

8 0x007f864c98d3dc in std::sys_common::backtrace::__rust_end_short_backtrace::h28a5c7be595826cd

9 0x007f864c98e312 in rust_begin_unwind

10 0x007f864c8d0ee3 in core::panicking::panic_fmt::h8fa27a0b37dd98b7

11 0x007f864c8d1313 in core::result::unwrap_failed::h62abf61e411aaa08

12 0x007f864c8ddb18 in core::result::Result$LT$T$C$E$GT$::expect::h54fe755c33896c5a

13 0x007f864c8e594e in gstscream::screamtx::imp::callback::h11dff52b2a2873a4

14 0x007f86900b8297 in transmitRtpThread(void*)

15 0x007f87d06c3402 in pthread_condattr_setpshared

16 0x007f87d0752590 in __xmknodat

realuptime commented 9 months ago

and another:

summary 1628.9 Transmit rate = 8367kbps, PLR = 0.00%( 0.03%), RTT = 0.049s, Queue delay = 0.015s encoder_rate 8000 kbbps InputServer: Received Heartbeat InputServer: Received Heartbeat summary 1630.9 Transmit rate = 8264kbps, PLR = 0.00%( 0.03%), RTT = 0.049s, Queue delay = 0.015s encoder_rate 8000 kbbps refresh 1631 RTP queue 1 packets discarded for SSRC 1 refresh 1631 RTP queue 1 packets discarded for SSRC 1 refresh 1631 RTP queue 1 packets discarded for SSRC 1 refresh 1631 RTP queue 1 packets discarded for SSRC 1 refresh 1631 RTP queue 1 packets discarded for SSRC 1 refresh 1631 RTP queue 1 packets discarded for SSRC 1 refresh 1631 RTP queue 1 packets discarded for SSRC 1 refresh 1631 RTP queue 1 packets discarded for SSRC 1 refresh 1631 RTP queue 1 packets discarded for SSRC 1 refresh 1631 RTP queue 1 packets discarded for SSRC 1 refresh 1631 RTP queue 1 packets discarded for SSRC 1 refresh 1631 RTP queue 1 packets discarded for SSRC 1 refresh 1631 RTP queue 1 packets discarded for SSRC 1 refresh 1631 RTP queue 1 packets discarded for SSRC 1 Error: gst_pad_push: assertion 'GST_IS_BUFFER (buffer)' failed thread '' panicked at 'Screamtx callback srcpad.push failed: Error', src/screamtx/imp.rs:360:22 note: run with RUST_BACKTRACE=1 environment variable to display a backtrace fatal runtime error: failed to initiate panic, error 5 Caught fatal signal - signo:6 code:-6 errno:0 addr:0x3e800005ba0 refresh 1631.03 RTP queue 1 packets discarded for SSRC 1 refresh 1631.03 RTP queue 1 packets discarded for SSRC 1 refresh 1631.03 RTP queue 1 packets discarded for SSRC 1 refresh 1631.03 RTP queue 1 packets discarded for SSRC 1 refresh 1631.03 RTP queue 1 packets discarded for SSRC 1 refresh 1631.03 RTP queue 1 packets discarded for SSRC 1 refresh 1631.03 RTP queue 1 packets discarded for SSRC 1 refresh 1631.03 RTP queue 1 packets discarded for SSRC 1 refresh 1631.03 RTP queue 1 packets discarded for SSRC 1 refresh 1631.03 RTP queue 1 packets discarded for SSRC 1 refresh 1631.03 RTP queue 1 packets discarded for SSRC 1 refresh 1631.03 RTP queue 1 packets discarded for SSRC 1 refresh 1631.03 RTP queue 1 packets discarded for SSRC 1 refresh 1631.03 RTP queue 1 packets discarded for SSRC 1 refresh 1631.03 RTP queue 1 packets discarded for SSRC 1 refresh 1631.03 RTP queue 1 packets discarded for SSRC 1 refresh 1631.03 RTP queue 1 packets discarded for SSRC 1 refresh 1631.03 RTP queue 1 packets discarded for SSRC 1 refresh 1631.03 RTP queue 1 packets discarded for SSRC 1 refresh 1631.03 RTP queue 1 packets discarded for SSRC 1 refresh 1631.05 RTP queue 1 packets discarded for SSRC 1 refresh 1631.05 RTP queue 1 packets discarded for SSRC 1 refresh 1631.05 RTP queue 1 packets discarded for SSRC 1 refresh 1631.05 RTP queue 1 packets discarded for SSRC 1 refresh 1631.05 RTP queue 1 packets discarded for SSRC 1 refresh 1631.05 RTP queue 1 packets discarded for SSRC 1 refresh 1631.05 RTP queue 1 packets discarded for SSRC 1 refresh 1631.05 RTP queue 1 packets discarded for SSRC 1 refresh 1631.05 RTP queue 1 packets discarded for SSRC 1 refresh 1631.05 RTP queue 1 packets discarded for SSRC 1 refresh 1631.05 RTP queue 1 packets discarded for SSRC 1 refresh 1631.05 RTP queue 1 packets discarded for SSRC 1 refresh 1631.05 RTP queue 1 packets discarded for SSRC 1 refresh 1631.05 RTP queue 1 packets discarded for SSRC 1 refresh 1631.05 RTP queue 1 packets discarded for SSRC 1 refresh 1631.05 RTP queue 1 packets discarded for SSRC 1 refresh 1631.07 RTP queue 1 packets discarded for SSRC 1 refresh 1631.07 RTP queue 1 packets discarded for SSRC 1 refresh 1631.07 RTP queue 1 packets discarded for SSRC 1 refresh 1631.07 RTP queue 1 packets discarded for SSRC 1 refresh 1631.07 RTP queue 1 packets discarded for SSRC 1 refresh 1631.07 RTP queue 1 packets discarded for SSRC 1 refresh 1631.07 RTP queue 1 packets discarded for SSRC 1 refresh 1631.07 RTP queue 1 packets discarded for SSRC 1 refresh 1631.07 RTP queue 1 packets discarded for SSRC 1 refresh 1631.07 RTP queue 1 packets discarded for SSRC 1 refresh 1631.07 RTP queue 1 packets discarded for SSRC 1 refresh 1631.07 RTP queue 1 packets discarded for SSRC 1 refresh 1631.07 RTP queue 1 packets discarded for SSRC 1 refresh 1631.07 RTP queue 1 packets discarded for SSRC 1 refresh 1631.07 RTP queue 1 packets discarded for SSRC 1 refresh 1631.07 RTP queue 1 packets discarded for SSRC 1 refresh 1631.09 RTP queue 1 packets discarded for SSRC 1 refresh 1631.09 RTP queue 1 packets discarded for SSRC 1 refresh 1631.09 RTP queue 1 packets discarded for SSRC 1 refresh 1631.09 RTP queue 1 packets discarded for SSRC 1 refresh 1631.09 RTP queue 1 packets discarded for SSRC 1 Obtained 17 stack frames.

0 0x007fa46c7d2cf0 in __sigaction

1 0x007fa46c82926b in pthread_kill

2 0x007fa46c7d2c46 in gsignal

3 0x007fa46c7b97fc in abort

4 0x007fa2f9390327 in std::sys::unix::abort_internal::hf940f5a54d8a80b4

5 0x007fa2f938ea42 in rust_panic

6 0x007fa2f938e8ca in std::panicking::rust_panic_with_hook::h5cafdc4b3bfd5528

7 0x007fa2f938e609 in std::panicking::begin_panichandler::$u7b$$u7b$closure$u7d$$u7d$::hf31c60f40775892c

8 0x007fa2f938d3dc in std::sys_common::backtrace::__rust_end_short_backtrace::h28a5c7be595826cd

9 0x007fa2f938e312 in rust_begin_unwind

10 0x007fa2f92d0ee3 in core::panicking::panic_fmt::h8fa27a0b37dd98b7

11 0x007fa2f92d1313 in core::result::unwrap_failed::h62abf61e411aaa08

12 0x007fa2f92ddb18 in core::result::Result$LT$T$C$E$GT$::expect::h54fe755c33896c5a

13 0x007fa2f92e594e in gstscream::screamtx::imp::callback::h11dff52b2a2873a4

14 0x007fa30c04c297 in transmitRtpThread(void*)

15 0x007fa46c827402 in pthread_condattr_setpshared

16 0x007fa46c8b6590 in __xmknodat

jacobt21 commented 9 months ago

Pushed the fix