Crash (SIGSEGV) when SRT stream loses packets

[Phygon]

Describe the bug

OME randomly crashes with a segmentation fault when using the SRT provider and LLHLS publisher. This happened on our production server and it could be reproduced on our test server by randomly dropping UDP packets.

To Reproduce

1) Send one or more SRT streams to OME, for example via ffmpeg. 2) watch LLHLS stream 3) simulate bad connection or network congestion by randomly dropping UDP packets on the sender: iptables -A OUTPUT -p udp --dport 9999 -m statistic --mode random --probability 0.1 -j DROP

Expected behavior

No crash, please.

Logs

log output:

[2023-06-27 04:09:25.368] W [SPSRT-s9999:10299] MPEGTS_DEPACKETIZER | mpegts_depacketizer.cpp:84   | An out-of-order packet was received.(PID : 256 Expected : 0, Received : 7
[2023-06-27 04:09:25.388] I [Dech264:10313] FFmpeg | third_parties.cpp:119  | [AVCodecContext: 0x7ff2e6e4f000] concealing 763 DC, 763 AC, 763 MV errors in P frame
[2023-06-27 04:09:27.236] W [SPSRT-s9999:10299] MPEGTS_DEPACKETIZER | mpegts_depacketizer.cpp:84   | An out-of-order packet was received.(PID : 256 Expected : 9, Received : 0
[2023-06-27 04:09:27.282] I [Dech264:10313] FFmpeg | third_parties.cpp:119  | [AVCodecContext: 0x7ff2e6e4f000] concealing 547 DC, 547 AC, 547 MV errors in P frame
[2023-06-27 04:09:28.000] W [SPSRT-s9999:10299] MPEGTS_DEPACKETIZER | mpegts_depacketizer.cpp:84   | An out-of-order packet was received.(PID : 256 Expected : 4, Received : 11
[2023-06-27 04:09:29.038] E [Dech264:10313] FFmpeg | third_parties.cpp:111  | [AVCodecContext: 0x7ff2e6e4f000] cabac decode of qscale diff failed at 20 7
[2023-06-27 04:09:29.038] E [Dech264:10313] FFmpeg | third_parties.cpp:111  | [AVCodecContext: 0x7ff2e6e4f000] error while decoding MB 20 7, bytestream 10481
[2023-06-27 04:09:29.038] I [Dech264:10313] FFmpeg | third_parties.cpp:119  | [AVCodecContext: 0x7ff2e6e4f000] concealing 3069 DC, 3069 AC, 3069 MV errors in P frame
[2023-06-27 04:09:29.884] W [SPSRT-s9999:10299] MPEGTS_DEPACKETIZER | mpegts_depacketizer.cpp:84   | An out-of-order packet was received.(PID : 256 Expected : 4, Received : 11
[2023-06-27 04:09:29.921] E [Dech264:10313] FFmpeg | third_parties.cpp:111  | [AVCodecContext: 0x7ff2e6e4f000] left block unavailable for requested intra mode
[2023-06-27 04:09:29.921] E [Dech264:10313] FFmpeg | third_parties.cpp:111  | [AVCodecContext: 0x7ff2e6e4f000] error while decoding MB 0 35, bytestream 1165
[2023-06-27 04:09:29.921] I [Dech264:10313] FFmpeg | third_parties.cpp:119  | [AVCodecContext: 0x7ff2e6e4f000] concealing 849 DC, 849 AC, 849 MV errors in P frame
[2023-06-27 04:09:32.448] W [SPSRT-s9999:10299] MPEGTS_DEPACKETIZER | mpegts_depacketizer.cpp:84   | An out-of-order packet was received.(PID : 257 Expected : 9, Received : 11
[2023-06-27 04:09:32.449] E [Decaac:10314] FFmpeg | third_parties.cpp:111  | [AVCodecContext: 0x7ff2e6e4f800] TYPE_FIL: Input buffer exhausted before END element found
[2023-06-27 04:09:32.449] E [Decaac:10314] Transcoder | decoder_aac.cpp:166  | An error occurred while sending a packet for decoding: Unhandled error (-1094995529:Invalid data found when processing input) 
[2023-06-27 04:09:32.449] E [Decaac:10314] FFmpeg | third_parties.cpp:111  | [AVCodecContext: 0x7ff2e6e4f800] channel element 2.15 is not allocated
[2023-06-27 04:09:32.449] E [Decaac:10314] Transcoder | decoder_aac.cpp:166  | An error occurred while sending a packet for decoding: Unhandled error (-1094995529:Invalid data found when processing input) 
[2023-06-27 04:09:32.451] C [AW-LLHLS0:10306] OvenMediaEngine | signals.cpp:120  | OME received signal 11 (SIGSEGV), interrupt.

gdb output (separate run):

Thread 24 "AW-LLHLS0" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffe62fc700 (LWP 7669)]
bmff::Packager::Samples::AppendSample (media_packet=std::shared_ptr<const MediaPacket> (use count 2, weak count 0) = {...}, this=<optimized out>) at projects/base/ovlibrary/./data.h:113
113     projects/base/ovlibrary/./data.h: No such file or directory.
(gdb) bt
#0  bmff::Packager::Samples::AppendSample (media_packet=std::shared_ptr<const MediaPacket> (use count 2, weak count 0) = {...}, this=<optimized out>) at projects/base/ovlibrary/./data.h:113
#1  bmff::FMP4Packager::AppendSample (this=<optimized out>, media_packet=std::shared_ptr<const MediaPacket> (use count 2, weak count 0) = {...}) at projects/modules/containers/bmff/fmp4_packager/fmp4_packager.cpp:266
#2  0x000055555569c41c in LLHlsStream::AppendMediaPacket (this=0x7fffe0f89710, media_packet=std::shared_ptr<MediaPacket> (use count 2, weak count 0) = {...}) at /usr/include/c++/8/ext/atomicity.h:96
#3  0x000055555590cd0a in pub::ApplicationWorker::WorkerThread (this=0x7fffec2a7990) at projects/base/publisher/application.cpp:97
#4  0x00007ffff6ba2b2f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007ffff7f83fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#6  0x00007ffff6882eff in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Server (please complete the following information):

• OS: Debian 10.13
• OvenMediaEngine Version: v0.15.13 (build from tar archive)

[getroot]

Perfect bug report! I'll fix this soon. Thank you.

[getroot]

I solved this problem and committed it. Could you please check if the issue is fixed in the latest master branch?

[Phygon]

No change.

Version: OvenMediaEngine v0.15.13 (v0.15.13-20-gf121868a)

[2023-06-29 19:48:43.273] W [SPSRT-s9999:15695] MPEGTS_DEPACKETIZER | mpegts_depacketizer.cpp:83   | An out-of-order packet was received.(PID : 256 Expected : 4, Received : 7
[2023-06-29 19:48:43.282] E [Dech264:15736] FFmpeg | third_parties.cpp:111  | [AVCodecContext: 0x7ff219c6e000] error while decoding MB 18 41, bytestream -5
[2023-06-29 19:48:43.282] I [Dech264:15736] FFmpeg | third_parties.cpp:119  | [AVCodecContext: 0x7ff219c6e000] concealing 351 DC, 351 AC, 351 MV errors in P frame
[2023-06-29 19:48:44.234] W [SPSRT-s9999:15695] MPEGTS_DEPACKETIZER | mpegts_depacketizer.cpp:83   | An out-of-order packet was received.(PID : 257 Expected : 1, Received : 3
[2023-06-29 19:48:44.235] E [Decaac:15737] FFmpeg | third_parties.cpp:111  | [AVCodecContext: 0x7ff219c6e800] channel element 3.11 is not allocated
[2023-06-29 19:48:44.235] E [Decaac:15737] Transcoder | decoder_aac.cpp:166  | An error occurred while sending a packet for decoding: Unhandled error (-1094995529:Invalid data found when processing input) 
[2023-06-29 19:48:44.235] E [Decaac:15737] FFmpeg | third_parties.cpp:111  | [AVCodecContext: 0x7ff219c6e800] Reserved bit set.
[2023-06-29 19:48:44.235] E [Decaac:15737] FFmpeg | third_parties.cpp:111  | [AVCodecContext: 0x7ff219c6e800] Prediction is not allowed in AAC-LC.
[2023-06-29 19:48:44.235] E [Decaac:15737] Transcoder | decoder_aac.cpp:166  | An error occurred while sending a packet for decoding: Unhandled error (-1094995529:Invalid data found when processing input) 
[2023-06-29 19:48:44.239] C [AW-LLHLS0:15702] OvenMediaEngine | signals.cpp:120  | OME received signal 11 (SIGSEGV), interrupt.

[getroot]

Sorry, I committed wrongly. please test again

[Phygon]

Looks good. It was running for an hour and so far no crash.

[Phygon]

No crash over the weekend. Thanks! I'll close this.