search

sipwise / rtpengine

The Sipwise media proxy for Kamailio
GNU General Public License v3.0
778 stars 368 forks source link

Transcoding crash with ptime change #1479

Open szcom opened 2 years ago

szcom commented 2 years ago

Rtpengine is at 10.2 (ad76db6b321da7f0d9b11f5c89d229f8bf4ff4d7). Following sequence causing it to crash:

0s. Alice(AMR) to Bob(PCMA) - agree on a=ptime:20, transcoding is on 10s. Bob puts Alice onhold with re-INVITE a=sendonly and a=ptime:10 20s. Bob resumes with re-INVITE a=sendrecv and a=ptime:10 21s. Bob puts Alice onhold with re-INVITE a=sendonly and a=ptime:10 - rpengine's gone, no response

rtpengine: codeclib.c:2251: packetizer_amr: Assertion `output->len >= pkt->size + 3' failed.

Guessing here: after 1st re-INVITE, Bob starts to send 10ms PCMA packets but transcoding still expects 20ms long payload. Assertion `output->len >= pkt->size + 3' failed. appears after the 1st reinvite and eventually we get segfault.

Working on getting a stack trace.

rfuchs commented 2 years ago

Stack trace and the contents of both output and pkt would be of interest

szcom commented 2 years ago

Stack looks corrupt:

(gdb) where
#0  0x00007f0a18ce2611 in ?? ()
#1  0x0000000000000020 in ?? ()
#2  0x0000000000000000 in ?? ()
(gdb) i th
  Id   Target Id                      Frame
* 1    Thread 0x7f0a06ff7700 (LWP 21) 0x00007f0a18ce2611 in ?? ()
  2    Thread 0x7f0a0c656700 (LWP 11) futex_abstimed_wait_cancelable (private=0, abstime=0x7f0a0c655040, expected=0, futex_word=0x555e54ae6e98 <ice_agents_timer_thread+88>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  3    Thread 0x7f0a0c455700 (LWP 12) 0x00007f0a18dae819 in fts_build (sp=0x7f0a0d25a010, type=0) at ../sysdeps/wordsize-64/../../io/fts.c:735
  4    Thread 0x7f0a07fff700 (LWP 9)  0x00007f0a18d86720 in __getpwnam_r (name=0x55e7d950 <error: Cannot access memory at address 0x55e7d950>, resbuf=0x7f0a07ffe060, buffer=0x555e55eca170 "\003", buflen=93863656858080,
    result=0x5f5e100) at ../nss/getXXbyYY_r.c:275
  5    Thread 0x7f0a0ca58700 (LWP 8)  0x00007f0a18cf84cc in __libc_longjmp (env=0x0, val=212168448) at ../sysdeps/x86/longjmp.c:44
  6    Thread 0x7f0a0c254700 (LWP 13) 0x00007f0a18db97ef in __libc_ifunc_impl_list (name=<optimized out>, array=0x3, max=<optimized out>) at ../sysdeps/x86_64/multiarch/ifunc-impl-list.c:497
  7    Thread 0x7f0977fff700 (LWP 44) futex_abstimed_wait_cancelable (private=0, abstime=0x7f0977ffe040, expected=0, futex_word=0x555e54ae70d8 <send_timer_thread+88>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  8    Thread 0x7f0a06df6700 (LWP 22) futex_abstimed_wait_cancelable (private=0, abstime=0x7f0a06df5040, expected=0, futex_word=0x555e54ae7158 <media_player_thread+88>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  9    Thread 0x7f0a07bfd700 (LWP 15) 0x00007f0a18db97ef in __libc_ifunc_impl_list (name=<optimized out>, array=0x3, max=<optimized out>) at ../sysdeps/x86_64/multiarch/ifunc-impl-list.c:497
  10   Thread 0x7f0a05bed700 (LWP 31) futex_abstimed_wait_cancelable (private=0, abstime=0x7f0a05bec040, expected=0, futex_word=0x555e54ae7158 <media_player_thread+88>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  11   Thread 0x7f0a05fef700 (LWP 29) futex_abstimed_wait_cancelable (private=0, abstime=0x7f0a05fee040, expected=0, futex_word=0x555e54ae70d8 <send_timer_thread+88>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  12   Thread 0x7f0a079fc700 (LWP 16) 0x00007f0a18db97ef in __libc_ifunc_impl_list (name=<optimized out>, array=0x3, max=<optimized out>) at ../sysdeps/x86_64/multiarch/ifunc-impl-list.c:497
  13   Thread 0x7f0a04fe7700 (LWP 37) futex_abstimed_wait_cancelable (private=0, abstime=0x7f0a04fe6040, expected=0, futex_word=0x555e54ae7158 <media_player_thread+88>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  14   Thread 0x7f0a043e1700 (LWP 43) futex_abstimed_wait_cancelable (private=0, abstime=0x7f0a043e0040, expected=0, futex_word=0x555e54ae7158 <media_player_thread+88>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  15   Thread 0x7f0a055ea700 (LWP 34) futex_abstimed_wait_cancelable (private=0, abstime=0x7f0a055e9040, expected=0, futex_word=0x555e54ae7158 <media_player_thread+88>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  16   Thread 0x7f0a057eb700 (LWP 33) futex_abstimed_wait_cancelable (private=0, abstime=0x7f0a057ea040, expected=0, futex_word=0x555e54ae6ff8 <codec_timers_thread+88>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  17   Thread 0x7f0a0c857700 (LWP 10) 0x00007f0a18d86720 in __getpwnam_r (name=0x33 <error: Cannot access memory at address 0x33>, resbuf=0x7fffbc1ac0ff, buffer=0x7f09f8000b40 "", buflen=139681136468072, result=0x1dcd6500)
    at ../nss/getXXbyYY_r.c:275
  18   Thread 0x7f0a04de6700 (LWP 38) futex_abstimed_wait_cancelable (private=0, abstime=0x7f0a04de5040, expected=0, futex_word=0x555e54ae70dc <send_timer_thread+92>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  19   Thread 0x7f0a047e3700 (LWP 41) futex_abstimed_wait_cancelable (private=0, abstime=0x7f0a047e2040, expected=0, futex_word=0x555e54ae70d8 <send_timer_thread+88>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  20   Thread 0x7f0a065f2700 (LWP 26) futex_abstimed_wait_cancelable (private=0, abstime=0x7f0a065f1040, expected=0, futex_word=0x555e54ae70d8 <send_timer_thread+88>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  21   Thread 0x7f0a051e8700 (LWP 36) futex_abstimed_wait_cancelable (private=0, abstime=0x7f0a051e7040, expected=0, futex_word=0x555e54ae6ff8 <codec_timers_thread+88>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  22   Thread 0x7f0a06bf5700 (LWP 23) futex_abstimed_wait_cancelable (private=0, abstime=0x7f0a06bf4040, expected=0, futex_word=0x555e54ae70d8 <send_timer_thread+88>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  23   Thread 0x7f0a045e2700 (LWP 42) futex_abstimed_wait_cancelable (private=0, abstime=0x7f0a045e1040, expected=0, futex_word=0x555e54ae6ff8 <codec_timers_thread+88>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  24   Thread 0x7f0a073f9700 (LWP 19) 0x00007f0a18db97ef in __libc_ifunc_impl_list (name=<optimized out>, array=0x3, max=<optimized out>) at ../sysdeps/x86_64/multiarch/ifunc-impl-list.c:497
  25   Thread 0x7f0a061f0700 (LWP 28) futex_abstimed_wait_cancelable (private=0, abstime=0x7f0a061ef040, expected=0, futex_word=0x555e54ae7158 <media_player_thread+88>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  26   Thread 0x7f0a05dee700 (LWP 30) futex_abstimed_wait_cancelable (private=0, abstime=0x7f0a05ded040, expected=0, futex_word=0x555e54ae6ff8 <codec_timers_thread+88>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  27   Thread 0x7f0a077fb700 (LWP 17) 0x00007f0a18db97ef in __libc_ifunc_impl_list (name=<optimized out>, array=0x3, max=<optimized out>) at ../sysdeps/x86_64/multiarch/ifunc-impl-list.c:497
  28   Thread 0x7f0977dfe700 (LWP 45) futex_abstimed_wait_cancelable (private=0, abstime=0x7f0977dfd040, expected=0, futex_word=0x555e54ae6ff8 <codec_timers_thread+88>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  29   Thread 0x7f0a04be5700 (LWP 39) futex_abstimed_wait_cancelable (private=0, abstime=0x7f0a04be4040, expected=0, futex_word=0x555e54ae6ff8 <codec_timers_thread+88>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
--Type <RET> for more, q to quit, c to continue without paging--
  30   Thread 0x7f0a0da8c540 (LWP 1)  0x00007f0a18d86720 in __getpwnam_r (name=0x0, resbuf=0x555e54a96900 <send_timer_loop>, buffer=0x7f0a0c655030 "Hn\256T^U", buflen=93863635654096, result=0x5f5e100) at ../nss/getXXbyYY_r.c:275
  31   Thread 0x7f0a069f4700 (LWP 24) futex_abstimed_wait_cancelable (private=0, abstime=0x7f0a069f3040, expected=0, futex_word=0x555e54ae6ff8 <codec_timers_thread+88>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  32   Thread 0x7f0a075fa700 (LWP 18) 0x00007f0a18db97ef in __libc_ifunc_impl_list (name=<optimized out>, array=0x3, max=<optimized out>) at ../sysdeps/x86_64/multiarch/ifunc-impl-list.c:497
  33   Thread 0x7f0a063f1700 (LWP 27) futex_abstimed_wait_cancelable (private=0, abstime=0x7f0a063f0040, expected=0, futex_word=0x555e54ae6ff8 <codec_timers_thread+88>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  34   Thread 0x7f0a071f8700 (LWP 20) 0x00007f0a18db97ef in __libc_ifunc_impl_list (name=<optimized out>, array=0x3, max=<optimized out>) at ../sysdeps/x86_64/multiarch/ifunc-impl-list.c:497
  35   Thread 0x7f0a067f3700 (LWP 25) futex_abstimed_wait_cancelable (private=0, abstime=0x7f0a067f2040, expected=0, futex_word=0x555e54ae7158 <media_player_thread+88>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  36   Thread 0x7f0a07dfe700 (LWP 14) 0x00007f0a18db97ef in __libc_ifunc_impl_list (name=<optimized out>, array=0x3, max=<optimized out>) at ../sysdeps/x86_64/multiarch/ifunc-impl-list.c:497
  37   Thread 0x7f0a059ec700 (LWP 32) futex_abstimed_wait_cancelable (private=0, abstime=0x7f0a059eb040, expected=0, futex_word=0x555e54ae70dc <send_timer_thread+92>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  38   Thread 0x7f0a053e9700 (LWP 35) futex_abstimed_wait_cancelable (private=0, abstime=0x7f0a053e8040, expected=0, futex_word=0x555e54ae70d8 <send_timer_thread+88>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  39   Thread 0x7f0a049e4700 (LWP 40) futex_abstimed_wait_cancelable (private=0, abstime=0x7f0a049e3040, expected=0, futex_word=0x555e54ae7158 <media_player_thread+88>) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
rfuchs commented 2 years ago

That would be surprising because a failed assertion leads to a deliberate core (through abort()) and in this function the assertion is made before anything affecting the stack is done.

Make sure the core file matches the binary that generated it (the exact same version of it). Install the -dbgsym package (if on Debian) and pull the backtrace again.

szcom commented 2 years ago

checked md5s - binary at run time and gdb time are same.

szcom commented 2 years ago

looks like thread 40 - is the only one mentioning codeclib.c in its backtrace:

Thread 40 (Thread 0x7f32cd4d0700 (LWP 20)):
#0  0x00007f32d8a8bef2 in ?? () from /usr/lib/x86_64-linux-gnu/libvo-amrwbenc.so.0
#1  0x00007f32d8a8e92e in ?? () from /usr/lib/x86_64-linux-gnu/libvo-amrwbenc.so.0
#2  0x00007f32d8a8ea03 in ?? () from /usr/lib/x86_64-linux-gnu/libvo-amrwbenc.so.0
#3  0x00007f32d8a7faec in E_IF_encode () from /usr/lib/x86_64-linux-gnu/libvo-amrwbenc.so.0
#4  0x00007f32dbff5d6e in ?? () from /usr/lib/x86_64-linux-gnu/libavcodec.so.58
#5  0x00007f32dbdb861a in avcodec_encode_audio2 () from /usr/lib/x86_64-linux-gnu/libavcodec.so.58
#6  0x00007f32dbdb9079 in ?? () from /usr/lib/x86_64-linux-gnu/libavcodec.so.58
#7  0x00007f32dbdb91c2 in avcodec_send_frame () from /usr/lib/x86_64-linux-gnu/libavcodec.so.58
#8  0x000056459db741be in avc_encoder_input (enc=0x7f329059e250, frame=0x7f32cd4cc2b8) at codeclib.c:1395
#9  0x000056459db78555 in encoder_input_data (enc=enc@entry=0x7f329059e250, frame=<optimized out>, callback=callback@entry=0x56459db4dda0 <packet_encoded_rtp>,
    u1=u1@entry=0x7f32a0608760, u2=u2@entry=0x7f32cd4cc688) at codeclib.c:1470
#10 0x000056459db7867d in encoder_fifo_flush (u2=<optimized out>, u1=<optimized out>, callback=<optimized out>, enc=<optimized out>) at codeclib.c:1511
#11 encoder_input_fifo (enc=0x7f329059e250, frame=<optimized out>, callback=0x56459db4dda0 <packet_encoded_rtp>, u1=u1@entry=0x7f32a0608760, u2=u2@entry=0x7f32cd4cc688)
    at codeclib.c:1525
#12 0x000056459db4d2ba in packet_decoded_common (frame=<optimized out>, u1=<optimized out>, u2=0x7f32cd4cc688, input_func=0x56459db78610 <encoder_input_fifo>,
    decoder=<optimized out>) at codec.c:2751
#13 0x000056459db724c7 in __decoder_input_data (dec=0x7f32a0095540, data=<optimized out>, ts=<optimized out>, ptime=0x7f32cd4cc750,
    callback=0x56459db4dd60 <packet_decoded_fifo>, u1=0x7f32a0608760, u2=0x7f32cd4cc688) at codeclib.c:902
#14 0x000056459db777ed in decoder_input_data_ptime (dec=<optimized out>, data=<optimized out>, ts=<optimized out>, ptime=ptime@entry=0x7f32cd4cc750,
    callback=<optimized out>, u1=u1@entry=0x7f32a0608760, u2=0x7f32cd4cc688) at codeclib.c:925
#15 0x000056459db4c4c2 in __rtp_decode (ch=ch@entry=0x7f32a0608760, input_ch=input_ch@entry=0x7f32a0608760, packet=packet@entry=0x7f32a4005f70,
    mp=mp@entry=0x7f32cd4cc688) at codec.c:2770
#16 0x000056459db4e737 in packet_decode (ch=0x7f32a0608760, input_ch=0x7f32a0608760, packet=0x7f32a4005f70, mp=0x7f32cd4cc688) at codec.c:2803
#17 0x000056459db50358 in __handler_func_sequencer (mp=0x7f32cd4cc688, packet=0x7f32a4005f70) at codec.c:1493
#18 0x000056459db38c26 in stream_packet (phc=phc@entry=0x7f32cd4cc610) at media_socket.c:2405
#19 0x000056459db3a858 in stream_fd_readable (fd=884, p=0x7f329c2f3e00, u=<optimized out>) at media_socket.c:2550
#20 0x000056459dae20c3 in poller_poll (p=0x56459f58c1a0, timeout=<optimized out>) at poller.c:440
#21 0x000056459dae259d in poller_loop2 (d=0x56459f58c1a0) at poller.c:612
#22 0x000056459dae26f2 in thread_detach_func (d=0x56459f55ec00) at aux.c:232
#23 0x00007f32db30dfa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#24 0x00007f32db23e4cf in __libc_ifunc_impl_list (name=<optimized out>, array=0x7f32cd4d0700, max=<optimized out>) at ../sysdeps/x86_64/multiarch/ifunc-impl-list.c:385
#25 0x0000000000000000 in ?? ()
szcom commented 2 years ago

at line codeclib.c:1394: avcodec_send_frame(enc->u.avc.avcctx, *frame);

#8  0x000056459db741be in avc_encoder_input (enc=0x7f329059e250, frame=0x7f32cd4cc2b8) at codeclib.c:1395
1395    codeclib.c: No such file or directory.
(gdb) p enc->u.avc.avcctx
$1 = (AVCodecContext *) 0x7f32a4490400
(gdb) p *enc->u.avc.avcctx
$2 = {av_class = 0x7f32dc8476c0, log_level_offset = 0, codec_type = AVMEDIA_TYPE_AUDIO, codec = 0x7f32dc87b1e0, codec_id = AV_CODEC_ID_AMR_WB, codec_tag = 0,
  priv_data = 0x7f32a45e7300, internal = 0x7f32a40c2080, opaque = 0x0, bit_rate = 14250, bit_rate_tolerance = 0, global_quality = 0, compression_level = -1, flags = 0,
  flags2 = 0, extradata = 0x0, extradata_size = 0, time_base = {num = 1, den = 16000}, ticks_per_frame = 1, delay = 0, width = 0, height = 0, coded_width = 0,
  coded_height = 0, gop_size = 0, pix_fmt = AV_PIX_FMT_NONE, draw_horiz_band = 0x0, get_format = 0x7f32dbd6bbb0 <avcodec_default_get_format>, max_b_frames = 0,
  b_quant_factor = 0, b_frame_strategy = 0, b_quant_offset = 0, has_b_frames = 0, mpeg_quant = 0, i_quant_factor = 0, i_quant_offset = 0, lumi_masking = 0,
  temporal_cplx_masking = 0, spatial_cplx_masking = 0, p_masking = 0, dark_masking = 0, slice_count = 0, prediction_method = 0, slice_offset = 0x0,
  sample_aspect_ratio = {num = 0, den = 1}, me_cmp = 0, me_sub_cmp = 0, mb_cmp = 0, ildct_cmp = 0, dia_size = 0, last_predictor_count = 0, pre_me = 0, me_pre_cmp = 0,
  pre_dia_size = 0, me_subpel_quality = 0, me_range = 0, slice_flags = 0, mb_decision = 0, intra_matrix = 0x0, inter_matrix = 0x0, scenechange_threshold = 0,
  noise_reduction = 0, intra_dc_precision = 0, skip_top = 0, skip_bottom = 0, mb_lmin = 0, mb_lmax = 0, me_penalty_compensation = 0, bidir_refine = 0, brd_scale = 0,
  keyint_min = 0, refs = 0, chromaoffset = 0, mv0_threshold = 0, b_sensitivity = 0, color_primaries = AVCOL_PRI_RESERVED0, color_trc = AVCOL_TRC_RESERVED0,
  colorspace = AVCOL_SPC_RGB, color_range = AVCOL_RANGE_UNSPECIFIED, chroma_sample_location = AVCHROMA_LOC_UNSPECIFIED, slices = 0, field_order = AV_FIELD_UNKNOWN,
  sample_rate = 16000, channels = 1, sample_fmt = AV_SAMPLE_FMT_S16, frame_size = 320, frame_number = 1101, block_align = 0, cutoff = 0, channel_layout = 4,
  request_channel_layout = 0, audio_service_type = AV_AUDIO_SERVICE_TYPE_MAIN, request_sample_fmt = AV_SAMPLE_FMT_NONE,
  get_buffer2 = 0x7f32dbd6c440 <avcodec_default_get_buffer2>, refcounted_frames = 0, qcompress = 0, qblur = 0, qmin = 0, qmax = 0, max_qdiff = 0, rc_buffer_size = 0,
  rc_override_count = 0, rc_override = 0x0, rc_max_rate = 0, rc_min_rate = 0, rc_max_available_vbv_use = 0, rc_min_vbv_overflow_use = 0,
  rc_initial_buffer_occupancy = 0, coder_type = 0, context_model = 0, frame_skip_threshold = 0, frame_skip_factor = 0, frame_skip_exp = 0, frame_skip_cmp = 0,
  trellis = 0, min_prediction_order = -1, max_prediction_order = -1, timecode_frame_start = 0, rtp_callback = 0x0, rtp_payload_size = 0, mv_bits = 0, header_bits = 0,
  i_tex_bits = 0, p_tex_bits = 0, i_count = 0, p_count = 0, skip_count = 0, misc_bits = 0, frame_bits = 0, stats_out = 0x0, stats_in = 0x0, workaround_bugs = 0,
  strict_std_compliance = 0, error_concealment = 0, debug = 0, err_recognition = 0, reordered_opaque = -9223372036854775808, hwaccel = 0x0, hwaccel_context = 0x0,
  error = {0, 0, 0, 0, 0, 0, 0, 0}, dct_algo = 0, idct_algo = 0, bits_per_coded_sample = 0, bits_per_raw_sample = 0, lowres = 0, coded_frame = 0x7f32a41be800,
  thread_count = 1, thread_type = 3, active_thread_type = 0, thread_safe_callbacks = 0, execute = 0x7f32dc1f92a0 <avcodec_default_execute>,
  execute2 = 0x7f32dc1f9360 <avcodec_default_execute2>, nsse_weight = 0, profile = -99, level = -99, skip_loop_filter = AVDISCARD_DEFAULT,
  skip_idct = AVDISCARD_DEFAULT, skip_frame = AVDISCARD_DEFAULT, subtitle_header = 0x0, subtitle_header_size = 0, vbv_delay = 0, side_data_only_packets = 1,
  initial_padding = 80, framerate = {num = 0, den = 1}, sw_pix_fmt = AV_PIX_FMT_NONE, pkt_timebase = {num = 0, den = 1}, codec_descriptor = 0x7f32dc860070,
  pts_correction_num_faulty_pts = 0, pts_correction_num_faulty_dts = 0, pts_correction_last_pts = -9223372036854775808, pts_correction_last_dts = -9223372036854775808,
  sub_charenc = 0x0, sub_charenc_mode = 0, skip_alpha = 0, seek_preroll = 0, debug_mv = 0, chroma_intra_matrix = 0x0, dump_separator = 0x0, codec_whitelist = 0x0,
  properties = 0, coded_side_data = 0x0, nb_coded_side_data = 0, hw_frames_ctx = 0x0, sub_text_format = 0, trailing_padding = 0, max_pixels = 2147483647,
  hw_device_ctx = 0x0, hwaccel_flags = 0, apply_cropping = 0, extra_hw_frames = 0}
(gdb)
szcom commented 2 years ago 
p **frame
$4 = {data = {0x7f32a448fcc0 "\b", 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, linesize = {640, 0, 0, 0, 0, 0, 0, 0}, extended_data = 0x7f32a4130400, width = 0, height = 0,
  nb_samples = 320, format = 1, key_frame = 1, pict_type = AV_PICTURE_TYPE_NONE, sample_aspect_ratio = {num = 0, den = 1}, pts = 352320,
  pkt_pts = -9223372036854775808, pkt_dts = -9223372036854775808, coded_picture_number = 0, display_picture_number = 0, quality = 0, opaque = 0x0, error = {0, 0, 0, 0,
    0, 0, 0, 0}, repeat_pict = 0, interlaced_frame = 0, top_field_first = 0, palette_has_changed = 0, reordered_opaque = 0, sample_rate = 16000, channel_layout = 4,
  buf = {0x7f32a43d93c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, extended_buf = 0x0, nb_extended_buf = 0, side_data = 0x0, nb_side_data = 0, flags = 0,
  color_range = AVCOL_RANGE_UNSPECIFIED, color_primaries = AVCOL_PRI_UNSPECIFIED, color_trc = AVCOL_TRC_UNSPECIFIED, colorspace = AVCOL_SPC_UNSPECIFIED,
  chroma_location = AVCHROMA_LOC_UNSPECIFIED, best_effort_timestamp = -9223372036854775808, pkt_pos = -1, pkt_duration = 0, metadata = 0x0, decode_error_flags = 0,
  channels = 1, pkt_size = -1, qscale_table = 0x0, qstride = 0, qscale_type = 0, qp_table_buf = 0x0, hw_frames_ctx = 0x0, opaque_ref = 0x0, crop_top = 0,
  crop_bottom = 0, crop_left = 0, crop_right = 0, private_ref = 0x0}
szcom commented 2 years ago

Here is the PCMA payload coming from Bob for transcoding - and it is 20ms long apparently:

#16 0x000056459db4e737 in packet_decode (ch=0x7f32a0608760, input_ch=0x7f32a0608760, packet=0x7f32a4005f70, mp=0x7f32cd4cc688) at codec.c:2803
2803    in codec.c
(gdb) p *packet.payload
$11 = {
  s = 0x7f3290369730 "UU\325\325\325UUU\325U\325\325\325U\325\325U\325UU\325UUUUUU\325", 'U' <repeats 40 times>, "\325UUUUUU\325UUUU\325UUUU\325UUUUU\325", 'U' <repeats 15 times>, "\325UUUUUU\325\325UUU\325\325UUU\325UU\325U\325\325\325\325U", '\325' <repeats 16 times>, "UUUUU\325UUU\325", len = 160}

Looks like thread 40 is doing ok wrt ptime. But the one that made it crash - does not have a stack trace to follow. For example stack trace of Thread 5:

(gdb) bt
#0  0x00007f32db20b720 in __getpwnam_r (name=0x0,
    resbuf=0x56459db5e900 <send_timer_loop>,
    buffer=0x7f32ce8d9030 "H\356\272\235EV", buflen=94856998655440,
    result=0x5f5e100) at ../nss/getXXbyYY_r.c:275
#1  0xffffffffffffffff in ?? ()
#2  0xfffffffe7ffffbdb in ?? ()
#3  0xffffffffffffffff in ?? ()
#4  0xffffffffffffffff in ?? ()
#5  0xffffffffffffffff in ?? ()
#6  0xffffffffffffffff in ?? ()
#7  0xffffffffffffffff in ?? ()
#8  0xffffffffffffffff in ?? ()
#9  0xffffffffffffffff in ?? ()
#10 0xffffffffffffffff in ?? ()
#11 0xffffffffffffffff in ?? ()
#12 0xffffffffffffffff in ?? ()
#13 0xffffffffffffffff in ?? ()
#14 0xffffffffffffffff in ?? ()
#15 0xffffffffffffffff in ?? ()
#16 0xffffffffffffffff in ?? ()
#17 0xffffffffffffffff in ?? ()
#18 0x00007f32ded5f530 in ?? () from /lib64/ld-linux-x86-64.so.2
#19 0x3155a1a64f0ea300 in ?? ()
#20 0x0000000000000000 in ?? ()
rfuchs commented 2 years ago

What other threads are doing shouldn't really be relevant. The core should be generated immediately when the assertion is triggered and the thread that triggered it should be the first one.

If you can reproduce this, consider running rtpengine in valgrind if the performance penalty allows it. If that's not possible you can build rtpengine with libasan included (set DO_ASAN_FLAGS=1 when compiling) and then see if that reveals anything when running it.