[bug]: Call between tls/dtls configured client and udp configured client fails with ‘called a function you should not call’

AngryGami commented 4 months ago

Severity

Minor

Versions

Asterisk 20.5.2

Components/Modules

res_rtp_asterisk

Operating Environment

Alpine Linux 3.19, docker container

Frequency of Occurrence

Constant

Issue Description

I have two clients connected in following way: "Intercom" client uses udp for both sip and rtp

# docker exec hosp-asterisk rasterisk -x 'pjsip show endpoint intercom1'

Endpoint:  intercom1                                            Not in use    0 of inf
    OutAuth:  intercom1/intercom1
     InAuth:  intercom1/intercom1
        Aor:  intercom1                                          1
      Contact:  intercom1/sip:intercom1@192.168.3.165:5060 2d9e929e55 NonQual         nan
  Transport:  transport-udp             udp      0      0  0.0.0.0:5060

 ParameterName                      : ParameterValue
 ===================================================================================================
 100rel                             : yes
 accept_multiple_sdp_answers        : false
 accountcode                        :
 acl                                :
 aggregate_mwi                      : true
 allow                              : (speex|g729|gsm|g723|ulaw|alaw|h264|h263)
 allow_overlap                      : true
 allow_subscribe                    : true
 allow_transfer                     : true
 allow_unauthenticated_options      : false
 aors                               : intercom1
 asymmetric_rtp_codec               : false
 auth                               : intercom1
 bind_rtp_to_media_address          : false
 bundle                             : false
 call_group                         :
 callerid                           : <unknown>
 callerid_privacy                   : allowed_not_screened
 callerid_tag                       :
 codec_prefs_incoming_answer        : prefer:pending, operation:intersect, keep:all, transcode:allow
 codec_prefs_incoming_offer         : prefer:pending, operation:intersect, keep:all, transcode:allow
 codec_prefs_outgoing_answer        : prefer:pending, operation:intersect, keep:all, transcode:allow
 codec_prefs_outgoing_offer         : prefer:pending, operation:union, keep:all, transcode:allow
 connected_line_method              : invite
 contact_acl                        :
 context                            : intercoms-callout
 cos_audio                          : 0
 cos_video                          : 0
 device_state_busy_at               : 0
 direct_media                       : false
 direct_media_glare_mitigation      : none
 direct_media_method                : invite
 disable_direct_media_on_nat        : false
 dtls_auto_generate_cert            : No
 dtls_ca_file                       :
 dtls_ca_path                       :
 dtls_cert_file                     :
 dtls_cipher                        :
 dtls_fingerprint                   : SHA-256
 dtls_private_key                   :
 dtls_rekey                         : 0
 dtls_setup                         : active
 dtls_verify                        : No
 dtmf_mode                          : rfc4733
 fax_detect                         : false
 fax_detect_timeout                 : 0
 follow_early_media_fork            : true
 force_avp                          : false
 force_rport                        : true
 from_domain                        : 192.168.3.253
 from_user                          :
 g726_non_standard                  : false
 geoloc_incoming_call_profile       :
 geoloc_outgoing_call_profile       :
 ice_support                        : false
 identify_by                        : username,ip
 ignore_183_without_sdp             : false
 inband_progress                    : false
 incoming_call_offer_pref           : local
 incoming_mwi_mailbox               :
 language                           :
 mailboxes                          :
 max_audio_streams                  : 1
 max_video_streams                  : 1
 media_address                      :
media_encryption                   : no
media_encryption_optimistic        : false
 media_use_received_transport       : false
 message_context                    :
 moh_passthrough                    : false
 moh_suggest                        : default
 mwi_from_user                      :
 mwi_subscribe_replaces_unsolicited : no
 named_call_group                   :
 named_pickup_group                 :
 notify_early_inuse_ringing         : false
 one_touch_recording                : false
 outbound_auth                      : intercom1
 outbound_proxy                     :
 outgoing_call_offer_pref           : remote_merge
 overlap_context                    :
 pickup_group                       :
 preferred_codec_only               : false
 record_off_feature                 : automixmon
 record_on_feature                  : automixmon
 refer_blind_progress               : true
 rewrite_contact                    : true
 rpid_immediate                     : false
 rtcp_mux                           : false
 rtp_engine                         : asterisk
 rtp_ipv6                           : false
 rtp_keepalive                      : 0
 rtp_symmetric                      : false
 rtp_timeout                        : 0
 rtp_timeout_hold                   : 0
 sdp_owner                          : -
 sdp_session                        : Asterisk
 security_mechanisms                :
 security_negotiation               : no
 send_aoc                           : false
 send_connected_line                : yes
 send_diversion                     : true
 send_history_info                  : false
 send_pai                           : false
 send_rpid                          : false
 set_var                            :
 srtp_tag_32                        : false
 stir_shaken                        : off
 stir_shaken_profile                :
 sub_min_expiry                     : 0
 subscribe_context                  :
 suppress_q850_reason_headers       : false
 t38_bind_udptl_to_media_address    : false
 t38_udptl                          : false
 t38_udptl_ec                       : none
 t38_udptl_ipv6                     : false
 t38_udptl_maxdatagram              : 0
 t38_udptl_nat                      : false
 timers                             : yes
 timers_min_se                      : 90
 timers_sess_expires                : 1800
 tone_zone                          :
 tos_audio                          : 0
 tos_video                          : 0
 transport                          : transport-udp
 trust_connected_line               : yes
 trust_id_inbound                   : false
 trust_id_outbound                  : false
 use_avpf                           : false
 use_ptime                          : false
 user_eq_phone                      : false
 voicemail_extension                :
 webrtc                             : no

and "User" client that uses tls/dtsl:

docker exec hosp-asterisk rasterisk -x 'pjsip show endpoint vlocal10011'

 Endpoint:  vlocal10011                                          Not in use    0 of inf
     InAuth:  vlocal10011/vlocal10011
        Aor:  vlocal10011                                        1
      Contact:  vlocal10011/sip:vlocal10011@192.168.3.137: 7d8cbb28e0 NonQual         nan
  Transport:  transport-tls             tls      0      0  0.0.0.0:5061

 ParameterName                      : ParameterValue
 ===================================================================================================
 100rel                             : yes
 accept_multiple_sdp_answers        : false
 accountcode                        :
 acl                                :
 aggregate_mwi                      : true
 allow                              : (speex|g729|alaw|ulaw|h263|h264)
 allow_overlap                      : true
 allow_subscribe                    : true
 allow_transfer                     : true
 allow_unauthenticated_options      : false
 aors                               : vlocal10011
 asymmetric_rtp_codec               : false
 auth                               : vlocal10011
 bind_rtp_to_media_address          : false
 bundle                             : false
 call_group                         :
 callerid                           : <unknown>
 callerid_privacy                   : allowed_not_screened
 callerid_tag                       :
 codec_prefs_incoming_answer        : prefer:pending, operation:intersect, keep:all, transcode:allow
 codec_prefs_incoming_offer         : prefer:pending, operation:intersect, keep:all, transcode:allow
 codec_prefs_outgoing_answer        : prefer:pending, operation:intersect, keep:all, transcode:allow
 codec_prefs_outgoing_offer         : prefer:pending, operation:union, keep:all, transcode:allow
 connected_line_method              : invite
 contact_acl                        :
 context                            : outbound-op
 cos_audio                          : 0
 cos_video                          : 0
 device_state_busy_at               : 0
 direct_media                       : false
 direct_media_glare_mitigation      : none
 direct_media_method                : invite
 disable_direct_media_on_nat        : false
 dtls_auto_generate_cert            : No
 dtls_ca_file                       :
 dtls_ca_path                       : /etc/ssl/certs/
 dtls_cert_file                     : /run/secrets/asterisktlscert
 dtls_cipher                        :
 dtls_fingerprint                   : SHA-256
 dtls_private_key                   : /run/secrets/asterisktlskey
 dtls_rekey                         : 0
 dtls_setup                         : actpass
 dtls_verify                        : Yes
 dtmf_mode                          : auto
 fax_detect                         : false
 fax_detect_timeout                 : 0
 follow_early_media_fork            : true
 force_avp                          : false
 force_rport                        : true
 from_domain                        : nightsterisk.devc.acme.com
 from_user                          :
 g726_non_standard                  : false
 geoloc_incoming_call_profile       :
 geoloc_outgoing_call_profile       :
 ice_support                        : false
 identify_by                        : username,ip
 ignore_183_without_sdp             : false
 inband_progress                    : false
 incoming_call_offer_pref           : local
 incoming_mwi_mailbox               :
 language                           :
 mailboxes                          :
 max_audio_streams                  : 1
 max_video_streams                  : 1
 media_address                      : 192.168.3.253
 media_encryption                   : dtls
 media_encryption_optimistic        : false
 media_use_received_transport       : true
 message_context                    :
 moh_passthrough                    : false
 moh_suggest                        : default
 mwi_from_user                      :
 mwi_subscribe_replaces_unsolicited : no
 named_call_group                   :
 named_pickup_group                 :
 notify_early_inuse_ringing         : false
 one_touch_recording                : false
 outbound_auth                      :
 outbound_proxy                     :
 outgoing_call_offer_pref           : remote_merge
 overlap_context                    :
 pickup_group                       :
 preferred_codec_only               : false
 record_off_feature                 : automixmon
 record_on_feature                  : automixmon
 refer_blind_progress               : true
 rewrite_contact                    : true
 rpid_immediate                     : false
 rtcp_mux                           : false
 rtp_engine                         : asterisk
 rtp_ipv6                           : false
 rtp_keepalive                      : 0
 rtp_symmetric                      : false
 rtp_timeout                        : 0
 rtp_timeout_hold                   : 0
 sdp_owner                          : -
 sdp_session                        : Asterisk
 security_mechanisms                :
 security_negotiation               : no
 send_aoc                           : false
 send_connected_line                : yes
 send_diversion                     : true
 send_history_info                  : false
 send_pai                           : false
 send_rpid                          : false
 set_var                            :
 srtp_tag_32                        : false
 stir_shaken                        : off
 stir_shaken_profile                :
 sub_min_expiry                     : 0
 subscribe_context                  :
 suppress_q850_reason_headers       : false
 t38_bind_udptl_to_media_address    : false
 t38_udptl                          : false
 t38_udptl_ec                       : none
 t38_udptl_ipv6                     : false
 t38_udptl_maxdatagram              : 0
 t38_udptl_nat                      : false
 timers                             : yes
 timers_min_se                      : 90
 timers_sess_expires                : 1800
 tone_zone                          :
 tos_audio                          : 0
 tos_video                          : 0
 transport                          : transport-tls
 trust_connected_line               : yes
 trust_id_inbound                   : false
 trust_id_outbound                  : false
 use_avpf                           : false
 use_ptime                          : false
 user_eq_phone                      : false
 voicemail_extension                :
 webrtc                             : no

When I try to make call from "User" to "Intercom" sip seems to work just fine and bridge is created on asterisk, but then suddenly call drops with following messages in the logs:

[2024-01-31 10:27:16.340] VERBOSE[768][C-00000003] app_dial.c: Called PJSIP/intercom1
[2024-01-31 10:27:16.465] VERBOSE[768][C-00000003] app_dial.c: PJSIP/intercom1-00000005 is ringing
[2024-01-31 10:27:16.469] VERBOSE[768][C-00000003] app_dial.c: PJSIP/intercom1-00000005 answered PJSIP/vlocal10011-00000004
[2024-01-31 10:27:16.473] VERBOSE[769][C-00000003] bridge_channel.c: Channel PJSIP/intercom1-00000005 joined 'simple_bridge' basic-bridge <ee84ed42-5a13-4b45-b849-aa8cc3c67375>
[2024-01-31 10:27:16.474] VERBOSE[768][C-00000003] bridge_channel.c: Channel PJSIP/vlocal10011-00000004 joined 'simple_bridge' basic-bridge <ee84ed42-5a13-4b45-b849-aa8cc3c67375>
[2024-01-31 10:27:17.009] ERROR[768][C-00000003] res_rtp_asterisk.c: DTLS failure occurred on RTP instance '0x7f0fe3253ee0' due to reason 'called a function you should not call', terminating
[2024-01-31 10:27:17.010] WARNING[768][C-00000003] res_rtp_asterisk.c: RTP Read error: Invalid argument.  Hanging up.
[2024-01-31 10:27:17.011] VERBOSE[768][C-00000003] bridge_channel.c: Channel PJSIP/vlocal10011-00000004 left 'simple_bridge' basic-bridge <ee84ed42-5a13-4b45-b849-aa8cc3c67375>
[2024-01-31 10:27:17.012] VERBOSE[769][C-00000003] bridge_channel.c: Channel PJSIP/intercom1-00000005 left 'simple_bridge' basic-bridge <ee84ed42-5a13-4b45-b849-aa8cc3c67375>

Error is

DTLS failure occurred on RTP instance '0x7f0fe3253ee0' due to reason 'called a function you should not call', terminating

This error message most likely comes from openssl library here: https://github.com/openssl/openssl/blob/9170cc0398222778065e098e396b8eb8cd0de1d3/ssl/ssl_lib.c#L2291

which is probably called from asterisk code here: https://github.com/asterisk/asterisk/blob/master/res/res_rtp_asterisk.c#L3284

I've tried this setup on lower and lower Asterisk versions and it eventually worked on Asterisk 16.6.2 (Alpine 3.11 in container).

Relevant log output

No response

Asterisk Issue Guidelines

[X] Yes, I have read the Asterisk Issue Guidelines

jcolp commented 4 months ago

Alpine is not a core supported environment by the project. It is community supported, and may ultimately be the cause of this. There have been no other reports of such an issue with non-Alpine environments.

AngryGami commented 4 months ago

Well it does work in alpine 3.11 (asterisk 16.6.2). As far as I understand this is somehow related to libssl that asterisks uses which comes with asterisk as dependency but originates in openssl project. Maybe something is different in how asterisk is built for alpine environment.. Whom could I ask and/or where?

AngryGami commented 4 months ago

Asterisk package for Alpine is built using openssl-dev (3.1.4-r2) (from here), is this different from how it is built for e.g. Ubuntu?

jcolp commented 4 months ago

We don't distribute packages, people generally build it themselves on the various distros or use the packages provided by those distros. A fundamental difference is that Alpine uses a different libc library to the rest, which has caused issues in the past. I don't know of differences regarding OpenSSL across things. I can just safely say that across Debian, Ubuntu, and Redhat based distros things currently work fine in this area.

You could investigate yourself such as trying to figure out the differences between Asterisk versions and identify what caused the behavior change to elicit this.

jcolp commented 4 months ago

I do recall there are one or two people with Asterisk + Alpine experience, so perhaps they'll comment on here with insight too.

AngryGami commented 4 months ago

Thanks I understand that alpine is using musl for libc implementation but it is still strange that only this interaction is affected somehow - usually this difference manifests itself with lots of crashes and incompatibilities. Do you know which version of openssl is used when asterisk is build for e.g. Ubuntu? Or maybe know where to look for that?

jcolp commented 4 months ago

I have no involvement in Ubuntu packaging. It would also depend on the version of Ubuntu in question. I don't know off the top of my head where to look.

AngryGami commented 4 months ago

I've reproduced exactly same issue on ubuntu .

$ docker exec hosp-asterisk cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Asterisk 18.10.0~dfsg+~cs6.10.40431411-2 built by nobody @ buildd.debian.org on a unknown running Linux on 2022-02-12 18:24:51 UTC

jcolp commented 4 months ago

The current supported version of Asterisk 18 is 18.21.0. It would need to be reproduced under that version built by yourself without any patches. If the problem reoccurs then a complete log, as well as a packet capture would be needed. We would also need to know what client is using DTLS, since using it in a non-WebRTC environment is uncommon.

AngryGami commented 4 months ago

Client is liblinphone version 5.2.77, though calls between two DTLS clients just work - only when I try to make a call from DTLS to plain UDP client this error appears. I don't know how to build asterisk myself and this version is one I've got from official ubuntu repository for jammy - why it is not good enough... strange.

jcolp commented 4 months ago

Distro packages lag behind, and they also include patches not part of Asterisk.

AngryGami commented 4 months ago

Ok, I guess tcpdump taken now should be fine (i.e. I don't need to build asterisk from scratch). So here it is: dump.zip Logs from asterisk for this moment:

[2024-02-02 16:54:55.920] VERBOSE[901][C-00000007] bridge_channel.c: Channel PJSIP/vlocal10011-0000000c joined 'simple_bridge' basic-bridge <923214af-925c-4aaa-a00e-511ab87d5eda>
[2024-02-02 16:54:56.460] ERROR[901][C-00000007] res_rtp_asterisk.c: DTLS failure occurred on RTP instance '0x7fec74179800' due to reason 'called a function you should not call', terminating
[2024-02-02 16:54:56.460] WARNING[901][C-00000007] res_rtp_asterisk.c: RTP Read error: Unspecified.  Hanging up.
[2024-02-02 16:54:56.460] VERBOSE[901][C-00000007] bridge_channel.c: Channel PJSIP/vlocal10011-0000000c left 'simple_bridge' basic-bridge <923214af-925c-4aaa-a00e-511ab87d5eda>
[2024-02-02 16:54:56.461] VERBOSE[907][C-00000007] bridge_channel.c: Channel PJSIP/intercom1-0000000d left 'simple_bridge' basic-bridge <923214af-925c-4aaa-a00e-511ab87d5eda>

And packet that most likely caused issue is this one:

I'll try to build "proper" asterisk version later next week... though I doubt this will do anything useful.

jcolp commented 3 months ago

Were you able to reproduce this using a non-distro build?

AngryGami commented 3 months ago

I didn't have time to setup everything I need to build asterisk from sources. So - no, I didn't.

asterisk / asterisk