Genesys Cloud SIP integration with TLS fails when sofia global siptrace on

[gouthon]

Genesys Cloud SIP trunk works fine when using TCP UDP. It also work when using TLS. It fails when using TLS and we have sofia global siptrace on. TCP dump shows connection reset with multiple retransmissions

fs-genesys-TLS-5061-TRACES_ON.txt

Freeswitch version affected : 1.10.5 and 1.10.7 Sofia version affected : 1.13.2 and 1.13.7-37

PORT STATE SERVICE 5061/tcp open sip-tls | ssl-enum-ciphers: | TLSv1.2: | ciphers: | TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA - strong | TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA - strong | TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 - strong | TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 - strong | TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA - strong | TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 - strong | TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 - strong | TLS_ECDHE_RSA_WITH_RC4_128_SHA - strong | TLS_ECDH_anon_WITH_3DES_EDE_CBC_SHA - broken | TLS_ECDH_anon_WITH_AES_128_CBC_SHA - broken | TLS_ECDH_anon_WITH_AES_256_CBC_SHA - broken | TLS_ECDH_anon_WITH_RC4_128_SHA - broken | TLS_RSA_WITH_3DES_EDE_CBC_SHA - strong | TLS_RSA_WITH_AES_128_CBC_SHA - strong | TLS_RSA_WITH_AES_128_CBC_SHA256 - strong | TLS_RSA_WITH_AES_128_GCM_SHA256 - strong | TLS_RSA_WITH_AES_256_CBC_SHA - strong | TLS_RSA_WITH_AES_256_CBC_SHA256 - strong | TLS_RSA_WITH_AES_256_GCM_SHA384 - strong | TLS_RSA_WITH_CAMELLIA_128_CBC_SHA - strong | TLS_RSA_WITH_CAMELLIA_256_CBC_SHA - strong | TLS_RSA_WITH_IDEA_CBC_SHA - weak | TLS_RSA_WITH_RC4_128_SHA - strong | TLS_RSA_WITH_SEED_CBCSHA - strong | compressors: | NULL | least strength: broken

[briankwest]

Its probably a latency issue introduced due to the slight slowdown printing of the logs, the longs do not impact anything of of TLS nature.

[gouthon]

Hi Brian, We can see multiple retransmissions on the tcpdump, how can I remediate to the latency as it only affects Genesys cloud sip trunk? Both Genesys and Freeswitch are in the US. the siptrace is necessary to see the traffic and troubleshoot. Do you recommend any tool to keep the console sip trace off, and see the sip traffic while it is encrypted?

Thanks

[pk32495]

Hi @briankwest , I am also facing the same issue. The TCP connecting is just closing abruptly during call setup process after FS sends 183 Session Progress to Gensys.

nua.c:365 nua_handle_magic() nua: nua_handle_magic: entering
2022-07-07 22:40:56.726624 [DEBUG] switch_core_sqldb.c:2836 Secure Type: srtp:sdes:AES_CM_128_HMAC_SHA1_80
2022-07-07 22:40:56.726624 [DEBUG] switch_core_sqldb.c:2836 Secure Type: srtp:sdes:AES_CM_128_HMAC_SHA1_80
tport.c:2795 tport_wakeup() tport_wakeup(0x7fd3ec212350): events IN HUP ERR
nta.c:2761 agent_tp_error() nta_agent: tport: Connection reset by peer
tport.c:2098 tport_close() tport_close(0x7fd3ec212350): tls/34.211.206.63:38733/sips
tport.c:2803 tport_wakeup() tport_wakeup(0x7fd3ec212350): tport is closed! Setting secondary timer!
nta.c:1308 agent_timer() nta: timer set next to 58005 ms
nta.c:1308 agent_timer() nta: timer set next to 58005 ms
2022-07-07 22:41:04.986603 [DEBUG] switch_ivr_play_say.c:1933 done playing file /etc/freeswitch/audio/sbc-greeting.wav
EXECUTE [depth=0] sofia/external/35800@34.211.206.63 set(hangup_after_bridge=true)
2022-07-07 22:41:04.986603 [DEBUG] mod_dptools.c:1672 SET sofia/external/35800@34.211.206.63 [hangup_after_bridge]=[true]
EXECUTE [depth=0] sofia/external/35800@34.211.206.63 set(continue_on_fail=true)
2022-07-07 22:41:04.986603 [DEBUG] mod_dptools.c:1672 SET sofia/external/35800@34.211.206.63 [continue_on_fail]=[true]
EXECUTE [depth=0] sofia/external/35800@34.211.206.63 hangup()
2022-07-07 22:41:04.986603 [NOTICE] mod_dptools.c:1380 Hangup sofia/external/35800@34.211.206.63 [CS_EXECUTE] [NORMAL_CLEARING]
2022-07-07 22:41:04.986603 [DEBUG] switch_core_session.c:2905 sofia/external/35800@34.211.206.63 skip receive message [APPLICATION_EXEC_COMPLETE] (channel is hungup already)
2022-07-07 22:41:04.986603 [DEBUG] switch_core_state_machine.c:651 (sofia/external/35800@34.211.206.63) State EXECUTE going to sleep
2022-07-07 22:41:04.986603 [DEBUG] switch_core_state_machine.c:585 (sofia/external/35800@34.211.206.63) Running State Change CS_HANGUP (Cur 1 Tot 1182)
2022-07-07 22:41:04.986603 [DEBUG] switch_core_state_machine.c:848 (sofia/external/35800@34.211.206.63) Callstate Change EARLY -> HANGUP
2022-07-07 22:41:04.986603 [DEBUG] switch_core_state_machine.c:850 (sofia/external/35800@34.211.206.63) State HANGUP
2022-07-07 22:41:04.986603 [DEBUG] mod_sofia.c:453 Channel sofia/external/35800@34.211.206.63 hanging up, cause: NORMAL_CLEARING
2022-07-07 22:41:04.986603 [DEBUG] mod_sofia.c:598 Responding to INVITE with: 480

We can see the same in Wireshark capture.

No issue when we turn off sofia global siptrace or use TCP/UDP for signaling.

Any help on this is greatly appreciated. Thank

[unairc]

This is how a packet capture looks like when SIP TRACES are off on fs_cli:

and this is another packet capture on the same host once we turn SIP TRACES on:

Wireshark is reporting that the message is malformed with a "Header not terminated by empty line (CRLF)" error:

We've tested with the latest sofia-sip version and the issue it's still there. How can this get fixed?

[briankwest]

Please attach a full sip trace, this appears as if the remote is doing something incorrectly.

[unairc]

Attached thanks packet_captures.zip

[briankwest]

Nothing about this trace seems wrong, could you highlight the issue?

[unairc]

These screenshots come from the attached genesys-20220907.cap:

Are you able to see? Do you need the private key?

[unairc]

We've opened multiple tickets with Genesys Cloud and their claim is that they never receive the "100 Trying" message from us which seems to be reflected on Wireshark also. They do receive it though when we turn SIP TRACES off.

[unairc]

Any further clarification needed @briankwest ? Thank you

[briankwest]

Happen to have the key to decrypt the pcap? I can't seem to find the packet you're referring to.

[unairc]

I have it but cannot share it - let me generate a new pcap with a temporary priv key and get back to you - thank you

[apsilva]

Hi, did you manage to solve this issue?

I'm facing the same problem.. but not on all calls, when it fails Genesis side just closes the TCP connection when we are sending the 100 trying, in nua log i have:

nta.c:2767 agent_tp_error() nta_agent: tport: Connection reset by peer

This is when it completes the call correctly:

And when it fails:

The big difference i see is that the 100 Trying split into multiple frames..