herlesupreeth / docker_open5gs

Docker files to run open5gs + IMS + eNB + gNB + NR-UE in a docker
BSD 2-Clause "Simplified" License
311 stars 170 forks source link

sgwc is crashing if the call answered from Freeswitch #294

Closed NUCLEAR-WAR closed 5 months ago

NUCLEAR-WAR commented 6 months ago

Hi,

I've setup VoLTE with FS as an MTAS, the call between the UEs ( two IPhones) works fine, but if I tried to call FS teh SGWC crashes :

03/28 17:28:41.229: [gtp] DEBUG: [7] LOCAL Find GTPv2 peer [172.22.0.9]:2123 (../lib/gtp/xact.c:949) 03/28 17:28:41.229: [gtp] DEBUG: [7] LOCAL Receive peer [172.22.0.9]:2123 (../lib/gtp/xact.c:966) 03/28 17:28:41.229: [gtp] DEBUG: [7] LOCAL UPD RX-96 peer [172.22.0.9]:2123 (../lib/gtp/xact.c:448) 03/28 17:28:41.229: [sgwc] DEBUG: Create Bearer Response (../src/sgwc/s11-handler.c:707) 03/28 17:28:41.229: [gtp] DEBUG: [7] LOCAL Commit peer [172.22.0.9]:2123 (../lib/gtp/xact.c:629) 03/28 17:28:41.230: [gtp] DEBUG: [7] LOCAL Delete peer [172.22.0.9]:2123 (../lib/gtp/xact.c:1149) 03/28 17:28:41.230: [sgwc] FATAL: sgwc_s11_handle_create_bearer_response: Assertion `ul_tunnel' failed. (../src/sgwc/s11-handler.c:802) 03/28 17:28:41.231: [core] FATAL: backtrace() returned 8 addresses (../lib/core/ogs-abort.c:37) ./open5gs-sgwcd(+0x189b7) [0x5b3c92cf09b7] ./open5gs-sgwcd(+0x13c6d) [0x5b3c92cebc6d] /open5gs/install/lib/x86_64-linux-gnu/libogscore.so.2(ogs_fsm_dispatch+0x113) [0x70600ed63402] ./open5gs-sgwcd(+0x629d) [0x5b3c92cde29d] /open5gs/install/lib/x86_64-linux-gnu/libogscore.so.2(+0x11754) [0x70600ed54754] /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x70600ecfc609] /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x70600ec21353]

Any help will be match appropriated.

I attached the PCAP and the logs i can reproduce the issue always.

Best Regards hss.log mme.log pcrf.log sgwc.log sgwu.log smf.log upf.log swgc_crash_full.zip enb.log

herlesupreeth commented 6 months ago

@NUCLEAR-WAR

but if I tried to call FS

you mean call from UE to FS?

herlesupreeth commented 6 months ago

I had a look into pcap, it looks like an issue in open5gs. Can you try using the latest open5gs commits by modifying the commits hash here https://github.com/herlesupreeth/docker_open5gs/blob/f338aa9efdbf0ce017264e6340cbe5c78125728e/base/Dockerfile#L71 and recompile the base image?

btw, awesome work making use of FS as TAS :)

NUCLEAR-WAR commented 5 months ago

Hi herlesupreeth,

still crashing, here is the logs again.

mme.log pcrf.log sgwc.log sgwu.log smf.log upf.log TRACE.zip

I use now FS as TAS and Asterisk as Voicemail and IBCF utilizing ENUM to route the calls to IBCF, it was quit a journey, still have issues with RTPs, RTPs flowing from UE directly from IP Addrese of the UE to FS was expecting from UE to RTPEngine then to FS

herlesupreeth commented 5 months ago

I think its an issue in open5gs and would suggest to raise an issue in open5gs repo. Below is my breakdown of the scenario based on the pcap:

  1. packet 3425 Create Bearer Request is sent for VoLTE voice call
  2. packet 3433 Activate dedicated EPS bearer context request is sent to UE
  3. packet 3468 eNB replies E-RABSetupResponse saying the bearer was setup successfully (still UE hasnt replies it bearer creation was successful or not)
  4. packet 3588 UE is sending BYE SIP message saying "SIP;cause=200;text="Received reject SDP". This triggers deletion of bearer
  5. NOTE: At this point Create Bearer Response is not yet received (received in packet 3674)
  6. packet 3634 Delete Bearer Request is sent

I think not having received Create Bearer Response before receiving Delete Bearer Request is causing SGW-C to crash.

Regarding the RTP issue you can take a look at this comment - https://github.com/herlesupreeth/docker_open5gs/issues/142#issuecomment-1420801379

NUCLEAR-WAR commented 5 months ago

I will open an issue in Open5GS.

I dont know why the Iphone is rejecting the SDP with BYE, is it because there is not Bearer Response?

In the meantime the change in that comment solve the issue to all calls towards AS and term_unreg to Voicemail !! many thanks, the change could be more improved to be not only based on the User-Agent but for the time being it solve a big issue.

I see there is almost not guide on how to integrate FS and Asterisk as MTAS, I may write one when every things is stable is not so hard actually.

herlesupreeth commented 5 months ago

I dont know why the Iphone is rejecting the SDP with BYE, is it because there is not Bearer Response?

It could be that it not receiving sendrecv as part of SDP for INVITE

you could use this commits changes - https://github.com/herlesupreeth/docker_open5gs/commit/b14eecb9912615de6d5ba2cd2e2b6f06289ee10c as reference to alter the SDP parameters

In the meantime the change in that comment solve the issue to all calls towards AS and term_unreg to Voicemail !! many thanks, the change could be more improved to be not only based on the User-Agent but for the time being it solve a big issue.

Thanks for confirming it :). I have very less knowledge of working with AS so havent managed to come up with a generic solution

I see there is almost not guide on how to integrate FS and Asterisk as MTAS, I may write one when every things is stable is not so hard actually.

Looking forward to it

NUCLEAR-WAR commented 5 months ago

I opened Issue#3109 regarding the SWGC, Hope get a solution for it.

I looked at the commit above its looks like its already there and doing his job maybe the IPhone is just dropping the call because there is no Bearer Response, Apple devices are know to be weird with there errors ;( I will try to connect it to a console I tried the case with OnePlus 9 Pro there is no BYE but the SGWC is still crashing

herlesupreeth commented 5 months ago

I looked at the commit above its looks like its already there and doing his job

Notice that the changes in that commit is checking whether a line in SDP with a=inactive exists and only if it exists then modifies it. I would suggest you do the modification in FS to send 200 OK for INVITE with SDP containing a=sendrecv

IPhone is just dropping the call because there is no Bearer Response

I dont think so since all iPhone cares is whether it received Activate dedicated EPS bearer context request or not

I tried the case with OnePlus 9 Pro there is no BYE but the SGWC is still crashing

Can you post the pcap for this?

NUCLEAR-WAR commented 5 months ago

@herlesupreeth I noticed that with OnePlus the SGWC is not crashing here the Request is sent to MME, and the MME is not responding to this request, there is also working cases in the trace also. mme.log pcrf.log sgwc.log sgwu.log smf.log upf.log good_and_bad_cases_f.zip

Regarding IPhone, those devices are really strange they have a SIP Stack called "Bambi" act really sometimes weird and its very sensitive and always has issues with payload types we had to implements workarounds sometimes on our P-CSCFs to cover there issues ;(. I recorded the logs from the console and the IPhone is complaining about missing Audio Settings:

Apr 5 12:00:33 iPhone CommCenter(libIPTelephony.dylib)[90] : E [sdp.session]: Missing audio settings! Apr 5 12:00:33 iPhone CommCenter(libIPTelephony.dylib)[90] : E [bambi.call]: SipStack 018C044B-73C3-4458-A4BE-45ED279B6E89 [BambiCall 1cfYz1n4NJhmGmCa5XVkAlf4]: received rejection answer: Apr 5 12:00:33 iPhone CommCenter(libIPTelephony.dylib)[90] : E [bambi.call]: SipStack 018C044B-73C3-4458-A4BE-45ED279B6E89 [BambiCall 1cfYz1n4NJhmGmCa5XVkAlf4]: v=0 Apr 5 12:00:33 iPhone CommCenter(libIPTelephony.dylib)[90] : E [bambi.call]: o=FreeSWITCH 1712280579 1712280580 IN IP4 172.22.0.16 Apr 5 12:00:33 iPhone CommCenter(libIPTelephony.dylib)[90] : E [bambi.call]: s=FreeSWITCH Apr 5 12:00:33 iPhone CommCenter(libIPTelephony.dylib)[90] : E [bambi.call]: c=IN IP4 172.22.0.16 Apr 5 12:00:33 iPhone CommCenter(libIPTelephony.dylib)[90] : E [bambi.call]: t=0 0 Apr 5 12:00:33 iPhone CommCenter(libIPTelephony.dylib)[90] : E [bambi.call]: a=rtpengine:d3deba85bf1d Apr 5 12:00:33 iPhone CommCenter(libIPTelephony.dylib)[90] : E [bambi.call]: m=audio 49458 RTP/AVP 99 105 Apr 5 12:00:33 iPhone CommCenter(libIPTelephony.dylib)[90] : E [bambi.call]: a=rtpmap:99 AMR-WB/16000 Apr 5 12:00:33 iPhone CommCenter(libIPTelephony.dylib)[90] : E [bambi.call]: a=fmtp:99 octet-align=0; mode-set=0,1,2,5,7 Apr 5 12:00:33 iPhone CommCenter(libIPTelephony.dylib)[90] : E [bambi.call]: a=rtpmap:105 telephone-event/16000 Apr 5 12:00:33 iPhone CommCenter(libIPTelephony.dylib)[90] : E [bambi.call]: a=fmtp:105 0-15 Apr 5 12:00:33 iPhone CommCenter(libIPTelephony.dylib)[90] : E [bambi.call]: a=silenceSupp:off - - - - Apr 5 12:00:33 iPhone CommCenter(libIPTelephony.dylib)[90] : E [bambi.call]: a=sendrecv Apr 5 12:00:33 iPhone CommCenter(libIPTelephony.dylib)[90] : E [bambi.call]: a=rtcp:49459 Apr 5 12:00:33 iPhone CommCenter(libIPTelephony.dylib)[90] : E [bambi.call]: a=ptime:20 Apr 5 12:00:33 iPhone CommCenter(libIPTelephony.dylib)[90] : E [bambi.call]:

not sure if he is expecting the RS/RR/AS, here is a working case between two devices over FS : VoLTE Call RTPs filteredOut.zip

NUCLEAR-WAR commented 5 months ago

also related to a=sendrecv, the attribute is there looks like the P-CSCF is adding it on the Way to the UE

NUCLEAR-WAR commented 5 months ago

Hi @herlesupreeth

with the new Fix from Sukchan, the flood of Error Indication, also almost stopped, the rest I think is on SRS eNB, it should react on this Error Indicator, after the session closed the Device still sends Traffic over the Tunnel after tear down, to my knowledge eNB should release the E-RAB, instead it throw an error in Log about Unhandled GTP-U Extension Header.

I still have an issue when the device wake up and try to make a call, the MME is not responding to "Create Bearer Request", I will open an new issue for it, you can close this issue. The Problem with IPhone is that it does not like the Mode-set and prefer to have one, I need to find a way to solve it, or to leave all MRF Tasks to Asterisk.

herlesupreeth commented 5 months ago

Thanks for the update