Downlink stops working when performing a S1 handover

nachoroyuela commented 2 years ago

Hello, we are setting up a 4G network with srseNB as eNodeB, Open5GS as EPC and USRPB210 and BladeRF2.0xA9 as SDR. Everything is working correctly, with srsUE and with our COTS phones with programmable SIMs. However, we are trying to implement S1 handover and we have encountered a problem. The UEs connect correctly to the neighboring station when they are close to it. However, the downlink stops working, while the uplink is still working. Reading the logs of the target station in the handover we found that the Open5GS core correctly sends messages through GTP with the correct TEID and srseNB receives it. However srseNB leaves the messages buffered as we can see in the logs:

2021-12-18T13:31:30.458962 [GTPU   ] [I] Rx S1-U SDU, 192.168.0.102:0x1 > DL (buffered), rnti=0x46, eps-BearerID=5, n_bytes=90, IPv4 8.8.4.4 > 10.45.0.23
2021-12-18T13:31:30.458963 [GTPU   ] [I] Rx S1-U SDU, 192.168.0.102:0x1 > DL (buffered), rnti=0x46, eps-BearerID=5, n_bytes=90, IPv4 8.8.8.8 > 10.45.0.23
2021-12-18T13:31:30.863415 [GTPU   ] [I] Rx S1-U SDU, 192.168.0.102:0x1 > DL (buffered), rnti=0x46, eps-BearerID=5, n_bytes=48, IPv4 13.107.6.158 > 10.45.0.23
2021-12-18T13:31:31.106487 [GTPU   ] [I] Tx S1-U SDU, UL > 192.168.0.102:0x1, rnti=0x46, eps-BearerID=5, n_bytes=40, IPv4 10.45.0.23 > 13.107.6.158
2021-12-18T13:31:31.106508 [GTPU   ] [I] Tx S1-U SDU, UL > 192.168.0.102:0x1, rnti=0x46, eps-BearerID=5, n_bytes=40, IPv4 10.45.0.23 > 20.61.102.89
2021-12-18T13:31:31.406359 [GTPU   ] [I] Tx S1-U SDU, UL > 192.168.0.102:0x1, rnti=0x46, eps-BearerID=5, n_bytes=41, IPv4 10.45.0.23 > 140.82.112.25
2021-12-18T13:31:31.528070 [GTPU   ] [I] Rx S1-U SDU, 192.168.0.102:0x1 > DL (buffered), rnti=0x46, eps-BearerID=5, n_bytes=52, IPv4 140.82.112.25 > 10.45.0.23
2021-12-18T13:31:32.446190 [GTPU   ] [I] Tx S1-U SDU, UL > 192.168.0.102:0x1, rnti=0x46, eps-BearerID=5, n_bytes=62, IPv4 10.45.0.23 > 8.8.8.8
2021-12-18T13:31:32.446216 [GTPU   ] [I] Tx S1-U SDU, UL > 192.168.0.102:0x1, rnti=0x46, eps-BearerID=5, n_bytes=62, IPv4 10.45.0.23 > 8.8.4.4
2021-12-18T13:31:32.459109 [GTPU   ] [I] Rx S1-U SDU, 192.168.0.102:0x1 > DL (buffered), rnti=0x46, eps-BearerID=5, n_bytes=90, IPv4 8.8.4.4 > 10.45.0.23
2021-12-18T13:31:32.459213 [GTPU   ] [I] Rx S1-U SDU, 192.168.0.102:0x1 > DL (buffered), rnti=0x46, eps-BearerID=5, n_bytes=90, IPv4 8.8.8.8 > 10.45.0.23
2021-12-18T13:31:32.464518 [GTPU   ] [I] Rx S1-U SDU, 192.168.0.102:0x1 > DL (buffered), rnti=0x46, eps-BearerID=5, n_bytes=40, IPv4 13.107.6.158 > 10.45.0.23
2021-12-18T13:31:33.466655 [GTPU   ] [I] Tx S1-U SDU, UL > 192.168.0.102:0x1, rnti=0x46, eps-BearerID=5, n_bytes=60, IPv4 10.45.0.23 > 192.168.0.1

This only occurs after the handover process takes place.
Before the procedure everything works correctly.
The same problem occurs using either the USRP B210 or the bladeRF.
We are using the lastest version of srsRAN (21.10)
Both srseNB and Open5GS run inside intel nuc computers with a 6 core Intel i9, 32 GB ram, Ubuntu 20.04.3 LTS with low latency kernel. In addition, we run the software with real time priority. So we do not believe it is a processing capacity problem.

Attached are the logs of both base stations (server (enb1) and neighbor/target (enb2)), as well as their configuration files. ZIP with configuration files and logs

frankist commented 2 years ago

Hello @nachoroyuela. The target eNB is buffering PDUs because it is waiting for a GTPU End Marker to close the S1 indirect tunnel.

If my interpretation of the TS 36.300 is correct, we should expect the EPC to send an End Marker to the Source eNB, and the Source eNB should forward that same End Marker to the Target eNB via the S1 indirect tunnel. This End Marker is useful to keep the order of DL PDUs, prioritizing the PDUs coming from the indirect tunnel over those coming from the new EPC-Target eNB tunnel.

If the Open5GS doesn't send an End Marker, which seems to be the case, I think it is an issue in the Open5GS. As a workaround you can set a timeout to delete the S1 indirect tunnel using the parameter gtpu_tunnel_timeout in the enb.conf.

Let me know how it goes.

nachoroyuela commented 2 years ago

Hello, thank you for your reply.

Indeed it is what you say. It seems that Open5GS does not send any End Marker. I tried to set a timeout to delete the S1 indirect tunnel using the parameter gtpu_tunnel_timeout in the enb.conf and it worked perfectly.

nachoroyuela commented 2 years ago

Hello, after many tests we have found another issue that we hope that you can help us to solve it:

When performing handover the connection is established correctly, however the PHR (Power Headroom) value becomes negative (-21) even when our 4G modem is next to the base station to which it is connected. This brings us problems as the MCS of the upstream channel drops drastically and we do not achieve the upload speed we need. This only happens when the 4G modem is connected to an eNB using the HO procedure. However, when connecting to such eNB directly from the beginning we have positive PHR (34 approximately),.

I attach the enb logs when the modem connects directly without HO procedure and when the modem connects with the HO procedure and the configuration files of both enbs: logs_conf.zip

srsran / srsRAN_4G

Downlink stops working when performing a S1 handover #778