search

fenar / cnvopen5gcore

23 stars 13 forks source link

No internet access from tun interface #5

Closed elenifragkou closed 2 years ago

elenifragkou commented 2 years ago

Hello @fenar,

Thank you for this great work!

I followed your setup on an OCP cluster, created service mesh and deployed open5gs & UERANSIM.

From UE logs, I can see that the TUN interface is UP, however there is no internet access.

Attaching gNB logs

UERANSIM v3.1.7
[2022-01-11 15:13:48.030] [sctp] [info] Trying to establish SCTP connection... (172.16.27.62:30412)
[2022-01-11 15:13:48.042] [sctp] [info] SCTP connection established (172.16.27.62:30412)
[2022-01-11 15:13:48.042] [sctp] [debug] SCTP association setup ascId[7]
[2022-01-11 15:13:48.042] [ngap] [debug] Sending NG Setup Request
[2022-01-11 15:13:48.043] [ngap] [debug] NG Setup Response received
[2022-01-11 15:13:48.043] [ngap] [info] NG Setup procedure is successful
[2022-01-11 15:13:48.455] [rls] [debug] New UE signal detected, total [1] UEs in coverage
[2022-01-11 15:13:48.457] [rrc] [info] RRC Setup for UE[1]
[2022-01-11 15:13:48.457] [ngap] [debug] Initial NAS message received from UE[1]
[2022-01-11 15:13:48.567] [ngap] [debug] Initial Context Setup Request received
[2022-01-11 15:13:48.868] [ngap] [info] PDU session resource(s) setup for UE[1] count[1]

and UE logs

UERANSIM v3.1.7
[2022-01-11 15:13:48.455] [nas] [info] UE switches to state [MM-DEREGISTERED/PLMN-SEARCH]
[2022-01-11 15:13:48.457] [rls] [debug] Coverage change detected. [1] cell entered, [0] cell exited
[2022-01-11 15:13:48.457] [nas] [info] Serving cell determined [UERANSIM-gnb-208-93-1]
[2022-01-11 15:13:48.457] [nas] [info] UE switches to state [MM-DEREGISTERED/NORMAL-SERVICE]
[2022-01-11 15:13:48.457] [nas] [debug] Sending Initial Registration
[2022-01-11 15:13:48.457] [nas] [info] UE switches to state [MM-REGISTER-INITIATED/NA]
[2022-01-11 15:13:48.457] [rrc] [debug] Sending RRC Setup Request
[2022-01-11 15:13:48.457] [rrc] [info] RRC connection established
[2022-01-11 15:13:48.457] [nas] [info] UE switches to state [CM-CONNECTED]
[2022-01-11 15:13:48.509] [nas] [debug] Security Mode Command received
[2022-01-11 15:13:48.510] [nas] [debug] Selected integrity[2] ciphering[0]
[2022-01-11 15:13:48.568] [nas] [debug] Registration accept received
[2022-01-11 15:13:48.568] [nas] [info] UE switches to state [MM-REGISTERED/NORMAL-SERVICE]
[2022-01-11 15:13:48.568] [nas] [info] Initial Registration is successful
[2022-01-11 15:13:48.568] [nas] [info] Initial PDU sessions are establishing [1#]
[2022-01-11 15:13:48.568] [nas] [debug] Sending PDU Session Establishment Request
[2022-01-11 15:13:48.869] [nas] [debug] PDU Session Establishment Accept received
[2022-01-11 15:13:48.869] [nas] [info] PDU Session establishment is successful PSI[1]
[2022-01-11 15:13:48.901] [app] [info] Connection setup for PDU session[1] is successful, TUN interface[uesimtun0, 10.45.0.2] is up.

uebinder logs seem ok 

PING www.google.com (142.250.184.132) 56(84) bytes of data.
64 bytes from 142.250.184.132: icmp_seq=1 ttl=113 time=16.2 ms
64 bytes from 142.250.184.132: icmp_seq=2 ttl=113 time=15.7 ms
64 bytes from 142.250.184.132: icmp_seq=3 ttl=113 time=15.7 ms
64 bytes from 142.250.184.132: icmp_seq=4 ttl=113 time=15.7 ms
64 bytes from 142.250.184.132: icmp_seq=5 ttl=113 time=15.7 ms
64 bytes from 142.250.184.132: icmp_seq=6 ttl=113 time=15.6 ms

I opened a shell from uebinder and tried ping & tcpdump

ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
3: eth0@if85: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    link/ether 0a:58:0a:80:02:4c brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.128.2.76/23 brd 10.128.3.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::43f:1dff:feb6:3288/64 scope link 
       valid_lft forever preferred_lft forever
4: uesimtun0: <POINTOPOINT,PROMISC,NOTRAILERS,UP,LOWER_UP> mtu 1400 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none 
    inet 10.45.0.2/32 scope global uesimtun0
       valid_lft forever preferred_lft forever
    inet6 fe80::c059:b90b:7303:4d05/64 scope link stable-privacy 
       valid_lft forever preferred_lft forever
# ping -I uesimtun0 www.google.com
PING www.google.com (142.250.184.132) from 10.45.0.2 uesimtun0: 56(84) bytes of data.
^C
--- www.google.com ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3104ms

# tcpdump -i uesimtun0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on uesimtun0, link-type RAW (Raw IP), capture size 262144 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel
#

If a capture for eth0 I can see the ICMP packets.

Any suggestions what else should I look ?

Thanks a lot, Eleni

github-actions[bot] commented 2 years ago

Thank you for submitting a issue to make this repo better!

fenar commented 2 years ago

Thanks for reporting the issue, can you please take trace on UPF node to see if your packets coming from ue properly and make sure there is no routing issue on upf networking? Meanwhile I will try to replicate the issue.

elenifragkou commented 2 years ago

Hi fenar, thank you for your prompt reply!

On UPF side, I can see that ogstun interface is on UP state, although there is no packet transmission.

# ip -br a
lo               UNKNOWN        127.0.0.1/8 
eth0@if553       UP             10.129.2.75/23 
ogstun           UP             10.45.0.1/16 

# tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ogstun, link-type RAW (Raw IP), capture size 262144 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel

# ping -I ogstun 8.8.8.8
PING 8.8.8.8 (8.8.8.8) from 10.45.0.1 ogstun: 56(84) bytes of data.
^C
--- 8.8.8.8 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3083ms

If you manage to replicate this, please let me know how it goes.

fenar commented 2 years ago

Hi; I fixed two major flaws (both under Release1); upf network interface for gtptunnel selection as well as ueransim (i have rebuild the image based on lates release drop from project owner) ue nr-binder src ip use. Please let me know if these fixes your issue.

fenar commented 2 years ago

Fixes tested and nr-binder working as expected with ping test: PING google.com (142.251.116.102) 56(84) bytes of data. 64 bytes from rt-in-f102.1e100.net (142.251.116.102): icmp_seq=1 ttl=105 time=6.93 ms 64 bytes from rt-in-f102.1e100.net (142.251.116.102): icmp_seq=2 ttl=105 time=7.95 ms 64 bytes from rt-in-f102.1e100.net (142.251.116.102): icmp_seq=3 ttl=105 time=6.29 ms 64 bytes from rt-in-f102.1e100.net (142.251.116.102): icmp_seq=4 ttl=105 time=6.00 ms