Separate machines across different networks?

[galalmounir]

What is the best way to run the core dockers on one machine A on network A then run the ENB on a separate machine B on network B and have them communicate?

What I tried was putting Machine A's IP in DOCKER_HOST_IP, SGWU_ADVERTISE_IP and UPF_ADVERTISE_IP. I also uncommented the lines in the docker files. Finally, I port forwarded the two ports that are open in the docker files.

The first issue I faced was being unable to run the srsenb.yaml docker-compose on Machine B, because the virtual docker network (external) is not available on that machine. I tried to manually create the virtual network. But was unable to get machine B (ENB) to connect to Machine A's core.

Any ideas?

[herlesupreeth]

Hey, if you are running eNB on an another machine then you would need to change the networking to host mode (rather than external) of srsenb.yaml docker-compose file so that there is no NATing done by the docker network.

i.e. remove the networks sections and add under srsenb section in that docker-compose file the following

network_mode: host

[galalmounir]

Thanks!

I tried this and I got the following error:

srsenb    | bind(): Cannot assign requested address
srsenb    | Failed to bind on address 172.22.0.22, port 2152: Cannot assign requested address
srsenb    | Error initializing EUTRA stack.

This is what my srsenb.yaml looks like:

version: '3'
services:
  srsenb:
    image: galalhassan/docker_srslte
    container_name: srsenb
    stdin_open: true
    tty: true
    privileged: true
    devices:
      - "/dev/bus"
    volumes:
      - /dev/serial:/dev/serial:ro
      - /dev/bus/usb:/dev/bus/usb:ro
      - ./srslte:/mnt/srslte
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    environment:
      - COMPONENT_NAME=enb
    expose:
      - "36412/sctp"
      - "2152/udp"
    network_mode: "host"

[herlesupreeth]

Ah my bad, you need to set the SRS_ENB_IP to the host IP of the machine running eNB (rather than 172.22.022)

[galalmounir]

Interesting.. I am assuming this would be the public IP not the local IP of the machine. What port would we need to expose on the ENB machine/network?

(I'll create a PR to update the docs with this info once I get things up and running)

[herlesupreeth]

I am assuming this would be the public IP not the local IP of the machine.

The IP at which eNB is able to reach EPC

What port would we need to expose on the ENB machine/network?

Technically, its 36412 port for SCTP and 2152 for UDP. But since we are running in network_mode: host , those ports should be exposed across all interface of that machine running eNB

[galalmounir]

The IP at which eNB is able to reach EPC

So the IP of the machine running the EPC (Machine A) then not the eNB (Machine B)?

Technically, its 36412 port for SCTP and 2152 for UDP. But since we are running in network_mode: host , those ports should be exposed across all interface of that machine running eNB

These ports are for the mme and sgwu, which are running on Machine A.

To summarize my setup:

• Machine A
  • running the core containers (EPC)
  • Exposes ports 36412 and 2152
• Machine B
  • running the eNB
  • uses network_mode: host

[herlesupreeth]

SRS_ENB_IP should be the IP of machine B running eNB.

These ports are for the mme and sgwu, which are running on Machine A.

Those ports are also the endpoints at eNB through which eNB connects with EPC

[galalmounir]

Got it, let me give it a try and report back.

What's the best way to test the connection between the eNB and EPC? (other than running all components and checking logs)

[galalmounir]

I am still unable to get both machines working. I am now trying the same setup but on the same local network.

• Machine A
  • running the core containers (EPC)
  • Exposes ports 36412 and 2152
• Machine B
  • running the eNB
  • uses network_mode: host (without network_mode set as host, I am unable to start the enb container)
  • exposes ports 36412 and 2152

The .env file has the following: DOCKER_HOST_IP set to Machine A IP SGWU_ADVERTISE_IP set to Machine A IP UPF_ADVERTISE_IP set to Machine A IP SRS_ENB_IP set to Machine B IP

I am checking if they are able to communicate by checking the logs on the mme. I don't see the enb connecting to the mme

[herlesupreeth]

Can you please send me a trace take on both the machines (without any filters) when you try to connect eNB to EPC ??

[s5uishida]

Hi @galalmounir

I don't think UPF_ADVERTISE_IP should be changed.

[herlesupreeth]

@s5uishida ah true, in case of 4G UPF_ADVERTISE_IP should not be modified. But it shouldn't affect eNB from connecting to EPC

[galalmounir]

@herlesupreeth when you say trace, what's the best tool to use? (Wireshark?)

[galalmounir]

Without any filters using Wireshark is too much. Both machines are connected to the internet, plus SSH session and VNC sessions. I'll filter out the traffic with source/destination IPs. That way we filter out anything going out to the internet or over to my main/dev machine.

[galalmounir]

The IPs for the machines are as follows:

• Machine A (EPC) -> 192.168.7.168
• Machine B (eNB) -> 192.168.7.222
• Dev Machine -> 192.168.6.3

I used the following filter on Machine A:

(ip.src==192.168.7.168 || ip.addr==192.168.7.222) && not ssh && ip.addr!=192.168.6.3

And I used the following filter on Machine B:

(ip.src==192.168.7.222 || ip.addr==192.168.7.168) && not ssh && ip.addr!=192.168.6.3

All the logs are either MDNS or TCP to my dev machine (192.168.6.3).

This is from Machine A:

This is from Machine B:

[galalmounir]

I did some digging, and it seems like the sctp port is not exposed on Machine 1. Even though it is correctly mapped by docker. (tried using netcat/socat) I believe this has to do with me running the EPC stack on a MacOS.

I'll try running the EPC stack on a linux machine in the morning and try again.

[herlesupreeth]

I believe this has to do with me running the EPC stack on a MacOS.

That is it. MAC OS as far as I remember doesnt have libsctp

[s5uishida]

Hi @galalmounir

Have you tried the following?

brew install libusrsctp

https://open5gs.org/open5gs/docs/platform/05-macosx-apple-silicon/

[herlesupreeth]

@s5uishida But that would require compiling open5gs with usrsctp support which is not done in Dockerfile of this repo. Let me know if I am missing anything.

[galalmounir]

@s5uishida yes I tried installing libusrsctp with no avail. :( Might be due to what @herlesupreeth mentioned about building open5gs with usrsctp support.

@herlesupreeth, do you have a link to rebuilding with usrsctp support?

[herlesupreeth]

@herlesupreeth, do you have a link to rebuilding with usrsctp support?

hmmm, I wouldn't recommend it because usrsctp will be used only by MME for connection to eNB but whereas for diameter side of things freediameter will be used, which still uses libsctp (no other SCTP alternative for freediameter) so you wont have a working setup.

I would suggest running an Ubuntu VM on top of MAC OS and install docker_open5gs in that.

[galalmounir]

I’ll give that a try shortly and report back. Thanks!

[galalmounir]

Okay, so after a few trials and errors. I was able to get it working.

TLDR; the MME_IP in the .env file also needs to change to the DOCKER_HOST_IP

Details: I setup an AWS ubuntu VM and opened the firewall to the internet. This way I can rule out any ports that are not open. Then I started by changing things one by one in the .env and srsenb.yaml until I was able to start seeing some traffic on wireshark. This is the filter used: ip.addr=={AWS_IP} || sctp.port==36412 || udp.port==2152

I saw a few INIT requests going to IP 172.22.0.9, so after looking into the .env file. I found that the MME_IP is set to 172.22.0.9, after changing it to the {AWS_IP} (on the machine running the eNB), I started seeing more traffic on wireshark with successful S1 connections and heartbeats.

I'll push a PR to add/update the docs to reflect those requirements.

Thanks again for all the help!

[galalmounir]

Resolved with suggestions and readme updated in PR #236 to reflect the suggestions