How to enable ip_forwarding in calico microk8s for UPF pod

PacooMC commented 1 year ago

Hi everyone,

I am trying to deploy the 5G Core running on a two nodes cluster created by microk8s. Everyone seems to be working right, but, after too many hours trying to enable ip_forwarding inside UPF, I decided to come to the community.

The only way if have found to enable ip_forwarding is by using the following commands:

systemd-cgls | grep upf # To extract UPF's pid sudo nsenter -t -n sysctl -w net.ipv4.ip_forward=1

This makes the network work and the UE is able to reach Internet.

I have tried to configure it in many different ways as presented in the links shown at the documentation: "We remind you that some CNI plugins (e.g. Flannel) allow this functionality by default, while others (.e.g. Calico) require a special configuration."

I am not still sure of how calico works regarding to this.

After setting up the cluster (and installing plugins&modules), I run these commands in order to install calicoctl for kubectl, as shown in https://projectcalico.docs.tigera.io/maintenance/clis/calicoctl/install#install-calicoctl-as-a-kubectl-plugin-on-a-single-host

curl -L https://github.com/projectcalico/calico/releases/download/v3.24.1/calicoctl-linux-amd64 -o kubectl-calico chmod +x kubectl-calico sudo mv kubectl-calico /usr/bin

Assuming the tool is now already configured, I tried to run the following:

microk8s kubectl calico --allow-version-mismatch apply -f - <<EOF

apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: empty-default-allow spec: order: 0 selector: "all()" applyOnForward: true types:
- Ingress
- Egress ingress:
  - action: Allow egress:
  - action: Allow EOF

After this, I redeployed the free5gc charts, but running the following command in the UPF still gives 0 as a result: cat /proc/sys/net/ipv4/ip_forward

Thank you in advance for any help that could be provided.

pinoOgni commented 1 year ago

Hi @PacooMC I have never touched applyOnForward value that is a Policy, so I think you need only to set containerIPForwarding: "Enabled" in custom-resource file when you install Calico.

apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
  name: default
spec:
  # Configures Calico networking.
  calicoNetwork:
    # Note: The ipPools section cannot be modified post-install.
    ipPools:
    - blockSize: 26
      cidr: 172.22.0.0/16
      encapsulation: IPIP
      natOutgoing: Enabled
      nodeSelector: all()
    containerIPForwarding: "Enabled"
...

I did so, for example for Calico version v3.23:

kubectl create -f https://projectcalico.docs.tigera.io/archive/v3.23/manifests/tigera-operator.yaml
curl https://projectcalico.docs.tigera.io/archive/v3.23/manifests/custom-resources.yaml -O
sed -i '/\      nodeSelector: all()/a\   \ containerIPForwarding: "Enabled"' custom-resources.yaml
sed -i 's/192.168.0.0/172.22.0.0/g' custom-resources.yaml # this depends on your CIDR
kubectl create -f custom-resources.yaml

PacooMC commented 1 year ago

Hi @pinoOgni, thank you for your answer. Unfortunately, I could not manage to enable IP Forwarding in that way, as I am using the Microk8s Calico add-on. The way I found to enable it was by editing the config file located in /var/snap/microk8s/current/args/cni-network/cni.yaml, and adding the containerIPForwarding option.

From:

"ipam": { "type": "calico-ipam" }, "policy": { "type": "k8s" },

To:

"ipam": { "type": "calico-ipam" }, "container_settings": { "allow_ip_forwarding": true }, "policy": { "type": "k8s" },

Anyway, It might be a similar solution to the one you gave.

I am closing the issue.

Orange-OpenSource / towards5gs-helm