search

aws-samples / opensource-4g-core-eks

Opensource 4G Core On AWS EKS
MIT No Attribution
16 stars 17 forks source link

open5gs installation with AWS EKS - Control plane pods are not coming , they are in init state - help needed #15

Open hannanbeediwale opened 1 year ago

hannanbeediwale commented 1 year ago

Hi

Facing issue in the last step at Helm Deployment:

Install Helm chart with the following command: helm -n open5gs install -f values.yaml epc-core ./

[[ec2-user@ip-10-0-0-14 x86-Architecture]$ kubectl get -n open5gs pods NAME READY STATUS RESTARTS AGE open5gs-hss-deployment-5d856c4f7b-jgmfx 0/1 Init:0/1 0 19m open5gs-mme-deployment-848cd7766-5t82q 0/1 Init:0/1 0 19m open5gs-nrf-deployment-55cc9fc696-zczwz 1/1 Running 0 19m open5gs-pcrf-deployment-f8fb984fb-7w7d8 0/1 Init:0/1 0 19m open5gs-sgwc-deployment-5bb6d97475-9p6st 0/1 Init:0/1 0 19m open5gs-sgwu-deployment-bfcc6c8cf-c98rr 1/1 Running 0 19m open5gs-smf-deployment-7f679b945-x79l4 0/1 Init:0/1 0 19m open5gs-upf-deployment-64cf6b6499-wpzrz 1/1 Running 0 19m open5gs-webui-5dd5944c56-jc59c 1/1 Running 0 19m [ec2-user@ip-10-0-0-14 x86-Architecture]$ [ec2-user@ip-10-0-0-14 x86-Architecture]$

Completed all the install steps Following this link to install:https://aws.amazon.com/blogs/opensource/open-source-mobile-core-network-implementation-on-amazon-elastic-kubernetes-service/

  1. Run the CloudFormation for infra creation (open5gs-infra.yaml).
  2. Bastion host configuration and K8s ConfigMap update.
  3. DocumentDB initialization.
  4. CoreDNS ConfigMap update to use Route 53 for 3GPP service interfaces.
  5. Run the CloudFormation for Multus worker node group creation (open5gs-worker.yaml).
  6. DNS controller and Multus-IP update controller deployment for the automation.
  7. Run shell script for cluster initialization (setting up namespace, etc.).
  8. Helm installation for all network functions.

[ec2-user@ip-10-0-0-218 x86-Architecture]$ kubectl -n open5gs get secret NAME TYPE DATA AGE diameter-ca Opaque 1 8h hss-tls kubernetes.io/tls 2 8h mme-tls kubernetes.io/tls 2 8h mongodb-ca Opaque 1 8h pcrf-tls kubernetes.io/tls 2 8h smf-tls kubernetes.io/tls 2 8h

[ec2-user@ip-10-0-0-218 x86-Architecture]$ kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE aws-node-r5bsr 1/1 Running 0 9m6s coredns-594b88dc6c-njnw9 1/1 Running 0 19m coredns-594b88dc6c-pj24l 1/1 Running 0 19m kube-multus-ds-5nqt9 1/1 Running 0 9m6s kube-proxy-cdbj4 1/1 Running 0 9m6s multus-route53-service-operator-645c94c4bb-5c4wf 1/1 Running 0 19m multus-secondary-ip-ec2-mapping-operator-74586574d4-zrzwz 1/1 Running 0 19m

[ec2-user@ip-10-0-0-14 x86-Architecture]$ kubectl describe pod open5gs-smf-deployment-7f679b945-x79l4 --namespace open5gs Name: open5gs-smf-deployment-7f679b945-x79l4 Namespace: open5gs Priority: 0 Node: ip-10-0-2-39.ca-central-1.compute.internal/10.0.2.39 Start Time: Mon, 22 May 2023 20:01:00 +0000 Labels: epc-mode=smf pod-template-hash=7f679b945 Annotations: k8s.v1.cni.cncf.io/network-status: [{ "name": "aws-cni", "interface": "dummybb591efea73", "ips": [ "10.0.2.228" ], "mac": "0", "default": true, "dns": {} },{ "name": "open5gs/ipvlan-multus-sub-1-cp", "interface": "net1", "ips": [ "10.0.4.208" ], "mac": "02:13:f3:9b:f0:3c", "dns": {} }] k8s.v1.cni.cncf.io/networks: [ { "name": "ipvlan-multus-sub-1-cp", "interface": "net1" } ] k8s.v1.cni.cncf.io/networks-status: [{ "name": "aws-cni", "interface": "dummybb591efea73", "ips": [ "10.0.2.228" ], "mac": "0", "default": true, "dns": {} },{ "name": "open5gs/ipvlan-multus-sub-1-cp", "interface": "net1", "ips": [ "10.0.4.208" ], "mac": "02:13:f3:9b:f0:3c", "dns": {} }] route53-service-name: [ { "name": "s5.smf.open5gs.service", "multus-int": "ipvlan-multus-sub-1-cp" } ] Status: Pending IP: 10.0.2.228 IPs: IP: 10.0.2.228 Controlled By: ReplicaSet/open5gs-smf-deployment-7f679b945 Init Containers: init-smf: Container ID: docker://349bab56b9e0e28c70594489ae364e8d96d504c30d2c17611f070f39e986dc84 Image: busybox:1.28 Image ID: docker-pullable://busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47 Port: Host Port: Command: sh -c Args: until nslookup gx.pcrf.open5gs.service >> /dev/null; do echo waiting for pcrf DNS record to be ready;done; until nslookup sx.upf.open5gs.service >> /dev/null; do echo waiting for upf DNS record to be ready; done State: Running Started: Mon, 22 May 2023 20:01:02 +0000 Ready: False Restart Count: 0 Environment: Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-s4zq9 (ro) Containers: smf: Container ID:
Image: 523531615794.dkr.ecr.ca-central-1.amazonaws.com/hannanrepo:open5gs-x86-aio Image ID:
Port: Host Port: Command: /bin/sh -c Args: sleep 10; open5gs-smfd -c /open5gs/config-map/smf.yaml; State: Waiting Reason: PodInitializing Ready: False Restart Count: 0 Environment: Mounts: /open5gs/config-map/diameter-smf.conf from open5gs-smf-diameter (rw,path="diameter-smf.conf") /open5gs/config-map/smf.yaml from open5gs-smf-config (rw,path="smf.yaml") /open5gs/diameter-ca/ from diameter-ca (rw) /open5gs/tls/ from smf-tls (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-s4zq9 (ro) Conditions: Type Status Initialized False Ready False ContainersReady False PodScheduled True Volumes: open5gs-smf-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: open5gs-smf-config Optional: false open5gs-smf-diameter: Type: ConfigMap (a volume populated by a ConfigMap) Name: open5gs-smf-diameter Optional: false diameter-ca: Type: Secret (a volume populated by a Secret) SecretName: diameter-ca Optional: false smf-tls: Type: Secret (a volume populated by a Secret) SecretName: smf-tls Optional: false kube-api-access-s4zq9: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: BestEffort Node-Selectors: nodegroup=control-plane Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message

Normal Scheduled 6m4s default-scheduler Successfully assigned open5gs/open5gs-smf-deployment-7f679b945-x79l4 to ip-10-0-2-39.ca-central-1.compute.internal Normal AddedInterface 6m3s multus Add eth0 [10.0.2.228/32] from aws-cni Normal AddedInterface 6m3s multus Add net1 [10.0.4.208/24] from open5gs/ipvlan-multus-sub-1-cp Normal Pulled 6m3s kubelet Container image "busybox:1.28" already present on machine Normal Created 6m3s kubelet Created container init-smf Normal Started 6m2s kubelet Started container init-smf [ec2-user@ip-10-0-0-14 x86-Architecture]$

mgonzalezo commented 12 months ago

I'm facing same error for some pods in open5s ns:

[ec2-user@ip-10-1-0-111 x86-Architecture]$ kubectl get -n open5gs pods NAME READY STATUS RESTARTS AGE open5gs-hss-deployment-68b4dc778f-s9p95 0/1 Init:0/1 0 13m open5gs-mme-deployment-77c4b4f8c-4t7kk 0/1 Init:0/1 0 13m open5gs-nrf-deployment-54f9cd8b65-5dkhm 1/1 Running 0 13m open5gs-pcrf-deployment-64ff487f65-85dqx 0/1 Init:0/1 0 13m open5gs-sgwc-deployment-56944dcd88-ztz7r 0/1 Init:0/1 0 13m open5gs-sgwu-deployment-76d4f6779b-m5qzq 1/1 Running 0 13m open5gs-smf-deployment-7775846df8-wmf5k 0/1 Init:0/1 0 13m open5gs-upf-deployment-78bccbffdd-ztpzc 1/1 Running 0 13m open5gs-webui-78975d9746-q7d4v 1/1 Running 0 13m

kubectl logs: [ec2-user@ip-10-1-0-111 x86-Architecture]$ kubectl logs -n open5gs open5gs-mme-deployment-77c4b4f8c-4t7kk Defaulted container "mme" out of: mme, init-mme (init) Error from server: Get "https://10.1.2.231:10250/containerLogs/open5gs/open5gs-mme-deployment-77c4b4f8c-4t7kk/mme": remote error: tls: internal error

Same error message for kube-system NS multus pods:

[ec2-user@ip-10-1-0-111 x86-Architecture]$ kubectl get -n kube-system pods NAME READY STATUS RESTARTS AGE aws-node-kr87b 2/2 Running 0 109m aws-node-s75l7 2/2 Running 0 109m coredns-58488c5db-hg956 1/1 Running 0 118m coredns-58488c5db-mv9vj 1/1 Running 0 118m kube-proxy-dll75 1/1 Running 0 109m kube-proxy-rn2wk 1/1 Running 0 109m multus-route53-service-operator-7bd78b8d7c-cp54s 0/1 CrashLoopBackOff 6 (98s ago) 7m17s multus-secondary-ip-ec2-mapping-operator-75c68fb48c-tczb9 0/1 CrashLoopBackOff 6 (85s ago) 7m17s [ec2-user@ip-10-1-0-111 x86-Architecture]$ kubectl logs -n kube-system multus-route53-service-operator-7bd78b8d7c-cp54s Error from server: Get "https://10.1.2.158:10250/containerLogs/kube-system/multus-route53-service-operator-7bd78b8d7c-cp54s/shell-operator": remote error: tls: internal error [ec2-user@ip-10-1-0-111 x86-Architecture]$

Error is observed after executing Staging environment- step 2 - "Run the ./cluster_initializer.sh"