Jambonz -- SIP POD is always in CrashLoopBackOff state

[busyboy77]

Hi, I have successfully deployed the Jambonz on an on-prem K3s cluster. It has to be a brand new cluster to make things ready and there should be no already existing Ingress Controller installed to make things workable.

ef.com is my internal domain, so I can reach it internally.

now the main problems that I'm seeing

Helm Command that I used:

root@devops213:~# kubectl label node devops216.ef.com "voip-environment=sip"
node/devops216.ef.com labeled
root@devops213:~# kubectl label node devops217.ef.com "voip-environment=rtp"
node/devops217.ef.com labeled
root@devops213:~# kubectl taint node devops216.ef.com "sip=true:NoSchedule"
node/devops216.ef.com tainted
root@devops213:~# kubectl taint node devops217.ef.com "rtp=true:NoSchedule"
node/devops217.ef.com tainted

Added the HELM

root@devops213:~# helm repo add jambonz https://jambonz.github.io/helm-charts/
"jambonz" has been added to your repositories

HELM Install

helm install --namespace=jambonz \
--set "global.db.namespace=jambonz-db" \
--set "global.monitoring.namespace=jambonz-monitoring" \
--set "monitoring.grafana.hostname=grafana.ef.com" \
--set "monitoring.homer.hostname=homer.ef.com" \
--set "monitoring.jaeger.hostname=jaeger.ef.com" \
--set "webapp.hostname=portal.ef.com" \
--set "api.hostname=api.ef.com" \
--set cloud=none \
jambonz jambonz/jambonz

1 -- SIP Pod is always in CreashLoopBackOff State.

NAME                               READY   STATUS             RESTARTS         AGE
api-server-57b87d96cb-7j5pt        1/1     Running            0                149m
feature-server-86ff98b7f-xjf72     3/3     Running            0                149m
jambonz-sbc-rtp-vqhw8              2/2     Running            0                149m
jambonz-sbc-sip-9fqmb              1/3     CrashLoopBackOff   22 (2m57s ago)   35m
sbc-call-router-568678c4ff-dxppf   1/1     Running            0                149m
sbc-inbound-56bb6b697b-bzmpm       1/1     Running            5 (146m ago)     149m
sbc-outbound-6789b5bb67-tqx8q      1/1     Running            5 (147m ago)     149m
sbc-register-7bf8d6dc96-7jfpl      1/1     Running            0                149m
webapp-7859685df4-hmrft            1/1     Running            0                149m

2 -- logs for the pod and its subsequent containers are given below:

 kubectl -n jambonz logs -f jambonz-sbc-sip-9fqmb drachtio
SipTransport::init - contact: sip::5060;transport=udp,tcp
Uncaught exception: SipTransport::init - invalid contact sip::5060;transport=udp,tcp

sbc-options-handler

root@devops213:~# kubectl -n jambonz logs -f jambonz-sbc-sip-9fqmb sbc-options-handler

> sbc-options-handler@v0.7.5 start
> node app

node:events:498
      throw er; // Unhandled 'error' event
      ^

Error: connect ECONNREFUSED 127.0.0.1:9022
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1157:16)
Emitted 'error' event on Srf instance at:
    at Immediate.<anonymous> (/opt/app/node_modules/drachtio-srf/lib/srf.js:84:64)
    at processImmediate (node:internal/timers:466:21) {
  errno: -111,
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: '127.0.0.1',
  port: 9022
}

Node.js v17.4.0
npm notice
npm notice New minor version of npm available! 8.3.1 -> 8.15.1
npm notice Changelog: <https://github.com/npm/cli/releases/tag/v8.15.1>
npm notice Run `npm install -g npm@8.15.1` to update!
npm notice

SMPP

root@devops213:~# kubectl -n jambonz logs -f jambonz-sbc-sip-9fqmb smpp

> jambonz-smpp-esme@v0.7.5 start
> node app

{"level":30, "time": "2022-07-28T12:45:22.241Z","pid":19,"hostname":"devops216.ef.com","msg":"jambonz-smpp-esme listening for api requests at http://localhost:80"}
{"level":30, "time": "2022-07-28T12:45:22.249Z","pid":19,"hostname":"devops216.ef.com","msg":"jambonz-smpp-esme listening for smpp at http://localhost:2775"}

POd Describe

root@devops213:~# kubectl describe pod jambonz-sbc-sip-9fqmb -n jambonz
Name:         jambonz-sbc-sip-9fqmb
Namespace:    jambonz
Priority:     0
Node:         devops216.ef.com/192.168.2.216
Start Time:   Thu, 28 Jul 2022 17:45:04 +0500
Labels:       app=jambonz-sbc-sip
              controller-revision-hash=7486447d86
              pod-template-generation=1
Annotations:  <none>
Status:       Running
IP:           192.168.2.216
IPs:
  IP:           192.168.2.216
Controlled By:  DaemonSet/jambonz-sbc-sip
Init Containers:
  db-create-wait:
    Container ID:  containerd://2169eddea7d92256c9dfbff048b2444fe826d03c9f46748b002436c9c0082535
    Image:         kanisterio/mysql-sidecar:0.40.0
    Image ID:      docker.io/kanisterio/mysql-sidecar@sha256:7300e6d158fd9a82efef1387ac4cbfbbcebae1de27772dc69ae880721ab9cee4
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
      until mysql -u jambones -D jambones -h mysql.jambonz-db -p${MYSQL_PASSWORD} --protocol=tcp -e "select count(*) from accounts";
      do
        sleep 5
      done

    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 28 Jul 2022 17:45:05 +0500
      Finished:     Thu, 28 Jul 2022 17:45:05 +0500
    Ready:          True
    Restart Count:  0
    Environment:
      MYSQL_PASSWORD:  <set to the key 'MYSQL_PASSWORD' in secret 'jambonz-secrets'>  Optional: false
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-4m68s (ro)
Containers:
  drachtio:
    Container ID:  containerd://4d27b46fff58208824e5452b780802a6307b5508c6cd5e6d2df82bd18b36fdfb
    Image:         drachtio/drachtio-server:0.8.17-rc1
    Image ID:      docker.io/drachtio/drachtio-server@sha256:1b0a6e9f0b811ff9f9d70b8b5845337f5237f517992fe1c31a51bf6d544179ce
    Ports:         9022/TCP, 5060/UDP, 5060/TCP
    Host Ports:    9022/TCP, 5060/UDP, 5060/TCP
    Args:
      drachtio
      --loglevel
      info
      --cloud-deployment
      --sofia-loglevel
      3
      --homer
      heplify-server.jambonz-monitoring
      --homer-id
      10
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 28 Jul 2022 19:24:10 +0500
      Finished:     Thu, 28 Jul 2022 19:24:10 +0500
    Ready:          False
    Restart Count:  24
    Environment:
      DRACHTIO_SECRET:       <set to the key 'DRACHTIO_SECRET' in secret 'jambonz-secrets'>  Optional: false
      SOFIA_SEARCH_DOMAINS:  1
      SOFIA_SRES_NO_CACHE:   1
      CLOUD:                 none
      IMDSv2:
    Mounts:
      /etc/drachtio.conf.xml from jambonz-sbc-sip-conf (rw,path="drachtio.conf.xml")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-4m68s (ro)
  sbc-options-handler:
    Container ID:   containerd://beddab0598c300f43e3226b41f046069ade3e0e2e567d57c9d89cde7970c05e1
    Image:          jambonz/sbc-options-handler:0.7.5
    Image ID:       docker.io/jambonz/sbc-options-handler@sha256:325e5727907dd35d496e8a67a024ab35693b8448b6190f1aefee0adad4f7397f
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Thu, 28 Jul 2022 19:26:25 +0500
      Finished:     Thu, 28 Jul 2022 19:26:31 +0500
    Ready:          False
    Restart Count:  24
    Environment:
      NODE_ENV:                 production
      K8S:                      1
      JAMBONES_LOGLEVEL:        info
      JAMBONES_REDIS_HOST:      redis.jambonz-db
      JAMBONES_REDIS_PORT:      6379
      JAMBONES_MYSQL_DATABASE:  jambones
      JAMBONES_MYSQL_HOST:      mysql.jambonz-db
      JAMBONES_MYSQL_USER:      jambones
      JAMBONES_MYSQL_PASSWORD:  <set to the key 'MYSQL_PASSWORD' in secret 'jambonz-secrets'>  Optional: false
      DRACHTIO_HOST:            127.0.0.1
      DRACHTIO_SECRET:          <set to the key 'DRACHTIO_SECRET' in secret 'jambonz-secrets'>  Optional: false
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-4m68s (ro)
  smpp:
    Container ID:   containerd://7fb01bba5494e45520f3dc779dfa6376758dcae5bf7097c6e60bf02e6c623902
    Image:          jambonz/smpp-esme:0.7.5
    Image ID:       docker.io/jambonz/smpp-esme@sha256:858aab5e25dda9339d60eb0c17448c3fb5ba5f6721ad9913d5bc06dbb7f91b1f
    Port:           80/TCP
    Host Port:      80/TCP
    State:          Running
      Started:      Thu, 28 Jul 2022 17:45:14 +0500
    Ready:          True
    Restart Count:  0
    Environment:
      NODE_ENV:                 production
      JAMBONES_LOGLEVEL:        info
      K8S:                      1
      HTTP_PORT:                80
      AVOID_UDH:                1
      JAMBONES_REDIS_HOST:      redis.jambonz-db
      JAMBONES_REDIS_PORT:      6379
      JAMBONES_MYSQL_DATABASE:  jambones
      JAMBONES_MYSQL_HOST:      mysql.jambonz-db
      JAMBONES_MYSQL_USER:      jambones
      JAMBONES_MYSQL_PASSWORD:  <set to the key 'MYSQL_PASSWORD' in secret 'jambonz-secrets'>  Optional: false
      DRACHTIO_HOST:            127.0.0.1
      DRACHTIO_SECRET:          <set to the key 'DRACHTIO_SECRET' in secret 'jambonz-secrets'>  Optional: false
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-4m68s (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  jambonz-sbc-sip-conf:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      jambonz-sbc-sip-conf
    Optional:  false
  kube-api-access-4m68s:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              voip-environment=sip
Tolerations:                 node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/network-unavailable:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
                             sip:NoSchedule op=Exists
Events:
  Type     Reason   Age                  From     Message
  ----     ------   ----                 ----     -------
  Warning  BackOff  3m (x484 over 102m)  kubelet  Back-off restarting failed container

Any clue or comment will be much appreciate.

Also thanks for your YouTube video. It worked well for deploying the cluster.

[davehorton]

I think the issue is that the drachtio image in the sip daemonset needs to dynamically determine the ip address it should listen on. Most often, there is a private IP that the drachtio server binds to, as well as a public address assigned over the top. On cloud providers, like AWS, there is typically a metadata http request that can be made to determine the IPs.

Since you are running on bare metal, we can not use this requests to automagically determine the IPs. Instead, could you define these environment variables for the drachtio container in the sbc-sip daemonset:

LOCAL_IP PUBLIC_IP

[davehorton]

If you look at the entrypoint.sh file in the drachtio container, I think you will see what I am getting at..

[busyboy77]

So ideally, I will be using the DownwardAPI for the local IP and for the PUBLIC_IP, I'll have to assign it manually.. Please correct me if I'm wrong here.

I apologize for these botherations, but this is our project to deploy and we dont have any documentation except asking here.

[busyboy77]

I think I can use both LOCAL_IP and PUBLIC_IP from the DownwardAPI of Kubernetes

here is the link -- if you can suggest a better approach

https://kubernetes.io/docs/concepts/workloads/pods/downward-api/

spec.nodeName
the name of the [node](https://kubernetes.io/docs/concepts/architecture/nodes/) where the Pod is executing
status.hostIP
the primary IP address of the node to which the Pod is assigned
status.podIP
the pod's primary IP address (usually, its IPv4 address)

[davehorton]

you should join my slack channel -- just go to https://joinslack.jambonz.org

[busyboy77]

joining in