SIP in k8s question(s)

[davehorton]

Great work and great demo at astricon!

This is not really an "issue" more of a question, but since it seems you put this out there for education/enlightment perhaps you won't mind a question..

Running SIP servers in k8s seems to present certain challenges. Inbound call scenarios are more easily handled, but if one sip server in a k8s cluster sends a sip request to the outside world over udp, I always wondered how would we ensure that k8s wouldn't try to route the associated sip response to a different pod than the one that sent the original request?

In your case, I think I see what you have done, but want to make sure I do.

First, you are using kamailio as the sip server that will talk to the "outside world". You then: i) used a Daemonset, to ensure that only 1 kamailio pod will ever run on a single node ii) set hostNetwork: true here so that kamailio will bind to the actual IP address of the node it is running on. This goes hand-in-hand with the above point, since using hostNetwork we need to ensure k8s doesn't try to schedule two of these on the same node (one would fail to bind to port 5060) iii) used a nodeSelector to be able to specify only specific nodes that you want kamailio to run on (this way, you dont have kamailio running on every node, and also if needed for performance you could give it its own nodes).

Therefore, SIP requests that kamailio sends out of the cluster will have its "actual" IP address in the Via and Contact headers, and responses will get back to the same server.

Meanwhile, as far as asterisk, these will only take or send SIP messages to kamailios within the cluster I suppose, therefore they can be standard k8s deployments and services -- e.g you can run multiple asterisk pods on the same node, they will use simple dns/service names to locate kamailios, and they do not need to have hostNetwork enabled, etc. etc.

Have I got that right, and am I missing any key insights?

[Ulexus]

That's it exactly. Since the kamailio nodes do not, generally, need to scale anywhere near the degree as applications or Asterisks, the extra complexity and aberrations in the kamailio pods are less... concerning.

[davehorton]

Thanks. Now, as far as someone on the "outside" sending an INVITE to the kamailios......I guess there would have to be DNS entries for those IP addresses (of the nodes that kamailio is running on)? If so, is there also an intent or mechanism for updating those DNS records dynamically? Or are you doing something else?

[Ulexus]

Yes, there are several options in that realm. Each cloud provider offers their own DNS and API, of course. I personally use dnsimple.com, which offers a reasonably-flexible API (as well as support for more esoteric records like NAPTR).

I have written a few of these, though none is presently open source. In general, though, you just create a service which watches the Endpoints of the kamailio Service. When those change, you execute an update to the DNS service's API with the updated IP addresses.

You will generally have to look a little deeper than just the Endpoints (since those are not typically public IP addresses), and find the Nodes on which those kamailio Pods are running, and retrieve the public IP addresses for those Nodes.

[Ulexus]

I have also done this IP discovery in advance before, setting keys in a hosted etcd, redis, or SQL database with the public IPs. There isn't really much advantage to this, though, unless you cannot discover the public IP addresses for the Nodes from the k8s API.

[davehorton]

Thanks, this is super helpful

[davehorton]

one more question -- it looks like the service type you create for kamailio is a clusterIP. Since that doesn't give a publically reachable address, and by default a GCP cluster doesn't allow traffic into port 5060 in the kamailio node pool, how exactly do you allow the SIP traffic in from the outside world?

I'm trying to mimic a similar setup and cant get sip traffic to reach my sip edge servers at their public address...

[juansalvatella]

We have this setup with hostNetwork: true and opened up the port 5060 in the firewall rules.

On Tue, Mar 19, 2019 at 4:29 PM Dave Horton notifications@github.com wrote:

one more question -- it looks like the service type you create for kamailio is a clusterIP. Since that doesn't give a publically reachable address, and by default a GCP cluster doesn't allow traffic into port 5060 in the kamailio node pool, how exactly do you allow the SIP traffic in from the outside world?

I'm trying to mimic a similar setup and cant get sip traffic to reach my sip edge servers at their public address...

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/CyCoreSystems/asterisk-k8s-demo/issues/5#issuecomment-474424523, or mute the thread https://github.com/notifications/unsubscribe-auth/AELLA0ocsJMBozYxyAiAZZ_G_i_3qBYDks5vYQI-gaJpZM4Y4UvR .

[davehorton]

Interesting...current k8s on GCP does not let me set firewall rules via tags on the nodes themselves (i.e. as I would normally do on a plain on old compute node).

[Ulexus]

The clusterIP (internal) Service is for maintaining the internal Endpoint set, so that we can tell the Asterisk instances how to reach the kamailio instances. It is not used for external access.

As @juansalvatella mentioned, you do have to open the firewall appropriately. One of the important hooks here, which may be understated, is that you can tie firewall rules to node labels/tags in GCP. Thus, I tie SIP access to the kamailio node pool instance tags.

[Ulexus]

Unfortunately, kubernetes node tags and GCE instance tags are not the same thing. However, when you create your node pools, you can configure instance tags which will be applied.

[Ulexus]

Each cloud platform has a different way to approach the firewall rule application, so I considered that out-of-scope for this demonstration.

[davehorton]

got it. I think I see how to do it...