Working Kubernetes branch

[MatMeredith]

Peter. I'd like to merge this into master. Could I ask you to review it please? It's basically your branch with a set of changes that I made to get working calls on GKE (and in some places to remove references to stuff outside of PC etc. and made one or two simplifications like defaulting to a 1 pod etcd cluster). Questions...

• I assume the changes to icscf_uri, scscf_uri etc. in the base Dockerfile were so that we can have just sprout in DNS rather than icscf.sprout and scscf.sprout? Would it have been difficult to get Kubernetes to support both?
• I've added "loadbalancer" config to the Bono and Ellis yaml files so that the services are externally accessible. This obviously depends on Kubernetes running in a cloud provider that supports it but I didn't see any other obvious default... I've commented in the readme that it might need to be changed.

[plwhite]

I had a look through. Nothing except the DNS leaps out as being problematic, and if it works for you I think you can mostly release it - I'll make sure we try it out in due course in case of anything odd in an environmental sense.

The problem with DNS was that Kubernetes already has a DNS namespacing structure, which means that DNS names are all something like: svc.ns.base where svc is the service name, ns is the namespace name, and base is some global thing such as site1.kubernetes.local. Each pod has a base DNS of ns.base. You cannot put dots into either service or namespace names to avoid breaking this, and so you cannot have a DNS lookup of "iscf.sprout". There may be some way to get round this (other than just changing the name used - you could play games with a local dnsmasq instance, for example), but it's annoyingly deeply baked in.

[resouer]

@plwhite where is iscf.sprout used?

Kubernetes can set subdomain, for example: myapp.mysubdomain, does that help?

[resouer]

@MatMeredith Here's an example:

apiVersion: v1
kind: Service
metadata:
  name: sprout
spec:
  selector:
    service: sprout
  clusterIP: None
  ports:
    - name: foo # Actually, no port is needed.
      port: 1234 
      targetPort: 1234

Then two sprout deploys, one for iscf, one for scscf

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: iscf-sprout
spec:
  replicas: 1
  selector:
    matchLabels:
      service: sprout
  template:
    metadata:
      labels:
        service: sprout
        snmp: enabled
    spec:
      hostname: iscf
      subdomain: sprout
      containers:
      - image: busybox
        command:
        - sleep
        - "3600"
        name: busybox

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: scscf-sprout
spec:
  replicas: 1
  selector:
    matchLabels:
      service: sprout
  template:
    metadata:
      labels:
        service: sprout
        snmp: enabled
    spec:
      hostname: scscf
      subdomain: sprout
      containers:
      - image: busybox
        command:
        - sleep
        - "3600"
        name: busybox

@plwhite I can make sure DNS part works in my example, any other problem in your mind?

[MatMeredith]

@plwhite Thanks. Yes, I'd figured the issue you were working around with the DNS and I think I've maintained your behaviour, but I've moved the functionality to https://github.com/Metaswitch/clearwater-infrastructure/pull/441 instead of having these overrides in the Dockerfile.

@resouer So, the way that Clearwater usually works is described here: http://clearwater.readthedocs.io/en/stable/Clearwater_DNS_Usage.html. Specifically the Sprout cluster should be accessible via either icscf.sprout. or scscf.sprout. (and ideally SRV lookups on these names would return port 5052 or 5054 respectively).

I think that means that we need the an ICSCF sprout service exposed as icscf.sprout. and mapping to port 5052 on the underlying pods and an SCSCF sprout service exposed as scscf.sprout. and mapping to port 5054, but it looks like your subdomains are being applied to the pods? Or am i misunderstanding? Is there any way to achieve what is required here?

[resouer]

@MatMeredith It seems https://github.com/Metaswitch/clearwater-infrastructure/pull/441/files already fix the dns problem above?

Yes my yaml is only a workaround and subdomain is pod-hostname.subdomain

[MatMeredith]

@resouer Yes, that PR "solves" the issue, but it's really only a workaround rather than a fix.