Serve the frontend on port 80/443

[ThomasThelen]

Right now the web service has to be accessed on port 30443. This should be on port 80/443 for a better user experience.

[gothub]

We are using the NGINX Ingress Controller (NIC) for providing external client connections to k8s services. Our k8s configuration is termed "bare metal" because we are running our own k8s service as apposed to using a cloud provided k8s such as Amazon EKS, Google Cloud Platform, etc. These cloud based k8s services provide a load balancer that is in front of their k8s service. We do not have a load balancer that is external to our k8s.

The documentation for NIC summarizes some options for enabling access to k8s services for bare metal configurations here. We are using the NodePort option which is why port 30443 has to be specified for URLs that access k8s services. Notice the recommendation to not reconfigure k8s to allow NodePort on 80/443.

There are other options described in this document, but I believe the preferred one is the self-provisioned edge, which is a reverse proxy in front of our k8s cluster that routes traffic to the NGINX NodePort.

@mbjones @ThomasThelen Do y'all have other options or thoughts on the ones mentioned here?

[amoeba]

Heya @gothub, thanks for chiming in here. We're eager to get this and other services talking on 80/443.

I read the links above and did some of my own research and I agree with you that this is how it's done. Two other write-ups indicate that your conclusion is correct:

• https://itnext.io/bare-metal-kubernetes-with-kubeadm-nginx-ingress-controller-and-haproxy-bb0a7ef29d4e
• https://techbeacon.com/devops/one-year-using-kubernetes-production-lessons-learned

The single point of failure part is a bummer but I also realize the UCSB network is likely to go down more than any load balancer VM we put up.

If we went with this approach, do you feel like we have next steps for making the production and development clusters available on 443/80? I'm happy to help out where I can, though I'm still coming up to speed on all this.

[mbjones]

All of that does indeed sound like it requires an HA Proxy or similar external service. However in one last gasp I thought I would describe what I've been doing locally and see if it might work for us. There's probably some issue you have all considered already, but I thought I would at least raise this approach.

I have installed the nginx-ingress controller locally on my Docker Desktop instance, and have been able to get an externally routed service available via an Ingress. It seems to do the job, but I am probably missing something. In this case, the Deployment is a node app on port 8080, the Service is configured to serve that targetport 8080 on port 80, and the Ingress is set up make that available on the host dev.magisa.org (on my network behind my NAT).

I installed the NGINX ingress controller in my default namespace ahead of all of this using its helm chart, like so from the ingress-nginx docs:

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm install ingress-nginx ingress-nginx/ingress-nginx

Here's the configuration file I used with kubectl apply -f:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: hello-kubernetes
  name: hello-deployment
  labels:
    app: hellodemo
spec:
  replicas: 3
  selector:
    matchLabels:
      app: hellodemo
  template:
    metadata:
      labels:
        app: hellodemo
    spec:
      containers:
      - env:
        image: gcr.io/google-samples/node-hello:1.0
        imagePullPolicy: IfNotPresent
        name: hello-deployment
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: hellodemo
  name: hello-service
  namespace: hello-kubernetes
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 8080
  selector:
    app: hellodemo
  sessionAffinity: None
  type: ClusterIP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hello-ingress
  namespace: hello-kubernetes
spec:
  ingressClassName: nginx
  rules:
  - host: "dev.magisa.org"
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: hello-service
            port:
              number: 80

When I access http://dev.magisa.org/ the app works on port 80. Is this helpful? What else do we need?

[gothub]

@mbjones @amoeba The service shown above is 'ClusterIP', so I'm not sure how it is providing external connections without NodePort, maybe Docker is using iptables in the background for this to connect the external port to the internal network. I'm not sure how to replicate this in k8s.

The k8s config for the nginx-controller is:

# Source: ingress-nginx/templates/controller-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  type: NodePort
  ports:
    - name: http
      port: 80
      nodePort:30080
      protocol: TCP
      targetPort: http
    - name: https
      port: 443
      nodePort: 30443
      protocol: TCP
      targetPort: https

The nodePort:30443 could be changed to nodePort:443 with a reconfiguration of k8s, but as mentioned in the NGINX docs referenced about, this isn't recommended:

While it may sound tempting to reconfigure the NodePort range using the --service-node-port-range API server flag to include unprivileged ports and be able to expose ports 80 and 443, doing so may result in unexpected issues including (but not limited to) the use of ports otherwise reserved to system daemons and the necessity to grant kube-proxy privileges it may otherwise not require.
This practice is therefore discouraged. See the other approaches proposed in this page for alternatives.

I could try this out on dev k8s and see how it goes, or test any other config that y'all can recommend for the ingress controller.

[mbjones]

Well, I am unclear how it works (k8s networking is beyond my ken), but I do know that 1) I did not configure it with NodePort, and 2) I can access it from another machine on my local network using the IP address of my laptop (not the internal IP addresses that k8s allocates). This is why I thought this might be a viable approach. I read all of the documents you sent before, including the ones you just repeated, and I agree they seem to say it is not possible. But, I also have noted that kubernetes documentation often lags its features by 6 months, and blog posts from a year ago are frequently no longer correct. So, some degree of empiricism is I think required in what works. Can you t least try the approach I outlined and see if it works? The hello-world example that I set up only takes a few minutes to try out, and might be instructive. Of course, it does depend on having the nginx controller set up the same way, which might be different from how it is now installed on the clusters. Maybe worth a try?

[gothub]

Yes, I'll try the approach you outlined on the dev k8s cluster and report back here.

[gothub]

The NGINX Ingress Controller deployment has been modified and tested on dev k8s using the hostPort mechanism to specify that the NGINX pod is connected directly to the host machines port 80 and 443, without a k8s service using the 'NodePort' mechanism. This approach, on dev k8s, is currently working - providing external connections to port 80, 443, for example:

• https://api.test.dataone.org/bookkeeper/v1/quotas
• https://api.test.dataone.org/quality
• https://stage.gnis-ld.org/

Note that no NGINX k8s service is running, which was required for the previous configuration.

Also, it may be necessary to specify a k8s 'host affinity' so that NGINX pod is scheduled to run on the node that the DNS name points to, i.e. for dev k8s docker-dev-ucsb-1.test.dataone.org which the DNS name api.test.dataone.org points to, so that the correct node's ports 80 and 443 are connected to the NGINX pod.

To enable this change, the NGINX controller-deployment.yaml was modified to simply add the hostPort to the ports: section:

spec:
  ...
  selector:
  ...
    spec:
      ...
      containers:
      ...
          ports:
            - name: http
              containerPort: 80
              hostPort: 80          <<< new line
              protocol: TCP
            - name: https
              containerPort: 443
              hostPort: 443          <<< new line
              protocol: TCP
...

The official k8s documentation doesn't mention the use of hostPort, except to say don't use it unless you have no other choice: https://kubernetes.io/docs/concepts/configuration/overview/

Examples of using this approach are described here:

• https://alesnosek.com/blog/2017/02/14/accessing-kubernetes-pods-from-outside-of-the-cluster/
• https://downey.io/blog/skip-kubernetes-loadbalancer-with-hostport-daemonset/

[gothub]

Regarding the use of hostPort - additional steps that may be required for use in a multi-control node setup:

• update the controller-deployment.yaml so that the number of replicas matches the number of control nodes. This will ensure that if the active control node changes, the NGINX Ingress Controller running on that node will be available to receive traffic
• update the 'node affinityfor NGINX Ingress Controller so that these pods will only be scheduled on the control nodes, which will have anode label` defined for them. This approach is described here.

[ThomasThelen]

We have https://gnis-ld.org/ up and running, but we need to deploy an official tag on it (not related to this issue). Thanks all for the help in getting this put together!