nginx-ingress-controller does not bind to required ingress ports even after specifying http-port and https-port in RKE cluster config

[meappy]

RKE version:

$ rke -v 
rke version v1.0.0

Docker version: (docker version,docker info preferred)

$ docker version 
Client: Docker Engine - Community
 Version:           18.09.8
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        0dd43dd87f
 Built:             Wed Jul 17 17:38:58 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.8
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.8
  Git commit:       0dd43dd87f
  Built:            Wed Jul 17 17:48:49 2019
  OS/Arch:          linux/amd64
  Experimental:     false

$ docker info 
Containers: 23
 Running: 18
 Paused: 0
 Stopped: 5
Images: 12
Server Version: 18.09.8
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.14.138-rancher
Operating System: RancherOS v1.5.4
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.65GiB
Name: rke3
ID: XHDN:HTIG:B6NA:YMYU:76Q2:C74X:QSLV:U3AQ:GZ3L:AYGR:F2XF:M42A
Docker Root Dir: /mnt/data/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

Operating system and kernel: (cat /etc/os-release, uname -r preferred)

$ cat /etc/os-release
NAME="RancherOS"
VERSION=v1.5.4
ID=rancheros
ID_LIKE=
VERSION_ID=v1.5.4
PRETTY_NAME="RancherOS v1.5.4"
HOME_URL="http://rancher.com/rancher-os/"
SUPPORT_URL="https://forums.rancher.com/c/rancher-os"
BUG_REPORT_URL="https://github.com/rancher/os/issues"
BUILD_ID=

Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO) Azure VM

cluster.yml file:

nodes:
- address: rke1
  role:
    - controlplane
    - etcd
    - worker
  user: rancher
  ssh_key_path: ~/.ssh/id_rsa-rke
- address: rke2
  role:
    - controlplane
    - etcd
    - worker
  user: rancher
  ssh_key_path: ~/.ssh/id_rsa-rke
- address: rke3
  role:
    - worker
    - etcd
    - controlplane
  user: rancher
  ssh_key_path: ~/.ssh/id_rsa-rke
ingress:
  provider: nginx
  extra_args:
    http-port: 8080
    https-port: 8443

Steps to Reproduce:

• I wish to configure the provided Rancher nginx-ingress-controller to listen on ports HTTP/8080 and HTTPS/8443
• When installing RKE, I specify the above ports in the cluster.yml file
• The RKE cluster deploys fine, however nginx-ingress-controller continues to be bound to HostPort 80 and 443

Results:

• To fix this, after RKE is deployed, I modified nginx-ingress-controller daemonset resource like so:

  kubectl edit daemonset nginx-ingress-controller -n ingress-nginx

• I did a a search and replace, saved the resource config, then did kubectl delete pod -l app=ingress-nginx -n ingress-nginx to get fresh pods deployed with the modified config
• Rancher now is bound to HostPort 8080 and 8443
• Below is my current daemonset resource config which binds HostPort HTTP/8080 and HTTPS/78443

   $ kubectl get daemonset nginx-ingress-controller -n ingress-nginx -o yaml
   apiVersion: apps/v1
   kind: DaemonSet
   metadata:
     annotations:
       deprecated.daemonset.template.generation: "2"
       field.cattle.io/publicEndpoints: '[{"nodeName":"local:machine-bvcdx","addresses":["rke6"],"port":8080,"protocol":"TCP","podName":"ingress-nginx:nginx-ingress-controller-d2m8d","allNodes":false},{"nodeName":"local:machine-bvcdx","addresses":["rke6"],"port":8443,"protocol":"TCP","podName":"ingress-nginx:nginx-ingress-controller-d2m8d","allNodes":false},{"nodeName":"local:machine-2f5db","addresses":["rke4"],"port":8080,"protocol":"TCP","podName":"ingress-nginx:nginx-ingress-controller-9xt8b","allNodes":false},{"nodeName":"local:machine-2f5db","addresses":["rke4"],"port":8443,"protocol":"TCP","podName":"ingress-nginx:nginx-ingress-controller-9xt8b","allNodes":false},{"nodeName":"local:machine-gqw2x","addresses":["rke5"],"port":8080,"protocol":"TCP","podName":"ingress-nginx:nginx-ingress-controller-wk25s","allNodes":false},{"nodeName":"local:machine-gqw2x","addresses":["rke5"],"port":8443,"protocol":"TCP","podName":"ingress-nginx:nginx-ingress-controller-wk25s","allNodes":false}]'
       kubectl.kubernetes.io/last-applied-configuration: |
         {"apiVersion":"apps/v1","kind":"DaemonSet","metadata":{"annotations":{},"name":"nginx-ingress-controller","namespace":"ingress-nginx"},"spec":{"selector":{"matchLabels":{"app":"ingress-nginx"}},"template":{"metadata":{"annotations":{"prometheus.io/port":"10254","prometheus.io/scrape":"true"},"labels":{"app":"ingress-nginx"}},"spec":{"affinity":{"nodeAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":{"nodeSelectorTerms":[{"matchExpressions":[{"key":"beta.kubernetes.io/os","operator":"NotIn","values":["windows"]},{"key":"node-role.kubernetes.io/worker","operator":"Exists"}]}]}}},"containers":[{"args":["/nginx-ingress-controller","--default-backend-service=$(POD_NAMESPACE)/default-http-backend","--configmap=$(POD_NAMESPACE)/nginx-configuration","--tcp-services-configmap=$(POD_NAMESPACE)/tcp-services","--udp-services-configmap=$(POD_NAMESPACE)/udp-services","--annotations-prefix=nginx.ingress.kubernetes.io","--http-port=8080","--https-port=8443"],"env":[{"name":"POD_NAME","valueFrom":{"fieldRef":{"fieldPath":"metadata.name"}}},{"name":"POD_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}}],"image":"rancher/nginx-ingress-controller:nginx-0.25.1-rancher1","livenessProbe":{"failureThreshold":3,"httpGet":{"path":"/healthz","port":10254,"scheme":"HTTP"},"initialDelaySeconds":10,"periodSeconds":10,"successThreshold":1,"timeoutSeconds":1},"name":"nginx-ingress-controller","ports":[{"containerPort":80,"name":"http"},{"containerPort":443,"name":"https"}],"readinessProbe":{"failureThreshold":3,"httpGet":{"path":"/healthz","port":10254,"scheme":"HTTP"},"periodSeconds":10,"successThreshold":1,"timeoutSeconds":1},"securityContext":{"capabilities":{"add":["NET_BIND_SERVICE"],"drop":["ALL"]},"runAsUser":33}}],"hostNetwork":true,"serviceAccountName":"nginx-ingress-serviceaccount","tolerations":[{"effect":"NoExecute","operator":"Exists"},{"effect":"NoSchedule","operator":"Exists"}]}}}}
     creationTimestamp: "2019-12-18T05:35:07Z"
     generation: 2
     name: nginx-ingress-controller
     namespace: ingress-nginx
     resourceVersion: "6720179"
     selfLink: /apis/apps/v1/namespaces/ingress-nginx/daemonsets/nginx-ingress-controller
     uid: 91669baa-2495-43b4-8299-e53a2e5a0862
   spec:
     revisionHistoryLimit: 10
     selector:
       matchLabels:
         app: ingress-nginx
     template:
       metadata:
         annotations:
           prometheus.io/port: "10254"
           prometheus.io/scrape: "true"
         creationTimestamp: null
         labels:
           app: ingress-nginx
       spec:
         affinity:
           nodeAffinity:
             requiredDuringSchedulingIgnoredDuringExecution:
               nodeSelectorTerms:
               - matchExpressions:
                 - key: beta.kubernetes.io/os
                   operator: NotIn
                   values:
                   - windows
                 - key: node-role.kubernetes.io/worker
                   operator: Exists
         containers:
         - args:
           - /nginx-ingress-controller
           - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
           - --configmap=$(POD_NAMESPACE)/nginx-configuration
           - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
           - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
           - --annotations-prefix=nginx.ingress.kubernetes.io
           - --http-port=8080
           - --https-port=8443
           env:
           - name: POD_NAME
             valueFrom:
               fieldRef:
                 apiVersion: v1
                 fieldPath: metadata.name
           - name: POD_NAMESPACE
             valueFrom:
               fieldRef:
                 apiVersion: v1
                 fieldPath: metadata.namespace
           image: rancher/nginx-ingress-controller:nginx-0.25.1-rancher1
           imagePullPolicy: IfNotPresent
           livenessProbe:
             failureThreshold: 3
             httpGet:
               path: /healthz
               port: 10254
               scheme: HTTP
             initialDelaySeconds: 10
             periodSeconds: 10
             successThreshold: 1
             timeoutSeconds: 1
           name: nginx-ingress-controller
           ports:
           - containerPort: 8080
             hostPort: 8080
             name: http
             protocol: TCP
           - containerPort: 8443
             hostPort: 8443
             name: https
             protocol: TCP
           readinessProbe:
             failureThreshold: 3
             httpGet:
               path: /healthz
               port: 10254
               scheme: HTTP
             periodSeconds: 10
             successThreshold: 1
             timeoutSeconds: 1
           resources: {}
           securityContext:
             capabilities:
               add:
               - NET_BIND_SERVICE
               drop:
               - ALL
             runAsUser: 33
           terminationMessagePath: /dev/termination-log
           terminationMessagePolicy: File
         dnsPolicy: ClusterFirst
         hostNetwork: true
         restartPolicy: Always
         schedulerName: default-scheduler
         securityContext: {}
         serviceAccount: nginx-ingress-serviceaccount
         serviceAccountName: nginx-ingress-serviceaccount
         terminationGracePeriodSeconds: 30
         tolerations:
         - effect: NoExecute
           operator: Exists
         - effect: NoSchedule
           operator: Exists
     updateStrategy:
       rollingUpdate:
         maxUnavailable: 1
       type: RollingUpdate
   status:
     currentNumberScheduled: 3
     desiredNumberScheduled: 3
     numberAvailable: 3
     numberMisscheduled: 0
     numberReady: 3
     observedGeneration: 2
     updatedNumberScheduled: 3

[meappy]

Hi, just adding a note, also please see https://github.com/rancher/rancher/issues/17857.

These options in RKE cluster config, configures spec.template.spec.containers.args however does not change spec.containers.ports therefore HostPort is still bound to 80 and 443

ingress:
  provider: nginx
  extra_args:
    http-port: 8080
    https-port: 8443

[ibrokethecloud]

I believe the documentation does mention that ingress controller is bound to port 80 and 443 only.

Template code indicating the ingress ports are bound to 80/443: https://github.com/rancher/kontainer-driver-metadata/blob/dev/rke/templates/nginx-ingress.go#L614

Host networking enabled on the daemonset. https://github.com/rancher/kontainer-driver-metadata/blob/dev/rke/templates/nginx-ingress.go#L190

[meappy]

@ibrokethecloud thanks for this, yes if you're referring to this documentation https://rancher.com/docs/rke/latest/en/config-options/add-ons/ingress-controllers/

Then you're right it does mention that it is bound to 80 and 443 on HostPort. I did mention that I don't believe this is now a bug, would Rancher consider adding this feature to allow changing default HostPort default 80 and 443 for the DaemonSet?

[meappy]

Linking related PR https://github.com/rancher/kontainer-driver-metadata/pull/109

[stevenmcastano]

I've run into the same thing... followed the post from @meappy and got everything moved to 8080 and 8443 as well.

However, one of the things I've noticed now is that in the Workloads screen in the Rancher interface it still shows things deployed on port 80 and when you click the links they still go to port 80... which is obviously dead now. If you change the port to 8080, it works fine. Also, kubectl get ingress -A still shows everything running on port 80.

[kinarashah]

Available to test in RKE v1.1.11-rc2

[bmdepesa]

rancher/rancher:v2.4-4095-head Setting in the RKE config

ingress:
  provider: nginx
  network_mode: hostPort
  http_port: 8080
  https_port: 8443

Binds the ports correctly

Leaving open until merged in v2.5-head/master-head

[bmdepesa]

Also tested in:

• rancher/rancher:v2.5-head bebfe0d
• rancher/rancher:master-head 55fc963