Support contour on unmanaged-clusters multi-nodes

[jorgemoralespou]

Bug Report

When running an unmanaged cluster with multiple nodes, only the control plane node will be exposed to the host when using the -p flags. So if you create a cluster like this tanzu unmanaged-cluster create tanzu-cluster -p 80:80 -p 443:443 --worker-node-count=2 contour will not properly enable access to the users. In order to support contour in this setup, what I did was:

• Create a cluster:

  tanzu unmanaged-cluster create tanzu-cluster -p 80:80 -p 443:443 --worker-node-count=2

• Label the control plane node (as instructed in this doc). I could have used the provider configuration, but this was simple for troubleshooting.

  kubectl label node tanzu-cluster-control-plane ingress-ready=true

  NOTE: That the name of the node is dependant on the name of the cluster

• Deployed contour package (which will not work as expected):

  tanzu package install contour -p contour.community.tanzu.vmware.com -v 1.20.1 -f /tmp/values-contour.yaml

  The values file is:

  envoy:
  service:
    type: ClusterIP
  hostPorts:
    enable: true

  • NOTE*: This will expose hostPorts but on the nodes where envoy run, which is not the container that got exposed to the host

• Create a secret with an overlay. This step is here because tanzu CLI does not support creating a package with an overlay at deploy time.

  apiVersion: v1
  kind: Secret
  metadata:
  name: contour-envoy-to-controlplane-overlay
  stringData:
  contour-envoy-to-controlplane.yaml: |
    #@ load("@ytt:overlay", "overlay")
  
    #@overlay/match by=overlay.subset({"kind": "DaemonSet", "metadata":{"name":"envoy"}})
    ---
    spec:
      template:
        spec:
          #@overlay/match missing_ok=True
          nodeSelector:
            ingress-ready: "true"
          #@overlay/match missing_ok=True
          tolerations:
          - key: node-role.kubernetes.io/control-plane
            operator: Equal
            effect: NoSchedule
          - key: node-role.kubernetes.io/master
            operator: Equal
            effect: NoSchedule

  Apply this secret to the same namespace as the packageinstall (default by default).

• Modify the package install to have an annotation that instructs the package to use this secret:

  apiVersion: packaging.carvel.dev/v1alpha1
  kind: PackageInstall
  metadata:
  annotations:
    ext.packaging.carvel.dev/ytt-paths-from-secret-name.0: contour-envoy-to-controlplane-overlay
  ...

• Wait to reconcile, or use new kapp-controller cli to kick reconcile via:

  kctrl app kick -a contour -n default -y

• Deploy a test service to verify things work:

  apiVersion: v1
  kind: Namespace
  metadata:
  name: testapp
  ---
  apiVersion: v1
  kind: ServiceAccount
  metadata:
  name: testapp
  namespace: testapp
  ---
  apiVersion: apps/v1
  kind: Deployment
  metadata:
  labels:
    app: testapp
  name: testapp
  namespace: testapp
  spec:
  replicas: 1
  selector:
    matchLabels:
      app: testapp
  template:
    metadata:
      labels:
        app: testapp
    spec:
      containers:
      - image: gcr.io/kuar-demo/kuard-amd64:1
        name: testapp
      serviceAccountName: testapp
  ---
  apiVersion: v1
  kind: Service
  metadata:
  labels:
    app: testapp
  name: testapp
  namespace: testapp
  spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 8080
  selector:
    app: testapp
  sessionAffinity: None
  type: ClusterIP
  ---
  apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
  name: testapp
  labels:
    app: testapp
  namespace: testapp
  spec:
  rules:
  - host: testapp.127-0-0-1.sslip.io
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: testapp
            port:
              number: 80

• Curl testapp.127-0-0-1.sslip.io

[jpmcb]

only the control plane node will be exposed to the host when using the -p flags.

Would using the rawKindConfig via a configuration file not alleviate this step? Something like:

ClusterName: my-cluster
KubeconfigPath: ""
ExistingClusterKubeconfig: ""
NodeImage: ""
Provider: kind
ProviderConfiguration:
  rawKindConfig: |
    nodes:
    - role: worker
    - role: worker
    - role: control-plane
      kubeadmConfigPatches:
      - |
        kind: InitConfiguration
        nodeRegistration:
          kubletExtraArgs:
            node-labels: "ingress-ready=true"
            authorization-mode: "AlwaysAllow"
      extraPortMappings:
      - containerPort: 80
        hostPort: 80
        protocol: TCP
        listenAddress: "0.0.0.0"
      - containerPort: 443
        hostPort: 443
        protocol: TCP
        listenAddress: "0.0.0.0"
Cni: calico
CniConfiguration: {}
PodCidr: 10.244.0.0/16
ServiceCidr: 10.96.0.0/16
TkrLocation: ""
AdditionalPackageRepos: []
PortsToForward: []
SkipPreflightChecks: false
ControlPlaneNodeCount: "1"
WorkerNodeCount: "0"
InstallPackages: []
LogFile: ""

As far as applying the tolerations, that is something that kind also recommends.

I'd like to get @vmware-tanzu/pkg-contour-owners input here since this seems like something out of scope for the kind unmanaged-cluster provider.

It'd be really really great if contour just worked out of the box for unmanaged-cluster without having to apply special secrets and tolerations to make kapp happy. Is this something we can add to the spec of the contour deployment overlay? Something like:

envoy:
  service:
    type: ClusterIP
  hostPorts:
    enable: true
  nodeSelector:
    ingress-ready: "true"
  tolerations:
    - key: node-role.kubernetes.io/control-plane
      operator: Equal
      effect: NoSchedule
    - key: node-role.kubernetes.io/master
      operator: Equal
      effect: NoSchedule

[github-actions[bot]]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.