VPX-CIC CrashLoopBackOFF - Rbac 403 error /version

[mickael-decastro]

Describe the bug the vpx-ci pod are in CrashLoopBackOff and we have a Rbac 403 error to access to /namespaces/{namespace}/version during citrix-k8s-ingress-controller deployment. Auto route configuration is enable and cic namespace set.

Slack Channel To request an invitation to participate in the Slack channel, please supply your email address using this form: https://podio.com/webforms/22979270/1633242

To Reproduce

1. Steps
2. Version of the Citrix Ingress Controller -> citrix-k8s-ingress-controller:1.23.10
3. Version of MPX/VPX/CPX -> NSVPX-ESX-13.1-17.42_nc_64
4. Environment variables (minus secrets)
   • name: "NS_IP" value: "@IP"
   • name: "NS_USER" value: "nsroot"
   • name: "NS_PASSWORD" value: "**"
   • name: "EULA" value: "yes"
   • name: "NAMESPACE" value: "test-vpx"
   • name: "LOGLEVEL" value: "DEBUG"

Expected behavior

The vpx-cic pod should be start properly and create routes on external VPX appliance.

Logs kubectl logs cic-k8s-ingress-controller -n test-vpx

2022-04-14 08:55:27,532  - DEBUG - [ingressStatusUpdate.py:parseCPXServiceArg:52] (MainThread) Parsing the provided CPX Service argument
2022-04-14 08:55:27,541  - WARNING - [clienthelper.py:get:46] (MainThread) Request  to api server is forbidden, please check RBAC
2022-04-14 08:55:27,541  - WARNING - [kubernetes.py:call_K8sClientHelper_method:485] (MainThread) HTTP error occurred: 403 Client Error: Forbidden for url: https://@KUBE-API:443/apis/route.openshift.io/v1/namespaces/test-vpx/
2022-04-14 08:55:27,541  - INFO - [kubernetes.py:determine_environment:752] (MainThread) Failed to retrieve api "" with exception 403 Client Error: Forbidden for url: https://@KUBE-API:443/apis/route.openshift.io/v1/namespaces/test-vpx/. Assuming non-openshift environment
2022-04-14 08:55:27,547  - INFO - [clienthelper.py:get:49] (MainThread) Resource not found: /anthos.gke.io namespace test-vpx
2022-04-14 08:55:27,554  - INFO - [clienthelper.py:get:49] (MainThread) Resource not found: /ipamblocks/ namespace test-vpx
2022-04-14 08:55:27,555  - INFO - [nodeWatch.py:__init__:39] (MainThread) CNI found: default
2022-04-14 08:55:27,555  - DEBUG - [compositepattern.py:add:67] (MainThread) Adding [KubeAppInterfaceTestKube] component to the list
2022-04-14 08:55:27,555  - DEBUG - [compositepattern.py:add:69] (MainThread) Finished adding [KubeAppInterfaceTestKube] component to the list, current number of registered components is 1
2022-04-14 08:55:27,561  - WARNING - [clienthelper.py:get:46] (MainThread) Request /version to api server is forbidden, please check RBAC
2022-04-14 08:55:27,561  - WARNING - [kubernetes.py:call_K8sClientHelper_method:485] (MainThread) HTTP error occurred: 403 Client Error: Forbidden for url: https://@KUBE-API:443/namespaces/test-vpx/version
2022-04-14 08:55:27,562  - WARNING - [kubeeventwriter.py:write:114] (MainThread) CIC Event: FAILURE:/version-API-GET::Error access to API: Change Cluster Role for API authorization<Citrix-k8s-ingress-controller Cluster Role GitHub Link>
2022-04-14 08:55:27,573  - DEBUG - [clienthelper.py:post:87] (MainThread) <Response [201]>
Traceback (most recent call last):
  File "/usr/src/triton/nstriton.py", line 1118, in <module>
    main()
  File "/usr/src/triton/nstriton.py", line 1110, in main
    kubernetes(netskaler, conf)
  File "/usr/src/triton/nstriton.py", line 136, in kubernetes
    kube = KubernetesInterface(
  File "/usr/src/triton/kubernetes/kubernetes.py", line 263, in __init__
    self.k8s_version = self.get_kubernetes_version()
  File "/usr/src/triton/kubernetes/kubernetes.py", line 5553, in get_kubernetes_version
    success, response = self._get(api)
  File "/usr/local/lib/python3.9/dist-packages/retrying.py", line 49, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
  File "/usr/local/lib/python3.9/dist-packages/retrying.py", line 206, in call
    return attempt.get(self._wrap_exception)
  File "/usr/local/lib/python3.9/dist-packages/retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "/usr/local/lib/python3.9/dist-packages/six.py", line 719, in reraise
    raise value
  File "/usr/local/lib/python3.9/dist-packages/retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/usr/src/triton/kubernetes/kubernetes.py", line 528, in _get
    return self.call_K8sClientHelper_method(self.K8sClientHelper_GET_METHOD, api=api, namespace=namespace)
  File "/usr/src/triton/kubernetes/kubernetes.py", line 492, in call_K8sClientHelper_method
    raise http_err
  File "/usr/src/triton/kubernetes/kubernetes.py", line 473, in call_K8sClientHelper_method
    success, response = K8sClientHelper().get(api=api, namespace=namespace)
  File "/usr/local/lib/python3.9/dist-packages/retrying.py", line 49, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
  File "/usr/local/lib/python3.9/dist-packages/retrying.py", line 206, in call
    return attempt.get(self._wrap_exception)
  File "/usr/local/lib/python3.9/dist-packages/retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "/usr/local/lib/python3.9/dist-packages/six.py", line 719, in reraise
    raise value
  File "/usr/local/lib/python3.9/dist-packages/retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/usr/src/triton/../triton/kubernetes/clienthelper.py", line 47, in get
    response.raise_for_status()
  File "/usr/local/lib/python3.9/dist-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://@KUBE-API:443/namespaces/test-vpx/version

Additional context CIC configuration:

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cic-k8s-role
rules:
  - apiGroups: [""]
    resources: ["endpoints", "ingresses", "pods", "secrets", "nodes", "routes", "namespaces", "configmap", "service"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["namespaces"]
    resources: ["anthos.gke.io"]
    verbs: ["*"]
  - apiGroups: [""]
    resources: ["services/status"]
    verbs: ["patch"]
  - apiGroups: [""]
    resources: ["services"]
    verbs: ["get", "list", "watch", "patch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create"]
  - apiGroups: ["extensions"]
    resources: ["ingresses", "ingresses/status"]
    verbs: ["get", "list", "watch", "patch"]
  - apiGroups: ["apiextensions.k8s.io"]
    resources: ["customresourcedefinitions"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["citrix.com"]
    resources: ["rewritepolicies", "canarycrds", "authpolicies", "ratelimits"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["citrix.com"]
    resources: ["rewritepolicies/status", "canarycrds/status", "authpolicies/status", "ratelimits/status"]
    verbs: ["get", "list", "patch"]
  - apiGroups: ["citrix.com"]
    resources: ["vips"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: ["route.openshift.io"]
    resources: ["routes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["config.openshift.io"]
    resources: ["networks"]
    verbs: ["get", "list"]
  - apiGroups: ["network.openshift.io"]
    resources: ["hostsubnets"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["crd.projectcalico.org"]
    resources: ["ipamblocks"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["networking.k8s.io"]
    resources: ["ingresses", "ingresses/status", "ingressclasses"]
    verbs: ["get", "list", "watch", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cic-k8s-role
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cic-k8s-role
subjects:
- kind: ServiceAccount
  name: cic-k8s-role
  namespace: test-vpx
apiVersion: rbac.authorization.k8s.io/v1
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: cic-k8s-role
  namespace: test-vpx
---
apiVersion: v1
kind: Pod
metadata:
  name: cic-k8s-ingress-controller
  labels:
    app: cic-k8s-ingress-controller
spec:
      serviceAccountName: cic-k8s-role
      containers:
      - name: cic-k8s-ingress-controller
        image: "quay.io/citrix/citrix-k8s-ingress-controller:1.23.10"
        env:
        # Set NetScaler Management IP or SNIP in case of HA
        - name: "NS_IP"
          value: "@IP"
        # Set username for Nitro
        - name: "NS_USER"
          value: "nsroot"
        # Set user password for Nitro
        - name: "NS_PASSWORD"
          value: "*****"
        - name: "EULA"
          value: "yes"
        - name: "NAMESPACE"
          value: "test-vpx"
        - name: "LOGLEVEL"
          value: "DEBUG"
        args:
          - --ingress-classes
            tier-1-vpx
          - --feature-node-watch
            true
        imagePullPolicy: IfNotPresent
---

[aroraharsh23]

@mickael-decastro could you please try without using

• name: "NAMESPACE" value: "test-vpx" with the same cluster-roles.

[mickael-decastro]

Hi, thks for your quick reply.
Without Namespace it works ... but I would like to limit cic collection to a specific namesapace.

[aroraharsh23]

As of now, we don't support Role based CIC. You can run CIC in any namespace with cluster-role permissions

[mickael-decastro]

Thanks for the reply. Have you plan to add it to the road map ? Or something to limit CIC on a specific namespace ?

[subashd]

hi @mickael-decastro We have supported CIC with minimal privileges for a particular namespace with Role binding. Please refer below release notes for more detail. https://github.com/citrix/citrix-k8s-ingress-controller/releases/tag/1.28.2