nginx-meshctl fails on NGINX Ingress Controller CRDs

[darkn3rd]

When using manual injection with nginx-meshctl inject, there will be an error

STEPS

cat << EOF > manifests.yaml
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: dgraph
    chart: raw-v0.2.5
    component: ratel
    heritage: Helm
    release: ratel
  name: dgraph-ratel
spec:
  ports:
  - name: http-ratel
    port: 80
    targetPort: 8000
  selector:
    app: dgraph
    component: ratel
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: raw
    chart: raw-v0.2.5
    heritage: Helm
    release: ratel
  name: dgraph-ratel
spec:
  replicas: 1
  selector:
    matchLabels:
      app: dgraph
      component: ratel
  template:
    metadata:
      labels:
        app: dgraph
        component: ratel
    spec:
      containers:
      - command:
        - dgraph-ratel
        image: docker.io/dgraph/ratel:v21.03.2
        imagePullPolicy: null
        name: dgraph-ratel
        ports:
        - containerPort: 8000
          name: http-ratel
---
apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
  labels:
    app: raw
    chart: raw-v0.2.5
    heritage: Helm
    release: ratel
  name: dgraph-http
spec:
  externalDNS:
    enable: true
  host: ratel.devopsstudio.co
  routes:
  - action:
      pass: ratel
    path: /
  tls:
    cert-manager:
      cluster-issuer: letsencrypt-prod
    secret: tls-secret
  upstreams:
  - name: ratel
    port: 80
    service: dgraph-ratel
EOF 

cat manifest.yaml | nginx-meshctl inject

EXPECT RESULT

There wouldn't be an error for CRDs that are used by NGINX Ingress Controller like VirtualServer

ACTUAL RESULT

Cannot inject NGINX Service Mesh sidecar.
Error: NGINX Service Mesh returned an internal server error: error decoding file into k8s object: no kind "VirtualServer" is registered for version "k8s.nginx.org/v1" in scheme "k8s.io/client-go/kubernetes/scheme/register.go:74

NOTES/CONTEXT

I typically use helm template blah | nginx-meshctl inject or helmfile template | nginx-meshctl inject. These have a variety of resources created. With this limitation, I have to create multiple charts or helmfiles to segregate out the NGINX IC CRDs that cause errors.

NGINX tools should APIs created by NGINX.

[darkn3rd]

One reason I have to do manual injection with nginx-mesh-ctl inject is because there's no mechanism to exclude outbound/inbound for auto-injection, such as adding an annotation in the deployment spec template for example.

[sjberman]

One reason I have to do manual injection with nginx-mesh-ctl inject is because there's no mechanism to exclude outbound/inbound for auto-injection, such as adding an annotation in the deployment spec template for example.

See:

• https://docs.nginx.com/nginx-service-mesh/get-started/configuration/#pod-annotations
• https://docs.nginx.com/nginx-service-mesh/guides/inject-sidecar-proxy/#ignore-specific-ports

[darkn3rd]

That is great. Is there a web hook to auto-inject the side car based on annotations? Or is this now using labels at pod or namespace level? I would like to (1) only do injection when annotation is specified and (2) a web hook would use the annotation at pod or namespace, (3) have an annotation that can ignore ports, so when the web hook adds the side car, is plugs in the appropriate values. For (2) I guess a label is fine, not sure what the standard practice is for this.

If auto-injection is enabled, can it be limited to only pods/namespaces that have the label (or annotation)?

[sjberman]

Yes, if you take a look just above the Pod Annotation table that I linked above, you can see the injector.nsm.nginx.com/auto-inject Label, which be used on a Namespace or Pod to either enable or disable injection.

The recommended pattern here is to deploy the mesh with the --disable-auto-inject field set, and then enable the namespaces or pods that you want to have the sidecar using the label.

[f5-todd]

@darkn3rd Can you please us know if the above recommendation resolves your error?

[f5-todd]

Ahhh....with a more detailed reading of this ticket looks like there is more than one issue here. I'll create a bug and add to our backlog the CRD issue.