Kube-mgmt syncing issue with ConfigMap kube-root-ca.crt and unexpected logs

[DorB-P]

Description:

Kube-mgmt attempts to sync ConfigMap kube-root-ca.crt from namespace OPA when applying the admission-controller.yaml from the official OPA Kubernetes Ingress Validation tutorial. Even on a clean cluster with only the OPA namespace, tls secret and the bundle-server, the following issues occur:

Expected behavior:

• kube-mgmt should only sync ConfigMaps as mentioned in Policies and data loading
• kubectl logs -l app=opa -c opa -f -n opa should only show logs with "resp_status": 200 after tutorial step 6.

Actual effects: Logs show a PUT /v1/policies/opa/kube-root-ca.crt/ca.crt request every minute with "resp_status": 400.

• Debug log level example:

  {
  "client_addr": "127.0.0.1:43438",
  "level": "info",
  "msg": "Sent response.",
  "req_id": 179,
  "req_method": "PUT",
  "req_path": "/v1/policies/opa/kube-root-ca.crt/ca.crt",
  "resp_body": "{\n  \"code\": \"invalid_parameter\",\n  \"message\": \"error(s) occurred while compiling module(s)\",\n  \"errors\": [\n    {\n      \"code\": \"rego_parse_error\",\n      \"message\": \"unexpected minus token: expected number\",\n      \"location\": {\n        \"file\": \"opa/kube-root-ca.crt/ca.crt\",\n        \"row\": 1,\n        \"col\": 2\n      },\n      \"details\": {\n        \"line\": \"-----BEGIN CERTIFICATE-----\",\n        \"idx\": 1\n      }\n    }\n  ]\n}\n",
  "resp_bytes": 421,
  "resp_duration": 1.304073,
  "resp_status": 400,
  "time": "2024-01-22T20:22:52Z"
  } 

• kubectl get configmap kube-root-ca.crt -o yaml shows openpolicyagent.org/policy-status annotations:

  root-ca.crt:
  apiVersion: v1
  data:
  ca.crt: |
  -----BEGIN CERTIFICATE-----
  <censored>
  -----END CERTIFICATE-----
  kind: ConfigMap
  metadata:
  annotations:
  kubernetes.io/description: Contains a CA bundle that can be used to verify the
    kube-apiserver when using internal endpoints such as the internal service IP
    or kubernetes.default.svc. No other usage is guaranteed across distributions
    of Kubernetes clusters.
  openpolicyagent.org/policy-status: '{"status":"error","error":{"code":"invalid_parameter","message":"error(s)
    occurred while compiling module(s)","errors":[{"code":"rego_parse_error","message":"unexpected
    minus token: expected number","location":{"file":"opa/kube-root-ca.crt/ca.crt","row":1,"col":2},"details":{"line":"-----BEGIN
    CERTIFICATE-----","idx":1}}]}}'
  creationTimestamp: "2024-01-22T17:26:26Z"
  name: kube-root-ca.crt
  namespace: opa

Steps to Reproduce:

Follow the tutorial until step 6 and run the following, give it a minute:

kubectl logs -l app=opa -c opa -f -n opa
kubectl get configmap kube-root-ca.crt -o yaml

Environment:

• OPA kube-mgmt version:
• Kubernetes version: openpolicyagent/kube-mgmt:2.0.1
• Configuration: copied from the tutorial
• GKE Server Version: v1.27.3-gke.100
• Kind Version: kind v0.20.0 go1.21.5 windows/amd64
• Both Cluster nodes are running Linux, my Local Laptop Operating System: Windows 11

[eshepelyuk]

Few notes from current kube-mgmt spare time maintainer

1. that tutorial refers to extremely old version, you should use the latest one.
2. the preferred way to setup kube-mgmt is to use Helm chart that is covered with e2e tests, that are executed on each release.
3. there is no guarantee that mentioned tutorial is working at all, tbh not even sure who is updating it, if anyone.
4. also, if you are looking for admission controll for k8s resources, you'd rather should try Gatekeeper.

[DorB-P]

Thanks @eshepelyuk, I missed that helm! I'll try use it as soon as I can and update the issue if fixed. It does look like the tutorial is being maintained on OPA side (opa latest version), maybe not, who knows.

regarding Gatekeeper - I am also playing with that project atm, but the functionalities I require are so much harder to implement. AFAICT:

1. Decision logs (with logDenies option) are currently only streamed to the gatekeeper's pods stdout, and I need those, so I must have my own app in cluster to monitor and collect them (and send to my remote server). Those Gatekeeper Audit logs are not what I look for...
2. I love the Bundle API Ideology, perfect for my use-case! I wish Gatekeeper will implement something similar - maybe even a periodic kubectl apply --kustomize kustomization.yaml where kustomization.yaml points to my personal remote library... ATM I must a have a privileged pod that will periodically sync my remote library and apply the templates and constraints to the cluster. with Bundle API this was a gift!

So yea, I'm trying Gatekeeper, which means I need to create an image that will do 1 and 2 for my needs.

Thanks again for the clarification, I'll keep you posted if this was fixed by latest kube-mgmt version

[eshepelyuk]

Regarding decision logs there is 2 possibilities

• adjust OPA settings to send them to remote endpoint via HTTP, you can find the docs in OPA site.
• use some in-cluster collector, like fluent-bit, that watches stdout of pod container. we used to use this option because we could do some preprocessing with lua before storing the logs.

Regarding Gatekeeper, I can't advice myself, but looks like it has bigger community and get more attention, so you can also ask for help in their github or slack.

[DorB-P]

Closing for now. Will re-open if I see this happens with newer kube-mgmt