[Bug] Temporal Helm Deployment Constraints Violations on GKE Autopilot

[LukaGiorgadze]

What are you really trying to do?

I'm attempting to deploy Temporal on a GKE Autopilot cluster using Helm.

Describe the bug

When deploying Temporal on a GKE Autopilot cluster using Helm, I encounter constraints violations due to the configuration that Autopilot does not allow. This includes issues with hostNetwork, hostPID, hostPath, and privileged containers.

The violations details provided by the Helm installation failure are as follows:

• Enabling hostNetwork and hostPID are not allowed in Autopilot.
• Container node-exporter specifies host ports [9100], which are disallowed in Autopilot.
• Several hostPath volume configurations are used that are not allowed in Autopilot. Allowed path prefixes for hostPath volumes in Autopilot are [/var/log/].
• Container configure-sysctl is privileged, which is not allowed in Autopilot.

Minimal Reproduction

• Create a GKE Autopilot cluster.
• Use the Helm command to install Temporal: helm install temporaltest . --timeout 1200s
• Observe the mentioned constraint violations.

Environment/Versions

• OS and processor: M2 Mac
• Temporal Version: 0.28.0
• GKE Autopilot

Additional context

Output:

W0912 12:39:11.601304   56930 warnings.go:70] autopilot-default-resources-mutator:Autopilot updated DaemonSet default/temporaltest-prometheus-node-exporter: defaulted unspecified resources for containers [node-exporter] (see http://g.co/gke/autopilot-defaults)
W0912 12:39:11.877105   56930 warnings.go:70] autopilot-default-resources-mutator:Autopilot updated Deployment default/temporaltest-kube-state-metrics: defaulted unspecified resources for containers [kube-state-metrics] (see http://g.co/gke/autopilot-defaults)
W0912 12:39:11.877124   56930 warnings.go:70] autopilot-default-resources-mutator:Autopilot updated Deployment default/temporaltest-admintools: defaulted unspecified resources for containers [admin-tools] (see http://g.co/gke/autopilot-defaults)
W0912 12:39:11.911690   56930 warnings.go:70] autopilot-default-resources-mutator:Autopilot updated Deployment default/temporaltest-grafana: defaulted unspecified resources for containers [download-dashboards, grafana] (see http://g.co/gke/autopilot-defaults)
W0912 12:39:11.911741   56930 warnings.go:70] autopilot-default-resources-mutator:Autopilot updated Deployment default/temporaltest-prometheus-server: defaulted unspecified resources for containers [prometheus-server-configmap-reload, prometheus-server] (see http://g.co/gke/autopilot-defaults)
W0912 12:39:11.924644   56930 warnings.go:70] autopilot-default-resources-mutator:Autopilot updated Deployment default/temporaltest-prometheus-pushgateway: defaulted unspecified resources for containers [pushgateway] (see http://g.co/gke/autopilot-defaults)
W0912 12:39:11.941656   56930 warnings.go:70] autopilot-default-resources-mutator:Autopilot updated Deployment default/temporaltest-web: defaulted unspecified resources for containers [temporal-web] (see http://g.co/gke/autopilot-defaults)
W0912 12:39:11.949917   56930 warnings.go:70] autopilot-default-resources-mutator:Autopilot updated Deployment default/temporaltest-matching: defaulted unspecified resources for containers [check-cassandra-service, check-cassandra, check-cassandra-temporal-schema, check-cassandra-visibility-schema, check-elasticsearch-index, temporal-matching] (see http://g.co/gke/autopilot-defaults)
W0912 12:39:11.954034   56930 warnings.go:70] autopilot-default-resources-mutator:Autopilot updated Deployment default/temporaltest-history: defaulted unspecified resources for containers [check-cassandra-service, check-cassandra, check-cassandra-temporal-schema, check-cassandra-visibility-schema, check-elasticsearch-index, temporal-history] (see http://g.co/gke/autopilot-defaults)
W0912 12:39:11.980734   56930 warnings.go:70] autopilot-default-resources-mutator:Autopilot updated Deployment default/temporaltest-worker: defaulted unspecified resources for containers [check-cassandra-service, check-cassandra, check-cassandra-temporal-schema, check-cassandra-visibility-schema, check-elasticsearch-index, temporal-worker] (see http://g.co/gke/autopilot-defaults)
W0912 12:39:11.980904   56930 warnings.go:70] autopilot-default-resources-mutator:Autopilot updated Deployment default/temporaltest-frontend: defaulted unspecified resources for containers [check-cassandra-service, check-cassandra, check-cassandra-temporal-schema, check-cassandra-visibility-schema, check-elasticsearch-index, temporal-frontend] (see http://g.co/gke/autopilot-defaults)
W0912 12:39:12.269257   56930 warnings.go:70] autopilot-default-resources-mutator:Autopilot updated StatefulSet default/temporaltest-alertmanager: defaulted unspecified resources for containers [alertmanager] (see http://g.co/gke/autopilot-defaults)
W0912 12:39:12.269257   56930 warnings.go:70] autopilot-default-resources-mutator:Autopilot updated StatefulSet default/elasticsearch-master: defaulted unspecified resources for containers [configure-sysctl] (see http://g.co/gke/autopilot-defaults), and adjusted resources to meet requirements for containers [elasticsearch] (see http://g.co/gke/autopilot-resources)
W0912 12:39:12.281192   56930 warnings.go:70] autopilot-default-resources-mutator:Autopilot updated StatefulSet default/temporaltest-cassandra: defaulted unspecified resources for containers [temporaltest-cassandra] (see http://g.co/gke/autopilot-defaults)
Error: INSTALLATION FAILED: 2 errors occurred:
    * admission webhook "warden-validating.common-webhooks.networking.gke.io" denied the request: GKE Warden rejected the request because it violates one or more constraints.
Violations details: {"[denied by autogke-disallow-hostnamespaces]":["enabling hostNetwork is not allowed in Autopilot.","enabling hostPID is not allowed in Autopilot."],"[denied by autogke-no-host-port]":["container node-exporter specifies host ports [9100], which are disallowed in Autopilot."],"[denied by autogke-no-write-mode-hostpath]":["hostPath volume proc used in container node-exporter uses path /proc which is not allowed in Autopilot. Allowed path prefixes for hostPath volumes are: [/var/log/].","hostPath volume sys used in container node-exporter uses path /sys which is not allowed in Autopilot. Allowed path prefixes for hostPath volumes are: [/var/log/].","hostPath volume root used in container node-exporter uses path / which is not allowed in Autopilot. Allowed path prefixes for hostPath volumes are: [/var/log/]."]}
Requested by user: 'luka@******.com', groups: 'system:authenticated'.
    * admission webhook "warden-validating.common-webhooks.networking.gke.io" denied the request: GKE Warden rejected the request because it violates one or more constraints.
Violations details: {"[denied by autogke-disallow-privilege]":["container configure-sysctl is privileged; not allowed in Autopilot"]}
Requested by user: 'luka@******.com', groups: 'system:authenticated'.

[robholland]

These are related to prometheus, not Temporal. We will soon be removing Prometheus from our helm charts, but either way, this is not something we can fix.