Open vjoshi-hub opened 4 years ago
Can you share your module config and terraform version
? Thanks!
gke.txt Terraform v0.12.13
I have upgraded my provider versions - Terraform v0.12.13
I am creating additional nodepool with sandbox_enabled = true and image_type ="COS_CONTAINERD". It creates cluster and nodepool successfully. However, nodepool shows correct image type but GKE sandbox disabled.
You actually have to set the sandbox_enabled
variable to true
on the module itself, not for the individual node pool. Like this example. Can you change that and confirm it works?
I moved sandbox_enabled = true in module. I have image_type = "COS_CONTAINERD" in node pool configuration. However, I got error as below - Error: googleapi: Error 400: Gvisor sandbox type is only available on the COS_CONTAINERD image family., badRequest
on .terraform/modules/gke_cluster_hmt_prod/terraform-google-modules-terraform-google-kubernetes-engine-f99ddf4/modules/beta-private-cluster/cluster.tf line 22, in resource "google_container_cluster" "primary": 22: resource "google_container_cluster" "primary" {
Also I am using terraform-google-modules/kubernetes-engine/google//modules/beta-private-cluster module. Example mentioned above is using beta-public-cluster.
Can you provide the output of terraform plan
?
There shouldn't be a difference between private and public clusters when it comes to sandboxing.
Thanks, I think the issue is related to the embedded default node pool which we should probably remove entirely. We're going to have to do a bit more digging on this unfortunately.
@morgante I have been trying to use this with the safer-cluster
module. It appears the issue is due to the fact that this setting is meant to be enabled on a per node-pool
setting while in the module it is set at the cluster level and applied to all node-pools. In the sandbox docs it states that there must be at least one node pool with the sandbox disabled.
https://cloud.google.com/kubernetes-engine/docs/concepts/sandbox-pods#limitations-nodepool
Thanks for digging into it. It looks like we will indeed need to add a per-node-pool setting for sandboxing since we can't always rely on the default node pool.
The GKE Sandbox notes say that you need another nodepool without sandboxing to be able to use it, as the default node pool would be the first up I think this is a constraint from Google.
"When using GKE Sandbox, your cluster must have at least two node pools. You must always have at least one node pool where GKE Sandbox is disabled. This node pool must contain at least one node, even if all your workloads are sandboxed."
https://cloud.google.com/kubernetes-engine/docs/concepts/sandbox-pods
I am trying to create a cluster and nodepool with image_type as COS_CONTAINERD and node version as 1.14.8-gke.17 and sandbox_enabled as true. I am using submodule private-beta-cluster. The cluster gets created with no errors but in the console nodepool configuration shows Sandbox with gVisor as disabled. I am not sure if I am missing something.