search

terraform-google-modules / terraform-google-kubernetes-engine

Configures opinionated GKE clusters
https://registry.terraform.io/modules/terraform-google-modules/kubernetes-engine/google
Apache License 2.0
1.13k stars 1.14k forks source link

GKE Sandbox configuration #363

Open vjoshi-hub opened 4 years ago

vjoshi-hub commented 4 years ago

I am trying to create a cluster and nodepool with image_type as COS_CONTAINERD and node version as 1.14.8-gke.17 and sandbox_enabled as true. I am using submodule private-beta-cluster. The cluster gets created with no errors but in the console nodepool configuration shows Sandbox with gVisor as disabled. I am not sure if I am missing something.

morgante commented 4 years ago

Can you share your module config and terraform version? Thanks!

vjoshi-hub commented 4 years ago

gke.txt Terraform v0.12.13

vjoshi-hub commented 4 years ago

I have upgraded my provider versions - Terraform v0.12.13

vjoshi-hub commented 4 years ago

I am creating additional nodepool with sandbox_enabled = true and image_type ="COS_CONTAINERD". It creates cluster and nodepool successfully. However, nodepool shows correct image type but GKE sandbox disabled.

morgante commented 4 years ago

You actually have to set the sandbox_enabled variable to true on the module itself, not for the individual node pool. Like this example. Can you change that and confirm it works?

vjoshi-hub commented 4 years ago

I moved sandbox_enabled = true in module. I have image_type = "COS_CONTAINERD" in node pool configuration. However, I got error as below - Error: googleapi: Error 400: Gvisor sandbox type is only available on the COS_CONTAINERD image family., badRequest

on .terraform/modules/gke_cluster_hmt_prod/terraform-google-modules-terraform-google-kubernetes-engine-f99ddf4/modules/beta-private-cluster/cluster.tf line 22, in resource "google_container_cluster" "primary": 22: resource "google_container_cluster" "primary" {

Also I am using terraform-google-modules/kubernetes-engine/google//modules/beta-private-cluster module. Example mentioned above is using beta-public-cluster.

morgante commented 4 years ago

Can you provide the output of terraform plan?

There shouldn't be a difference between private and public clusters when it comes to sandboxing.

vjoshi-hub commented 4 years ago

terraform-plan.txt

morgante commented 4 years ago

Thanks, I think the issue is related to the embedded default node pool which we should probably remove entirely. We're going to have to do a bit more digging on this unfortunately.

robertb724 commented 3 years ago

@morgante I have been trying to use this with the safer-cluster module. It appears the issue is due to the fact that this setting is meant to be enabled on a per node-pool setting while in the module it is set at the cluster level and applied to all node-pools. In the sandbox docs it states that there must be at least one node pool with the sandbox disabled.

https://cloud.google.com/kubernetes-engine/docs/concepts/sandbox-pods#limitations-nodepool

morgante commented 3 years ago

Thanks for digging into it. It looks like we will indeed need to add a per-node-pool setting for sandboxing since we can't always rely on the default node pool.

iamasmith commented 1 year ago

The GKE Sandbox notes say that you need another nodepool without sandboxing to be able to use it, as the default node pool would be the first up I think this is a constraint from Google.

"When using GKE Sandbox, your cluster must have at least two node pools. You must always have at least one node pool where GKE Sandbox is disabled. This node pool must contain at least one node, even if all your workloads are sandboxed."

https://cloud.google.com/kubernetes-engine/docs/concepts/sandbox-pods