cloud_telemetry vs logging_service and monitoring_service in google_container_cluster resource

[socketbox]

Within the context of a google_container_cluster, Is there any difference between a cloud_telemetry block like so:

cloud_telemetry {
    type = "ENABLED"
}

and these two arguments:

  logging_service         = "logging.googleapis.com/kubernetes"
  monitoring_service    = "monitoring.googleapis.com/kubernetes"

?

[WillBeebe]

Same question! Does the telemetry flag expose OT metrics or "prometheus" metrics? I'm not sure what the combination of below gets me but I think I want all of it?

# LOGGING
logging_service = "logging.googleapis.com/kubernetes"
logging_config {
  enable_components = ["SYSTEM_COMPONENTS", "WORKLOADS"]
}

# MONITORING
monitoring_service = "monitoring.googleapis.com/kubernetes"
# The GKE components exposing logs.
monitoring_config {
  enable_components = ["SYSTEM_COMPONENTS", "WORKLOADS"]
}

cluster_telemetry {
  type = "ENABLED"
}

[WillBeebe]

Well at least there's this

╷
│ Error: Conflicting configuration arguments
│ 
│   with google_container_cluster.cluster,
│   on main.tf line 68, in resource "google_container_cluster" "cluster":
│   68:   logging_service = "logging.googleapis.com/kubernetes"
│ 
│ "logging_service": conflicts with cluster_telemetry
╵
╷
│ Error: Conflicting configuration arguments
│ 
│   with google_container_cluster.cluster,
│   on main.tf line 74, in resource "google_container_cluster" "cluster":
│   74:   monitoring_service = "monitoring.googleapis.com/kubernetes"
│ 
│ "monitoring_service": conflicts with cluster_telemetry

So maybe this is the intended use.

# turn it on
cluster_telemetry {
  type = "ENABLED"
}

# configure 

logging_config {
  enable_components = ["SYSTEM_COMPONENTS", "WORKLOADS"]
}
monitoring_config {
  enable_components = ["SYSTEM_COMPONENTS", "WORKLOADS"]
}

[WillBeebe]

Error: googleapi: Error 400: Cannot specify logging_config or monitoring_config together with cluster_telemetry., badRequest

[WillBeebe]

logging_service = "logging.googleapis.com/kubernetes"
logging_config {
  enable_components = ["SYSTEM_COMPONENTS", "WORKLOADS"]
}

monitoring_service = "monitoring.googleapis.com/kubernetes"
monitoring_config {
  enable_components = ["SYSTEM_COMPONENTS", "WORKLOADS"]
}

 Error: googleapi: Error 400: Cannot specify logging_config or monitoring_config together with logging_service or monitoring_service., badRequest

[WillBeebe]

Finally found something that works:

logging_config {
  enable_components = ["SYSTEM_COMPONENTS", "WORKLOADS"]
}
monitoring_config {
  enable_components = ["SYSTEM_COMPONENTS", "WORKLOADS"]
}

[alamothe]

What a mess.

[olliefr]

I stumbled upon this GitHub issue on my deep dive into google_container_cluster arguments.

I think I have the answer to the original question. I believe the Terraform resource documentation is not clear on this.

TL&DR

• cluster_telemetry block is "obsolete", don't use it.
• logging_service and monitoring_service arguments are "obsolete", don't use them.
• Use logging_config and monitoring_config 🙃

Details

GKE logging and monitoring system went through a few iterations and Terraform docs pages are not doing a good job explaining it on this occasion, unfortunately.

• The cluster_telemetry block stands for the REST API parameter clusterTelemetry. That parameter hasn't made it from v1beta into v1 API version. So we can just forget about it.
• The logging_service and monitoring_service arguments correspond to loggingService and monitoringService REST API parameters. They did make it into v1 API but they originally referred to Legacy Logging and Legacy Monitoring features. The two links lead to deprecated part of GKE docs for those two. So we can forget about those two as well.
• There was an interesting discussion in another GitHub issue on how the old logging_service and monitoring_service arguments are not compatible with the latest logging_config and monitoring_config arguments. This is enforced on the API level. There's also Deprecated Configurations Parameters section on 'Cloud Operations for GKE' docs page which corroborates the story if you compare the CLI flags between the old and new.
• Finally, if you click 'Create a Kubernetes cluster' in Cloud Console, there is an option to see the "Equivalent REST request" for the values that you choose in the UI. There's only loggingConfig and monitoringConfig and not a trace of the other parameters 🙃

Note Side note – terraform-google-kubernetes-engine module from terraform-google-modules shows that one could use logging_service and monitoring_service still but they don't provide advanced configuration options that logging_config and monitoring_config do, so I don't see the point.

Conclusion

✅ Use: logging_config and monitoring_config. This is the latest. ❌ Avoid: cluster_telemetry, logging_service, monitoring_service. These are either obsolete or less flexible.