GKE Default prometheus enable feature causing conflict

[oguzhanoyan]

GKE clusters enable Prometheus by default after March 15, 2023. But this feature has a conflict with the fields;

• https://github.com/GoogleCloudPlatform/kubeflow-distribution/blob/master/kubeflow/common/cluster/upstream/cluster.yaml#L59
• https://github.com/GoogleCloudPlatform/kubeflow-distribution/blob/master/kubeflow/common/cluster/upstream/cluster.yaml#L60

And you are getting the error;

Error 400: Cannot specify logging_config or monitoring_config together with logging_service or monitoring_service. 1

Solution; In order to fix this error, you can comment on lines 59 and 60 in https://github.com/GoogleCloudPlatform/kubeflow-distribution/blob/master/kubeflow/common/cluster/upstream/cluster.yaml#L60 .

Not sure that this line should be there because of them;

• loggingService default value is: logging.googleapis.com/kubernetes
• monitoringService default value is: monitoring.googleapis.com/kubernetes

[Linchin]

Thank you for reporting this issue, and suggesting potential solution. Enabling Prometheus by default is a new GKE feature starting with version 1.27. However, Kubeflow on GCP uses the GKE stable channel that's currently at version 1.25 (see release note, GKE channel info). We will keep an eye out for this issue, and verify the change once stable channel reaches version 1.27.