search

concourse / hush-house

Concourse k8s-based environment
https://hush-house.pivotal.io
30 stars 23 forks source link

upgrade stackdriver #103

Closed cirocosta closed 4 years ago

cirocosta commented 4 years ago

Hey,

Since the upgrade to 1.15 (see https://github.com/concourse/hush-house/issues/95), we end up with no metrics coming to the (web|worker) errors dashboards:

Screen Shot 2020-01-09 at 9 20 42 AM

It turns out that when moving from Kubernetes 1.14.x to 1.15.x (GKE), we end up losing support for what's now called Legacy stackdriver (which is what collects our logs and turns them into web/worker errors in our dashboards).

Screen Shot 2020-01-09 at 9 22 22 AM

(from https://cloud.google.com/monitoring/kubernetes-engine/)

It seems like to get the new Stackdriver we need to first disable legacy, and then enable the new one:

Screen Shot 2020-01-09 at 9 31 11 AM

That's something we might / might not be able to do right from Terraform

https://github.com/concourse/hush-house/blob/a3f7662a65cc32ea0de47e31851e577564455a4a/terraform/cluster/main.tf#L17-L19

although I'm not completely sure.

Thanks!

cirocosta commented 4 years ago

aaand .. it's active!

Screen Shot 2020-01-29 at 1 18 10 PM

as we can't get the legacy even if we wanted to, there's no need to have that in terraform.