search

XenitAB / terraform-modules

Xenit Terraform modules
MIT License
33 stars 18 forks source link

Azure gather metrics for vmss #480

Open NissesSenap opened 2 years ago

NissesSenap commented 2 years ago

We have seen that even if a node is deleted from kubernetes they some time linger inside vmss. Add metrics gathering from azure vmss metrics endpoint and expose them in prometheus.

Add the needed config to: https://github.com/XenitAB/terraform-modules/blob/main/modules/kubernetes/azure-metrics/charts/azure-metrics-exporter/templates/podmonitor.yaml

NissesSenap commented 2 years ago

@CalleB3 might this be something for you?

CalleB3 commented 2 years ago

Unfortunately at the current time there isnt any supported metrics in Azure for this.

Under Microsoft.Compute/virtualmachineScaleSets at the link below, there isnt anything related to vmss and its instances.

https://docs.microsoft.com/en-us/azure/azure-monitor/essentials/metrics-supported#microsoftcomputevirtualmachinescalesets

NissesSenap commented 2 years ago

It looks like there is: VmAvailabilityMetric but it's in preview, we could probably open a support ticket and ask to be added in this preview.

Another possible solution is that we actually don't need a specific metric we just need to know the count of nodes responding with metrics. Then we can compare this data with active k8s nodes in the cluster. So it we gather any metric from vmss that is related to k8s we should be okay.

Do you agree @phillebaba?