Currently the dask-scheduler of each cluter is metrics-enabled.
But, by the very nature of these external clusters they are autoscaled, and prometheus does'nt know where to scrape them.
Therefore we would need one centralized location to scrape these schedulers, which gives for now 2 options:
Option 1: Clusters-keeper could provide a /metrics endpoint that aggregate the /metrics endpoint of all the scheduler it created
Option 2: have an exporter that scrapes the external clusters
Option 3: have autoscaling instrumented + have a prometheus running on every computational clusters, then make central prometheus auto-discover them via service discovery via EC2
sanderegg
commented
8 months ago
Currently the dask-scheduler of each cluter is metrics-enabled. But, by the very nature of these external clusters they are autoscaled, and prometheus does'nt know where to scrape them.
Therefore we would need one centralized location to scrape these schedulers, which gives for now 2 options: