search

m-lab / etl

M-Lab ingestion pipeline
Apache License 2.0
22 stars 7 forks source link

GCE discovery and/or prometheus scraping not working for private network #769

Open gfr10598 opened 4 years ago

gfr10598 commented 4 years ago

The universal parser uses a private network, so it can make requests to gardener. This breaks the prometheus scraping. Probably interferes with GCE discover, and may also make the instances inaccessible.

We can get the metrics through the app-engine load balancer, but that is not instance specific, so likely not relevant.

stephen-soltesz commented 4 years ago

Discussed in prometheus-sync -- the main problem here is that the default network where gcp-service-discovery runs today cannot reach the private vpc network. While there is currently a minimal prometheus config running in the per-datatype data-processing-cluster there is not yet a prometheus instance in the new data-processing cluster. The insight is that that instance should run the gcp-service-discovery daemon as well (possibly with a modification to gcp-sd to filter services) so that it can reach the AEFlex instances over the private network. This is "working with" the architecture rather than trying to get prometheus-federaiton cluster to connect to the private vpc network.