Open rhysxevans opened 4 years ago
For anyone trying this in the future, it's possible!
Stackdriver has "Resource Types" that are implied/derived from prometheus metric labels by stackdriver-prometheus-sidecar. All the Resource Types and the expected labels are found in retrieval/resource_map.go. A metric must have label names that match the keys of the LabelMap
of a resource. The log you shared is given when a metric is found by the sidecar that does not have labels that match any of the resource types.
As an example, let's say that I want to have my metric be a "Generic Task". Looking at the values in resource_map.go here, my metric must have the labels:
_stackdriver_project_id
(which is the value of ProjectIDLabel
in resource_map.go)
_generic_location
(which is the value of GenericLocationLabel
in resource_map.go)
_generic_namespace
(which is the value of GenericNamespaceLabel
in resource_map.go)
job
instance
By default, job
and instance
were added by the prometheus server for me, and _stackdriver_project_id
was added by the sidecar for me (I saw this from looking at the log that's similar to the one you shared). So I just needed to add _generic_location
and _generic_namespace
, which I did by modifying my prometheus server config. In my case, I was just using a static config for scraping:
scrape_configs:
- job_name: my-job
scheme: https
metrics_path: /foo/bar
static_configs:
- targets:
- foo.bar.com:443
# --vv-- this is what matters right below
labels:
_generic_location: us-west1-a
_generic_namespace: foobar
Then it worked!
Another cool thing is if you go to /service-discovery
on the web view of the prometheus server, you can see all the labels for a specific job in the scrape_config.
Hope this helps!
Hi,
I am not sure this is possible, however we have prometheus exporting cloud based metrics that we would like to import into stackdriver. We are unable to use stackdriver directly as it doesn't support the AWS cloud service metrics we need.
Is this scenario possible with this tool ? At present I am getting in my testing (I am testing from a GCP instance)
caller=series_cache.go:361 component="Prometheus reader" msg="unknown resource" labels="{instance=\"<instance>\", job=\"cloud_metrics\", role=\"stackdriver_exporter\"}" discovered_labels="{__address__=\"10.10.1.5:9255\", __metrics_path__=\"/metrics\", __scheme__=\"http\", _kubernetes_cluster_name=\"aws\", _kubernetes_location=\"us-east-2\", _stackdriver_project_id=\"ntest-network-mgmt-002\", instance=\"<instance>\\", job=\"cloud_metrics\", role=\"stackdriver_exporter\"}"
Command I am running
./stackdriver-sidecar --stackdriver.project-id=<project_id>--prometheus.api-address=http://127.0.0.1:9090/ --prometheus.wal-directory="/prometheus/wal" --web.listen-address="0.0.0.0:9091" --stackdriver.kubernetes.location=us-east-2 --stackdriver.kubernetes.cluster-name=aws --stackdriver.metrics-prefix=aws --log.level=debug --log.format=logfmt --web.enable-statusz
Any help is appreciated
Thanks