Non Kubernetes based metrics

[rhysxevans]

Hi,

I am not sure this is possible, however we have prometheus exporting cloud based metrics that we would like to import into stackdriver. We are unable to use stackdriver directly as it doesn't support the AWS cloud service metrics we need.

Is this scenario possible with this tool ? At present I am getting in my testing (I am testing from a GCP instance)

caller=series_cache.go:361 component="Prometheus reader" msg="unknown resource" labels="{instance=\"<instance>\", job=\"cloud_metrics\", role=\"stackdriver_exporter\"}" discovered_labels="{__address__=\"10.10.1.5:9255\", __metrics_path__=\"/metrics\", __scheme__=\"http\", _kubernetes_cluster_name=\"aws\", _kubernetes_location=\"us-east-2\", _stackdriver_project_id=\"ntest-network-mgmt-002\", instance=\"<instance>\\", job=\"cloud_metrics\", role=\"stackdriver_exporter\"}"

Command I am running

./stackdriver-sidecar --stackdriver.project-id=<project_id>--prometheus.api-address=http://127.0.0.1:9090/ --prometheus.wal-directory="/prometheus/wal" --web.listen-address="0.0.0.0:9091" --stackdriver.kubernetes.location=us-east-2 --stackdriver.kubernetes.cluster-name=aws --stackdriver.metrics-prefix=aws --log.level=debug --log.format=logfmt --web.enable-statusz

Any help is appreciated

Thanks

[tkporter]

For anyone trying this in the future, it's possible!

Stackdriver has "Resource Types" that are implied/derived from prometheus metric labels by stackdriver-prometheus-sidecar. All the Resource Types and the expected labels are found in retrieval/resource_map.go. A metric must have label names that match the keys of the LabelMap of a resource. The log you shared is given when a metric is found by the sidecar that does not have labels that match any of the resource types.

As an example, let's say that I want to have my metric be a "Generic Task". Looking at the values in resource_map.go here, my metric must have the labels:

_stackdriver_project_id (which is the value of ProjectIDLabel in resource_map.go) _generic_location (which is the value of GenericLocationLabel in resource_map.go) _generic_namespace (which is the value of GenericNamespaceLabel in resource_map.go) job instance

By default, job and instance were added by the prometheus server for me, and _stackdriver_project_id was added by the sidecar for me (I saw this from looking at the log that's similar to the one you shared). So I just needed to add _generic_location and _generic_namespace, which I did by modifying my prometheus server config. In my case, I was just using a static config for scraping:

scrape_configs:
- job_name: my-job
  scheme: https
  metrics_path: /foo/bar
  static_configs:
    - targets:
      - foo.bar.com:443
      # --vv-- this is what matters right below
      labels:
        _generic_location: us-west1-a
        _generic_namespace: foobar

Then it worked!

Another cool thing is if you go to /service-discovery on the web view of the prometheus server, you can see all the labels for a specific job in the scrape_config.

Hope this helps!