search

Stackdriver / stackdriver-prometheus-sidecar

A sidecar for the Prometheus server that can send metrics to Stackdriver.
https://cloud.google.com/monitoring/kubernetes-engine/prometheus
Apache License 2.0
121 stars 43 forks source link

Unable to use filter with sidecar #123

Closed pranjal-harness closed 5 years ago

pranjal-harness commented 5 years ago

I have prometheus deployment and while trying to run stackdriver-promethues-sidecar. This is the error i am seeing in the container logs labels: {…}
logName: "projects/qa-setup/logs/sidecar"
receiveTimestamp: "2019-05-11T09:53:05.755076285Z"
resource: {…}
severity: "ERROR"
textPayload: "level=error ts=2019-05-11T09:52:30.840948761Z caller=main.go:291 msg="Error parsing filters" err="invalid filter \"{name~io_harness_custom_metric_learning_engine_clustering_task_queued_time_in_seconds}\"" "

Here is the snippet of the yaml file containing the sidecar related details. spec: containers:

jkohen commented 5 years ago

@pranjal-harness I believe you meant =~ instead of just ~. Can you try writing your filter as {__name__=~io_harness_custom_metric_learning_engine_clustering_task_queued_time_in_seconds} ?

Let me know whether that helped.

jkohen commented 5 years ago

@pranjal-harness I'm closing this because I haven't heard back from you. Please open another issue if you need more help.

raghuAtWings commented 5 years ago

@jkohen please see the output below

command

./stackdriver-prometheus-sidecar --stackdriver.project-id=qa-setup --prometheus.wal-directory=/prometheus/wal --prometheus.api-address=http://prometheus-service:8080 --stackdriver.kubernetes.location=us-west1-a --stackdriver.kubernetes.c luster-name=qa-setup --stackdriver.generic.location=harness --stackdriver.generic.namespace=harness --filter={__name__=~io_harness_custom_metric_learning_engine_clustering_task_queued_time_in_seconds}

OutPut

level=info ts=2019-06-03T23:28:49.0846418Z caller=main.go:256 msg="Starting Stackdriver Prometheus sidecar" version="(version=0.4.0, branch=master, revision=3c176b3f5c58e85645e598de5b82f95dca814497)" level=info ts=2019-06-03T23:28:49.0847477Z caller=main.go:257 build_context="(go=go1.12, user=kbuilder@kokoro-gcp-ubuntu-prod-283836442, date=20190325-14:24:29)" level=info ts=2019-06-03T23:28:49.0847949Z caller=main.go:258 host_details="(Linux 4.9.87-linuxkit-aufs #1 SMP Wed Mar 14 15:12:16 UTC 2018 x86_64 e3f138c121e2 (none))" level=info ts=2019-06-03T23:28:49.0849779Z caller=main.go:259 fd_limits="(soft=1048576, hard=1048576)" level=error ts=2019-06-03T23:28:49.0870045Z caller=main.go:291 msg="Error parsing filters" err="invalid filter \"{__name__=~io_harness_custom_metric_learning_engine_clustering_task_queued_time_in_seconds}\""

As you can see the issue still persists. Can you please reopen the ticket

StevenYCChou commented 5 years ago

Hi @raghuAtWings, thanks for getting back to us with the command and output. Can you follow the syntax of Prometheus instant vector selectors? With the snippets you provided, I believe the filter parameter will be

--filter='{__name__=~"io_harness_custom_metric_learning_engine_clustering_task_queued_time_in_seconds"}'

Please notice the single-quote for the value of filter, and the double-quote for the value of the __name__.

I will reopen this thread and keep track of this thread. Please let us know again if the issue still persists.

raghuAtWings commented 5 years ago

i got it to work using the latest release version and correct syntax. Thanks for your help

StevenYCChou commented 5 years ago

Glad to help @raghuAtWings !

Two feedback I'd like to provide before I close this issue:

  1. Try new parameter --include as a drop-in solution for --filter.

--filter is more flexible and powerful than --include, and --filter is introduced since v0.4.2. The details of --include parameter is documented in https://github.com/Stackdriver/stackdriver-prometheus-sidecar#filters.

  1. Use = instead of =~ If your goal is to only send metric with exact metric name. =~ does regex-match to the provided string.