search

free / sql_exporter

Database agnostic SQL exporter for Prometheus
MIT License
534 stars 171 forks source link

Getting: 194513 promhttp.go:38] Error gathering metrics: [from Gatherer #1]...Execution canceled by operator #68

Open bentzimortaboola opened 4 years ago

bentzimortaboola commented 4 years ago

Hi

I added support to Vertica database. On some clusters I get the following errors:

systemctl status vertica_exporter.service -l ● vertica_exporter.service - prometheus vertica exporter Loaded: loaded (/etc/systemd/system/vertica_exporter.service; enabled; vendor preset: disabled) Active: active (running) since Thu 2020-05-21 08:36:34 UTC; 1h 25min ago Main PID: 194513 (sql_exporter) CGroup: /system.slice/vertica_exporter.service └─194513 /etc/prometheus/sql_exporter/sql_exporter --config.file=/etc/prometheus/sql_exporter/vertica/config/vertica_sql_exporter.yml

May 21 09:59:02 vrtc003.taboolasyndication.com sql_exporter[194513]: I0521 09:59:02.216867 194513 promhttp.go:38] Error gathering metrics: [from Gatherer #1] [, collector="vertica_1_hour", query="vertica_global_status_top_ros_containers_tables_per_node"] Error: [57014] Execution canceled by operator

sql_exporter[194513]: * [from Gatherer #1] [, collector="vertica_standard_1_minute", query="vertica_global_status_catalog_locks"] context deadline exceeded

Tried to enable log but it prints same info. how do i set log in debug mode?

Thanks.

burningalchemist commented 4 years ago

@bentzimortaboola The error (Execution canceled by operator) seems to be rather specific to Vertica. I'm not sure if you had a look here already. In this case context deadline exceeded likely means that SQL-exporter hasn't received any response within a given timeout. You can try to increase that timeout via configuration or optimize your query to see if it helps.

With regards to the debug logs, it seems there is no such an option as debug messages have to be placed in the code beforehand. Since you've added Vertica driver, you can check that there are not many debug log lines available in the code. Alternatively, you can try to use /debug/pprof or put debug messages yourself for further digging.

Unfortunately, it's not that easy to help to address this issue, since there's no easy way to spin up a free Vertica cluster.

bentzimortaboola commented 4 years ago

Yes I am aware of the error.

  1. How do I control the query timeout? (want to validate it's the root cause)
  2. Vertica is free till 1TB

Thanks

burningalchemist commented 4 years ago

@bentzimortaboola You can try to increase scrape_timeout in the configuration file. Another approach would be to enable debug log levels provided by the Vertica driver itself (if you use the official one).