Long Execution Time of Flux Queries in Grafana with InfluxDB 2

tonypowa commented 1 month ago

What happened?

When running Flux queries from Grafana to fetch data from InfluxDB 2, the query execution time exceeds 5 minutes. Even after optimizing the queries by removing regex parts from variable definitions and adding the duration parameter to the aggregateWindow function, the execution time remains high, with the best time achieved being 2 minutes.

What did you expect to happen?

I expected the Flux queries to execute within a reasonable time frame, ideally under 1 minute, considering the size and complexity of the data being queried.

Did this work before?

No, the queries have consistently taken a long time to execute, even before attempting the optimizations mentioned above.

How do we reproduce it?

1Use Grafana to run Flux queries against an InfluxDB 2 database.

Construct queries that involve fetching data over a significant time range or from large datasets.

Observe that the queries take longer than expected to execute, often exceeding 5 minutes.

Attempt optimizations such as removing regex parts from variable definitions and adding the duration parameter to the aggregateWindow function, but notice minimal improvements in query execution time.

Is the bug inside a dashboard panel?

Yes.

Environment (with versions)?

Grafana: 9.3.6

OS: windows

Browser: Chrome

Grafana platform?

Kubernetes

Datasource(s)?

InfluxDB2

tonypowa commented 1 month ago

This issue is a test copy of an issue in another repo. Original issue: https://github.com/grafana/grafana/issues/#88015

moxious commented 1 month ago

Elaboration:

Request for additional information

Hi @tonypowa, thank you for reaching out with the issue you're experiencing. We want to ensure your concerns are addressed and investigations can proceed effectively. To help us better understand and replicate the issue you're facing, could you please provide the following additional information:

Query examples: Could you share some sanitized examples of the Flux queries that are running slowly? This will help us understand the structure and complexity of the queries.
Dataset size: What is the approximate size of the datasets you are querying against (e.g., row count, series count)?
InfluxDB2 version and specs: What version of InfluxDB 2 are you using? Additionally, can you provide some insights on the hardware or the specs of the InfluxDB server (CPU, Memory, Disk type, etc.)?
Performance metrics: Have you been able to capture any performance metrics from InfluxDB 2 during the execution of these queries, such as CPU usage, disk I/O, or memory usage?
Grafana dashboard configuration: Is there anything specific about the dashboard panel's configuration that could be contributing to the issue? Complex transformations, multiple queries being run on a single panel, etc.
Network conditions/speed: Since you're using Kubernetes, could it be possible that network latency within your cluster might be contributing to the slow response times?
Comparison benchmark: Have you tried executing these queries directly within the InfluxDB UI or CLI? If so, what are the performance differences when compared to running the queries through Grafana?
Logs and errors: Are there any logs or error messages from either Grafana or InfluxDB that could provide some clues?
Time range queried: What time ranges are you querying? Smaller time ranges might perform significantly different from larger ones.

Please attach any relevant screenshots, logs, and query examples as they will help in diagnosing the problem. Remember to redact any sensitive information from the details you share.

Your cooperation is greatly appreciated, and we look forward to resolving this issue together!

moxious commented 1 month ago

Hello @tonypowa, thanks for the detailed report. It seems like the issue might be related to the interaction with the InfluxDB datasource within Grafarka. Based on the details provided, this issue would best fit the Plugins Platform project, as Flux queries are part of the InfluxDB plugin. The team there might be able to help with optimizing queries or addressing potential plugin performance issues.

Could you provide the exact version of the InfluxDB plugin you are using? Also, if you could share a sanitized example of a query that exhibits the long execution times, it would assist the team in identifying the problem.

moxious commented 1 month ago

Summary: The issue is an OPEN problem where Flux queries in Grafana fetching data from InfluxDB 2 take excessively long to execute, over 5 minutes, and optimizations have not reduced times to below 2 minutes. The author expected the queries to run under 1 minute and has not had success in the past with quicker executions. Reproduction involves running Flux queries in Grafana against InfluxDB 2, particularly with large datasets or over sizable time ranges.

moxious / triage