ericarnoldy
commented
3 years ago I'm having the same issue with AWS Timestream. I've got Lambda sending data to both Influx and Timestream. The Influx queries work great over say 30 days, 4 series of data. AWS Timestream returns only a partial set of one series for the same.
I thought there was a limit on datapoints, so tried putting into bins using bin(time, 1h) and adding a group by bin(time, 1h), which did put them into bin's but didn't see to help with the amount of queries. Good find on 20,000 row limit @stefandroid1 I was thinking may be Apache was in the way, so tried going direct with the same results.
In the logs, whenever I see the message:
grafana-server[985403]: t=2021-03-25T18:35:31+0000 lvl=info msg="running continue query" logger=plugins.backend pluginId=grafana-timestream-datasource
The query won't complete.
I think Pagination is the root cause - from AWS Timestream:
query is a paginated operation. Multiple API calls may be issued in order to retrieve the entire data set of results. You can disable pagination by providing the --no-paginate argument. When using --output text and the --query argument on a paginated response, the --query argument must extract data from the results of the following query expressions: Rows
How can we get the timestream-datasource to either be compatible with pagination, or use the SDK equivalent of no-paginate
ryantxu
stefandroid1
I run the following query in Grafana.
select time, measure_value::bigint as GreasepressureMit from $database.$table where measure_name = 'PGMit' order by time ;
This query was taking a long time to execute >1min , which is due to the large number of rows. (> 1000000 rows) However when the query completed only 7-8k records were returned. Hence the values on the graph were less and I could not view previous data.
Further on I went to troubleshoot the issue to determine the maximum number of rows returned by the query in Grafana. For this I tried to use the LIMIT clause. Example: select time, measure_value::bigint as GreasepressureMit from $database.$table where measure_name = 'PGMit' order by time LIMIT 15000;
I tried limits of 10000, 15000, 17000 and these worked well. But when the query was crossing a result of 20000 rows, this query returned only a few rows. This behavior seems like there is some batch threshold after which all the older records are not returned and only latest result in the batch are returned.
I tried multiple variations of the query but I could find only a few datapoints being processed. The issue seems similar to : https://github.com/grafana/timestream-datasource/issues/14