Querying data exported from Prometheus snapshot returns empty results

karim-manaouil commented 4 years ago

I want to do offline analysis of a Kubernetes cluster so I generated a Prometheus snapshot based on this article, compressed it and moved to my local machine. Initially, I used a local Prometheus instance and it worked great, I can locally access all the cluster time series and perform the queries. The day after when I woke up, I uncompressed the snapshot again and launched a new local Prometheus instance but I can no more see the time series, everything returns empty results. Based on the Prometheus server log, it looks like it runs a garbage collector on old data so I tricked the system time and the GC is no more triggered but the same problem, no results.

I decided to use VictoriaMetrics, so I uncompressed the Prometheus snapshot again and loaded it into VictoMetrics using vmctl as explained in the docs. It was loaded successfully and it shows the correct min and max UNIX timestamps. I can get all the labels correctly but querying the time series returns empty results again. I have no idea what's the reason behind this behavior. I have many compressed tarballs of Prometheus snapshots that worked fine the night before with the local Prometheus instance but now they don't work neither with Prometheus nor VictoriaMetrics.

Here is the vmctl output:

$ export SNAP=20200424T010256Z-26811953682f91a2/
$ ./vmctl prometheus --prom-snapshot=$SNAP --vm-concurrency 1 --vm-batch-size=200000 --prom-concurrency 3
Prometheus import mode
Prometheus snapshot stats:
  blocks found: 1;
  blocks skipped: 0;
  min time: 1587688778083 (2020-04-24T02:39:38+02:00);
  max time: 1587690175941 (2020-04-24T03:02:55+02:00);
  samples: 8391388;
  series: 37061.
Filter is not taken into account for series and samples numbers.
Found 1 blocks to import. Continue? [Y/n] Y
1 / 1 [------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% 0 p/s
2020/04/24 20:52:47 Import finished!
2020/04/24 20:52:47 VictoriaMetrics importer stats:
  time spent while waiting: 4.852377614s;
  time spent while importing: 3.291038472s;
  total datapoints: 8391388;
  datapoints/s: 2549769.04;
  total bytes: 159.6 MB;
  bytes/s: 48.5 MB;
  import requests: 84;
  import requests retries: 0;
2020/04/24 20:52:47 Total time: 8.155349577s

$ curl -g "http://localhost:8428/api/v1/query?query=apiserver_request_count" && echo
{"status":"success","data":{"resultType":"vector","result":[]}}

hagen1778 commented 4 years ago

Hi @karim-manaouil! Looks like import was successful and the reason why you don't see query results may be following:

VM is configured to have less retention that data you imported. But looks like this is not the case based on min/max time values
VM may cache results for requested query before data was backfilled via vmctl. Please see these docs to get more details about backfilling issues.
The search query was wrongly configured. To verify if VM actually contains imported data you may use export handler: curl -g "http://localhost:8428/api/v1/export?match[]=apiserver_request_count". If you see data returned then VM actually has that data saved.

The third options seems most likely to me. The reason is that you use instant query handler that returns only last datapoint for last 5minute. It means, if there was no data in last 5 minutes - the result will be always empty. To avoid this please specify the time param (time=<rfc3339 | unix_timestamp>) in the query: curl -g "http://localhost:8428/api/v1/query?query=apiserver_request_count&time= 1587688778083". Another option is to use query range handler, that scans data over a time range. But the most handy tool to explore data in VM is to use Grafana and its Explore feature. It provides very useful tools for querying data from Prometheus compatible datasources.

karim-manaouil commented 4 years ago

Hi @hagen1778 Thanks for the detailed reply. Honestly, I tried the export handler with the Prometheus snapshots and it didn't work again. However, I wasn't really focusing and I know I was missing something somewhere. Anyway, I completely switched to using VictoriaMetrics as my Backend storage and I use the native exportation mechanism and it works really great. I really like this tool, it's very complete, robust and very well documented. Keep it up. Thanks.

karim-manaouil commented 4 years ago

Moved to native exportation mechanism of VictoriaMetrics.

VictoriaMetrics / vmctl

Querying data exported from Prometheus snapshot returns empty results #10