search

sepich / thanos-kit

Aftermarket Thanos block tools and utilities. Convert Mimir blocks to Thanos.
16 stars 2 forks source link

Uploaded data not queryable from Thanos query frontend #2

Open makeittotop opened 2 years ago

makeittotop commented 2 years ago

Hi,

I've followed along the README, and successfully managed to upload some test data into a Thanos stack via the thanos-kit import command. The command does indicate that the data has been successfully uploaded into the remote store. 

bash-4.4$ ./thanos-kit import --objstore.config-file=thanos.yaml --data-dir=/var/thanos/store/ --label=tenant_id="57236109-c0e3-4b53-8700-30232dbb073d" --input-file=prom-fake-metrics.txt 
level=info ts=2022-03-09T23:44:12.229943189Z caller=import.go:84 msg="cleanup cache dir"
level=info ts=2022-03-09T23:44:12.230056999Z caller=import.go:41 msg="started importing input data."
level=info ts=2022-03-09T23:44:12.234032216Z caller=import.go:78 msg="no more input data, committing appenders and flushing block(s)"
level=info ts=2022-03-09T23:44:12.234088423Z caller=writer.go:121 msg=flushing series_count=2 mint=1970-01-20T01:27:25.2Z maxt=1970-01-20T01:27:28.741Z
level=info ts=2022-03-09T23:44:12.251629736Z caller=compact.go:495 msg="write block" mint=1646845200 maxt=1646848741 ulid=01FXRH35EAM62FH7H2QY887FDT duration=17.480839ms
level=info ts=2022-03-09T23:44:12.25342421Z caller=import.go:87 msg="blocks flushed" ids=[01FXRH35EAM62FH7H2QY887FDT]
level=info ts=2022-03-09T23:44:12.253658362Z caller=factory.go:46 msg="loading bucket configuration"
level=info ts=2022-03-09T23:44:12.759972394Z caller=import.go:72 msg="uploaded block" id=01FXRH35EAM62FH7H2QY887FDT duration=505.960231ms
level=info ts=2022-03-09T23:44:12.760010495Z caller=import.go:84 msg="cleanup cache dir"

I see the related data in the bucket dir. created as well. 

# mc ls hgst/observatorium-spi-testbox-test-01/01FXRH35EAM62FH7H2QY887FDT
[2022-03-09 23:44:12 UTC]   476B STANDARD index
[2022-03-09 23:44:12 UTC]   421B STANDARD meta.json
[2022-03-10 15:14:44 UTC]     0B chunks/

However, I'm still not able ot query the data from the thanos query frontend, even after a restart.

What am I missing, can you please help?

knightsg commented 2 months ago

I know this issue is really old but I recently resolved this myself so figured I would add my experience for others who may come across this. Like @makeittotop, my blocks were being imported with timestamps in the 1970's and while the import appeared to be working, it would only take maybe 20s total instead of the ~2hrs mentioned in the README. I was also unable to query the import metrics via the thanos querier UI after they were "imported".

I suspected the issue was the timeframe and eventually discovered I was correct, because the metrics I had in my source prometheus textfile had the timestamp in seconds, not ms which is also in the README but which I'd overlooked. I fixed that and the import time increased to an hour or so. Afterwards, I was able to query the imported metrics without issues.