search

thanos-io / thanos

Highly available Prometheus setup with long term storage capabilities. A CNCF Incubating project.
https://thanos.io
Apache License 2.0
13.12k stars 2.1k forks source link

Thanos store preload series: read series range: read range: context canceled #5371

Open davidy718 opened 2 years ago

davidy718 commented 2 years ago

Thanos, Prometheus and Golang version used: Prometheus: 2.23.1 Thanos: 0.26.0

Object Storage Provider: aws s3 What happened: when querying big amounts of data from thanos store I get an error (if its small amounts it works) What you expected to happen: to receive the data How to reproduce it (as minimally and precisely as possible): We have Prometheus with thanos sidecar uploading to aws s3 thanos store connected to s3 thanos query with thanos store setup Full logs to relevant components:

Logs

``` level=debug ts=2022-05-18T14:57:21.407420227Z caller=bucket.go:1115 msg="stats query processed" request="min_time:1651799336000 max_time:1651877044000 matchers: matchers: matchers: aggregates:COUNT aggregates:SUM partial_response_disabled:true " stats="&{blocksQueried:0 postingsTouched:35 PostingsTouchedSizeSum:1KiB312B postingsToFetch:0 postingsFetched:0 PostingsFetchedSizeSum:0B postingsFetchCount:0 PostingsFetchDurationSum:0s cachedPostingsCompressions:0 cachedPostingsCompressionErrors:0 CachedPostingsOriginalSizeSum:0B CachedPostingsCompressedSizeSum:0B CachedPostingsCompressionTimeSum:0s cachedPostingsDecompressions:35 cachedPostingsDecompressionErrors:0 CachedPostingsDecompressionTimeSum:28.522µs seriesTouched:10 SeriesTouchedSizeSum:586B seriesFetched:0 SeriesFetchedSizeSum:0B seriesFetchCount:0 SeriesFetchDurationSum:0s chunksTouched:40 ChunksTouchedSizeSum:2KiB76B chunksFetched:40 ChunksFetchedSizeSum:157KiB1011B chunksFetchCount:10 ChunksFetchDurationSum:7m42.865372171s GetAllDuration:0s mergedSeriesCount:0 mergedChunksCount:0 MergeDuration:0s}" err="rpc error: code = Aborted desc = fetch series for block 01G2DSKZP3JC7VWGYZR8X2NA2D: load chunks: get range reader: Get \"https://BUCKET_NAME.s3.dualstack.eu-west-1.amazonaws.com/01G2DSKZP3JC7VWGYZR8X2NA2D/chunks/000001\": context canceled" level=debug ts=2022-05-18T14:57:21.425815469Z caller=bucket.go:1115 msg="stats query processed" request="min_time:1651673548000 max_time:1651707714000 matchers: matchers: matchers: aggregates:COUNT aggregates:SUM partial_response_disabled:true " stats="&{blocksQueried:0 postingsTouched:36 PostingsTouchedSizeSum:13KiB362B postingsToFetch:0 postingsFetched:0 PostingsFetchedSizeSum:0B postingsFetchCount:0 PostingsFetchDurationSum:0s cachedPostingsCompressions:0 cachedPostingsCompressionErrors:0 CachedPostingsOriginalSizeSum:0B CachedPostingsCompressedSizeSum:0B CachedPostingsCompressionTimeSum:0s cachedPostingsDecompressions:36 cachedPostingsDecompressionErrors:0 CachedPostingsDecompressionTimeSum:129.819µs seriesTouched:174 SeriesTouchedSizeSum:10KiB1008B seriesFetched:0 SeriesFetchedSizeSum:0B seriesFetchCount:0 SeriesFetchDurationSum:0s chunksTouched:696 ChunksTouchedSizeSum:112KiB505B chunksFetched:696 ChunksFetchedSizeSum:30MiB412KiB253B chunksFetchCount:117 ChunksFetchDurationSum:26m2.320674944s GetAllDuration:0s mergedSeriesCount:0 mergedChunksCount:0 MergeDuration:0s}" err="rpc error: code = Aborted desc = fetch series for block 01G28MTFCFKTY7NX9JYMHPFRQV: preload series: read series range: read range: context canceled" level=debug ts=2022-05-18T14:57:21.437563301Z caller=bucket.go:1115 msg="stats query processed" request="min_time:1652142834000 max_time:1652225380000 matchers: matchers: matchers: aggregates:COUNT aggregates:SUM partial_response_disabled:true " stats="&{blocksQueried:0 postingsTouched:87 PostingsTouchedSizeSum:26KiB21B postingsToFetch:0 postingsFetched:0 PostingsFetchedSizeSum:0B postingsFetchCount:0 PostingsFetchDurationSum:0s cachedPostingsCompressions:0 cachedPostingsCompressionErrors:0 CachedPostingsOriginalSizeSum:0B CachedPostingsCompressedSizeSum:0B CachedPostingsCompressionTimeSum:0s cachedPostingsDecompressions:87 cachedPostingsDecompressionErrors:0 CachedPostingsDecompressionTimeSum:268.839µs seriesTouched:354 SeriesTouchedSizeSum:22KiB566B seriesFetched:48 SeriesFetchedSizeSum:6MiB803KiB192B seriesFetchCount:2 SeriesFetchDurationSum:13.423399746s chunksTouched:1356 ChunksTouchedSizeSum:196KiB111B chunksFetched:1356 ChunksFetchedSizeSum:46MiB102KiB950B chunksFetchCount:271 ChunksFetchDurationSum:1h30m11.553651591s GetAllDuration:0s mergedSeriesCount:0 mergedChunksCount:0 MergeDuration:0s}" err="rpc error: code = Aborted desc = fetch series for block 01G2Q7QWY922QGY8TF5PSJWP3Z: load chunks: get range reader: Get \"https://BUCKET_NAME.s3.dualstack.eu-west-1.amazonaws.com/01G2Q7QWY922QGY8TF5PSJWP3Z/chunks/000001\": context canceled" level=debug ts=2022-05-18T14:57:21.443031385Z caller=bucket.go:1115 msg="stats query processed" request="min_time:1651712252000 max_time:1651794798000 matchers: matchers: matchers: aggregates:COUNT aggregates:SUM partial_response_disabled:true " stats="&{blocksQueried:0 postingsTouched:72 PostingsTouchedSizeSum:14KiB572B postingsToFetch:0 postingsFetched:0 PostingsFetchedSizeSum:0B postingsFetchCount:0 PostingsFetchDurationSum:0s cachedPostingsCompressions:0 cachedPostingsCompressionErrors:0 CachedPostingsOriginalSizeSum:0B CachedPostingsCompressedSizeSum:0B CachedPostingsCompressionTimeSum:0s cachedPostingsDecompressions:72 cachedPostingsDecompressionErrors:0 CachedPostingsDecompressionTimeSum:140.032µs seriesTouched:192 SeriesTouchedSizeSum:11KiB1005B seriesFetched:0 SeriesFetchedSizeSum:0B seriesFetchCount:0 SeriesFetchDurationSum:0s chunksTouched:766 ChunksTouchedSizeSum:105KiB988B chunksFetched:766 ChunksFetchedSizeSum:35MiB972KiB182B chunksFetchCount:124 ChunksFetchDurationSum:36m48.443071486s GetAllDuration:0s mergedSeriesCount:0 mergedChunksCount:0 MergeDuration:0s}" err="rpc error: code = Aborted desc = fetch series for block 01G299DN4N2Y6YFR52PX15G0E0: preload series: read series range: read range: context canceled" level=debug ts=2022-05-18T14:57:21.443217318Z caller=bucket.go:1115 msg="stats query processed" request="min_time:1652055750000 max_time:1652138296000 matchers: matchers: matchers: aggregates:COUNT aggregates:SUM partial_response_disabled:true " stats="&{blocksQueried:0 postingsTouched:117 PostingsTouchedSizeSum:28KiB425B postingsToFetch:0 postingsFetched:0 PostingsFetchedSizeSum:0B postingsFetchCount:0 PostingsFetchDurationSum:0s cachedPostingsCompressions:0 cachedPostingsCompressionErrors:0 CachedPostingsOriginalSizeSum:0B CachedPostingsCompressedSizeSum:0B CachedPostingsCompressionTimeSum:0s cachedPostingsDecompressions:117 cachedPostingsDecompressionErrors:0 CachedPostingsDecompressionTimeSum:288.31µs seriesTouched:398 SeriesTouchedSizeSum:23KiB847B seriesFetched:84 SeriesFetchedSizeSum:11MiB999KiB336B seriesFetchCount:3 SeriesFetchDurationSum:49.902125157s chunksTouched:1392 ChunksTouchedSizeSum:193KiB561B chunksFetched:1392 ChunksFetchedSizeSum:51MiB920KiB777B chunksFetchCount:278 ChunksFetchDurationSum:1h16m19.520176018s GetAllDuration:0s mergedSeriesCount:0 mergedChunksCount:0 MergeDuration:0s}" err="rpc error: code = Aborted desc = fetch series for block 01G2NQNSMN55AH7JNMHBR105BM: preload series: read series range: read range: context canceled" level=debug ts=2022-05-18T14:57:21.443215129Z caller=bucket.go:1115 msg="stats query processed" request="min_time:1651881582000 max_time:1651964128000 matchers: matchers: matchers: aggregates:COUNT aggregates:SUM partial_response_disabled:true " stats="&{blocksQueried:0 postingsTouched:46 PostingsTouchedSizeSum:2KiB475B postingsToFetch:0 postingsFetched:0 PostingsFetchedSizeSum:0B postingsFetchCount:0 PostingsFetchDurationSum:0s cachedPostingsCompressions:0 cachedPostingsCompressionErrors:0 CachedPostingsOriginalSizeSum:0B CachedPostingsCompressedSizeSum:0B CachedPostingsCompressionTimeSum:0s cachedPostingsDecompressions:46 cachedPostingsDecompressionErrors:0 CachedPostingsDecompressionTimeSum:43.376µs seriesTouched:22 SeriesTouchedSizeSum:1KiB267B seriesFetched:0 SeriesFetchedSizeSum:0B seriesFetchCount:0 SeriesFetchDurationSum:0s chunksTouched:78 ChunksTouchedSizeSum:4KiB26B chunksFetched:78 ChunksFetchedSizeSum:346KiB990B chunksFetchCount:22 ChunksFetchDurationSum:12m31.729556724s GetAllDuration:0s mergedSeriesCount:0 mergedChunksCount:0 MergeDuration:0s}" err="rpc error: code = Aborted desc = fetch series for block 01G2EE735HW0D8KD52Q0A5WC4A: preload series: read series range: read range: context canceled" level=debug ts=2022-05-18T14:57:21.443226111Z caller=bucket.go:1115 msg="stats query processed" request="min_time:1651968666000 max_time:1652051212000 matchers: matchers: matchers: aggregates:COUNT aggregates:SUM partial_response_disabled:true " stats="&{blocksQueried:0 postingsTouched:68 PostingsTouchedSizeSum:12KiB914B postingsToFetch:0 postingsFetched:0 PostingsFetchedSizeSum:0B postingsFetchCount:0 PostingsFetchDurationSum:0s cachedPostingsCompressions:0 cachedPostingsCompressionErrors:0 CachedPostingsOriginalSizeSum:0B CachedPostingsCompressedSizeSum:0B CachedPostingsCompressionTimeSum:0s cachedPostingsDecompressions:68 cachedPostingsDecompressionErrors:0 CachedPostingsDecompressionTimeSum:131.343µs seriesTouched:164 SeriesTouchedSizeSum:10KiB238B seriesFetched:0 SeriesFetchedSizeSum:0B seriesFetchCount:0 SeriesFetchDurationSum:0s chunksTouched:626 ChunksTouchedSizeSum:85KiB12B chunksFetched:626 ChunksFetchedSizeSum:30MiB727KiB173B chunksFetchCount:106 ChunksFetchDurationSum:21m19.674413343s GetAllDuration:0s mergedSeriesCount:0 mergedChunksCount:0 MergeDuration:0s}" err="rpc error: code = Aborted desc = fetch series for block 01G2J9T5MD86290KVV3CDMY1H5: preload series: read series range: read range: context canceled" level=debug ts=2022-05-18T14:57:21.443228229Z caller=bucket.go:1115 msg="stats query processed" request="min_time:1652229918000 max_time:1652280981328 matchers: matchers: matchers: aggregates:COUNT aggregates:SUM partial_response_disabled:true " stats="&{blocksQueried:0 postingsTouched:108 PostingsTouchedSizeSum:49KiB888B postingsToFetch:0 postingsFetched:0 PostingsFetchedSizeSum:0B postingsFetchCount:0 PostingsFetchDurationSum:0s cachedPostingsCompressions:0 cachedPostingsCompressionErrors:0 CachedPostingsOriginalSizeSum:0B CachedPostingsCompressedSizeSum:0B CachedPostingsCompressionTimeSum:0s cachedPostingsDecompressions:108 cachedPostingsDecompressionErrors:0 CachedPostingsDecompressionTimeSum:429.174µs seriesTouched:741 SeriesTouchedSizeSum:51KiB860B seriesFetched:24 SeriesFetchedSizeSum:3MiB421KiB928B seriesFetchCount:1 SeriesFetchDurationSum:23.018540757s chunksTouched:2226 ChunksTouchedSizeSum:375KiB1020B chunksFetched:2226 ChunksFetchedSizeSum:51MiB687KiB170B chunksFetchCount:659 ChunksFetchDurationSum:4h50m22.359801907s GetAllDuration:0s mergedSeriesCount:0 mergedChunksCount:0 MergeDuration:0s}" err="rpc error: code = Aborted desc = fetch series for block 01G2T10AMS2PZCW5QHPF09JNX8: load chunks: get range reader: Get \"https://BUCKET_NAME.s3.dualstack.eu-west-1.amazonaws.com/01G2T10AMS2PZCW5QHPF09JNX8/chunks/000001\": context canceled" ```

Environment: K8S Using bitnami helm chart version: 10.4.0 Docker image: bitnami/thanos:0.26.0-scratch-r0

stale[bot] commented 2 years ago

Hello 👋 Looks like there was no activity on this issue for the last two months. Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗 If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.