Can not see downsampled data

[DirkFries]

Hi @all,

I have some problems to see downsampled data.

Software involved

thanos, version 0.6.0-rc.0 (branch: HEAD, revision: 7f2200906ae112035ba4c973cf545a28f74cc9d5) build user: root@8b35e55d6107 build date: 20190712-11:25:23 go version: go1.12.5

prometheus, version 2.10.0 (branch: HEAD, revision: d20e84d0fb64aff2f62a977adc8cfb656da4e286) build user: root@a49185acd9b0 build date: 20190525-12:28:13 go version: go1.12.5

I try to downsample data on a regular base (15Minutes) with this command:

./thanos downsample --objstore.config-file=s3.yaml --http-address="0.0.0.0:10982

Additionally I do a compact run once a day

./thanos compact --data-dir /tmp/thanos-compact --objstore.config-file=s3.yaml --retention.resolution-raw=5d --retention.resolution-5m=31d --retention.resolution-1h=365d --http-address="0.0.0.0:10910" --log.level=debug

My expection is, that I can see data that are older than 5 days, but that those data are downsampled to 5 Minutes.

If I do a bucket inspect I get :

|            ULID            |        FROM         |        UNTIL        |  RANGE  | UNTIL-COMP | #SERIES |  #SAMPLES   |  #CHUNKS   | COMP-LEVEL | COMP-FAILED |                      LABELS                      | RESOLUTION |  SOURCE   |
|----------------------------|---------------------|---------------------|---------|------------|---------|-------------|------------|------------|-------------|--------------------------------------------------|------------|-----------|
| 01DFT8QA2HEJ0D56BXFC4G2JHX | 13-07-2019 02:00:00 | 15-07-2019 02:00:00 | 48h0m0s | 192h0m0s   | 10,436  | 6,004,455   | 52,132     | 4          | false       | monitor=infrastructure,region=xxx,replica=A | 5m0s       | compactor |
| 01DFZJJZQ6F2ZF4F7JMKTB83FP | 15-07-2019 02:00:00 | 17-07-2019 02:00:00 | 48h0m0s | 192h0m0s   | 10,772  | 6,039,189   | 52,783     | 4          | false       | monitor=infrastructure,region=xxx,replica=A | 5m0s       | compactor |
| 01DGFKN51Z900EJG969CB7TQ9K | 17-07-2019 02:00:00 | 19-07-2019 02:00:00 | 48h0m0s | 192h0m0s   | 11,474  | 6,057,053   | 52,874     | 4          | false       | monitor=infrastructure,region=xxx,replica=A | 5m0s       | compactor |
| 01DGFKNSMRMCM55TEV1ZW291M3 | 19-07-2019 02:00:00 | 21-07-2019 02:00:00 | 48h0m0s | 192h0m0s   | 10,484  | 6,027,606   | 52,341     | 4          | false       | monitor=infrastructure,region=xxx,replica=A | 5m0s       | compactor |
| 01DGFKMJQ85CBF88A4YYRP69H2 | 21-07-2019 02:00:00 | 23-07-2019 02:00:00 | 48h0m0s | -8h0m0s    | 10,485  | 148,893,233 | 15,068,445 | 4          | false       | monitor=infrastructure,region=xxx,replica=A | 0s         | compactor |
| 01DGFKPEEDNNS4FJV0KJXDHVFP | 21-07-2019 02:00:00 | 23-07-2019 02:00:00 | 48h0m0s | 192h0m0s   | 10,485  | 6,027,407   | 52,342     | 4          | false       | monitor=infrastructure,region=xxx,replica=A | 5m0s       | compactor |
| 01DGFKKDM7WFRV9QY6NSKM305D | 23-07-2019 02:00:00 | 23-07-2019 10:00:00 | 8h0m0s  | 32h0m0s    | 10,474  | 24,816,728  | 2,511,507  | 3          | false       | monitor=infrastructure,region=xxx,replica=A | 0s         | compactor |
| 01DGHYNAVCFB8QGB0YBWZYTBE3 | 23-07-2019 10:00:00 | 23-07-2019 18:00:00 | 8h0m0s  | 32h0m0s    | 10,505  | 24,694,878  | 2,501,100  | 3          | false       | monitor=infrastructure,region=xxx,replica=A | 0s         | compactor |
| 01DGHYNEBTQJD0K946KV9MC032 | 23-07-2019 18:00:00 | 24-07-2019 02:00:00 | 8h0m0s  | 32h0m0s    | 10,422  | 24,721,020  | 2,499,849  | 3          | false       | monitor=infrastructure,region=xxx,replica=A | 0s         | compactor |
| 01DGHYNHSFNQPGZ5JZ5R1F6QWH | 24-07-2019 02:00:00 | 24-07-2019 10:00:00 | 8h0m0s  | 32h0m0s    | 10,418  | 24,720,981  | 2,499,846  | 3          | false       | monitor=infrastructure,region=xxx,replica=A | 0s         | compactor |
| 01DGHYN7GPW95CVS8DE33RQCSP | 24-07-2019 10:00:00 | 24-07-2019 12:00:00 | 2h0m0s  | 38h0m0s    | 10,508  | 6,098,310   | 622,197    | 2          | false       | monitor=infrastructure,region=xxx,replica=A | 0s         | compactor |
| 01DGHZTG7FJ09ZECW3KC1P2648 | 24-07-2019 12:00:00 | 24-07-2019 14:00:00 | 2h0m0s  | 38h0m0s    | 10,526  | 6,229,326   | 630,894    | 2          | false       | monitor=infrastructure,region=xxx,replica=A | 0s         | compactor |
| 01DGHXTCB6ERSVGD38A3K7DPJX | 24-07-2019 14:00:00 | 24-07-2019 14:02:00 | 2m0s    | 39h58m0s   | 6,664   | 54,372      | 6,664      | 1          | false       | monitor=infrastructure,region=xxx,replica=A | 0s         | sidecar   |
| 01DGHXXP576V3ZNJE1VF5KBPS8 | 24-07-2019 14:02:00 | 24-07-2019 14:04:00 | 2m0s    | 39h58m0s   | 10,473  | 39,960      | 10,473     | 1          | false       | monitor=infrastructure,region=xxx,replica=A | 0s         | sidecar   |
| 01DGHY1BBDBD6MFB5NVR3E2E9X | 24-07-2019 14:04:00 | 24-07-2019 14:06:00 | 2m0s    | 39h58m0s   | 10,473  | 103,504     | 10,473     | 1          | false       | monitor=infrastructure,region=xxx,replica=A | 0s         | sidecar   |
| 01DGHY50H7SFKNWZ65HXQY9NXB | 24-07-2019 14:06:00 | 24-07-2019 14:08:00 | 2m0s    | 39h58m0s   | 10,473  | 103,504     | 10,473     | 1          | false       | monitor=infrastructure,region=xxx,replica=A | 0s         | sidecar   |

At that point it looks to me as if there are downsampled data back in time until July, 13th.

If I connect to the thanos query daemon via Browser and do a query, the oldest data I get is from July, 18th. In Grafana I can also see no data older that July, 18th.

Next I tried the new web UI Feature, but honestly - I have to idea what the meaning of the colors is.

Cany anyone give me a hint why I can't see the downsampled data or where/how I can further investigate ?

Thanks a lot !

[GiedriusS]

Hi, what kind of query are you writing exactly?

[DirkFries]

I tried with several querys. e.g. just a simple "node_load5"

[DirkFries]

While trying to troubleshoot this issue I came across another interesting behaviour.

If I start the thanos query process with "--query.auto-downsampling" I eventually can see older data in Grafana if I choose a huge time range (e.g. 30d), but as soon as I zoom in they disappear again.

So both bucket inspection and "--query.auto-downsampling" prove, that there are downsampled data in Thanos, but they just don't appear in Grafana or the Thanos Web Query.

[GiedriusS]

Yes but that's expected if there is only one data point every 5 minutes, no? You should increase the Min Step. value in Grafana to 5m and you should see the data.

[DirkFries]

Well I don't zoom into a level of minutes, but day. All the sceenshots were made with "--query.auto-downsampling"

e.g. Here you can see the data of the last 30 days. On July, 13 the first samples appear. So far so good.

In the next step I am trying to zoom in to see if the samples are raw samples or downsampled ones and the time range I am starting with covers 3 whole days. Additionally I configured the 5 Min Step as suggested.

As you can see there are now samples anymore.

Maybe there's something wrong in my understanding of thanos. I thought that "--query.auto-downsampling" was a kind of "on the fly" downsampling during a request to limit the samples the Browser has to render and that you don't really need it if you run thanos compact with --retention.resolution-raw=7d --retention.resolution-5m=31d --retention.resolution-1h=365d on a regular base ?

The "--query.auto-downsampling" option was just a try to see any downsampled data. My preferred way would be to downsample my samples on a regular base.

[GiedriusS]

It sounds like you're running into https://github.com/thanos-io/thanos/issues/1170. Care to read through the whole ticket and see if it's the same problem?

[DirkFries]

Well I don't think that my experience is the same as in #1170 , because I have no 5m range defined in my query. In fact I have to time range at all defined in my query.

Is there anywhere a kind of documentation about the web UI so that I can check what those colors mean ? I'm still noch 100% sure if have really downsampled data.

[DirkFries]

Hi @all,

I tried to push Grafana for my troubleshoot aside and focus on Thanos, just to be sure that it is really a matter of Thanos.

So here are the facts:

This is an up-to-date bucket inspect

ULID	FROM	UNTIL	RANGE	UNTIL-COMP	#SERIES	#SAMPLES	#CHUNKS	COMP-LEVEL	COMP-FAILED	LABELS	RESOLUTION	SOURCE
01DGY92ZEQVAT0DH29PPA8D618	13-07-2019 02:00:00	25-07-2019 00:00:00	286h0m0s	-46h0m0s	11,523	34,897,623	283,539	6	false	monitor=infrastructure,region=Saarland,replica=A	5m0s	compactor
01DGY932KN9QHK99FX3HKCZ9P2	13-07-2019 02:00:00	25-07-2019 00:00:00	286h0m0s	-	11,523	2,932,159	31,855	6	false	monitor=infrastructure,region=Saarland,replica=A	1h0m0s	compactor
01DGY7TDPTZ0H1E4HFD55EDB23	25-07-2019 02:00:00	27-07-2019 02:00:00	48h0m0s	192h0m0s	10,519	4,946,076	41,734	3	false	monitor=infrastructure,region=Saarland,replica=A	5m0s	compactor
01DGY7TAFJGRYZBV73W6MNJE1P	27-07-2019 02:00:00	29-07-2019 02:00:00	48h0m0s	-8h0m0s	10,412	148,127,895	1,234,398	3	false	monitor=infrastructure,region=Saarland,replica=A	0s	compactor
01DGY7TNXXMHCA76VBBVEM21N2	27-07-2019 02:00:00	29-07-2019 02:00:00	48h0m0s	192h0m0s	10,412	5,990,291	52,012	3	false	monitor=infrastructure,region=Saarland,replica=A	5m0s	compactor
01DH3GN6AW1ZN7V8HEEM7QEBW6	29-07-2019 02:00:00	31-07-2019 02:00:00	48h0m0s	-8h0m0s	10,536	148,507,872	1,218,188	3	false	monitor=infrastructure,region=Saarland,replica=A	0s	compactor
01DH3GN9GHA7Q4KM20TGK0AJBZ	29-07-2019 02:00:00	31-07-2019 02:00:00	48h0m0s	192h0m0s	10,536	6,012,848	52,349	3	false	monitor=infrastructure,region=Saarland,replica=A	5m0s	compactor
01DH2ZGSJBZPD4B6QD4FCHMWXE	31-07-2019 02:00:00	31-07-2019 04:00:00	2h0m0s	38h0m0s	10,481	6,211,440	51,762	1	false	monitor=infrastructure,region=Saarland,replica=A	0s	sidecar
01DH36CGTH2K8PACQEDJNZDW3F	31-07-2019 04:00:00	31-07-2019 06:00:00	2h0m0s	38h0m0s	10,485	6,211,510	51,766	1	false	monitor=infrastructure,region=Saarland,replica=A	0s	sidecar
01DH3D882CK1J8W55H4ZQ81QWH	31-07-2019 06:00:00	31-07-2019 08:00:00	2h0m0s	38h0m0s	10,482	6,211,447	51,763	1	false	monitor=infrastructure,region=Saarland,replica=A	0s	sidecar

level=info ts=2019-07-31T08:00:04.89823058Z caller=main.go:202 msg=exiting

The first ULID is from 13-07-2019 02:00:00 So my expectation is, that there are data from around that time.

This expecation is supported by the Thanos Bucket Viewer:

BUT

If I try to query Thanos (not Prometheus!) via Webinterface there are no data prior to July 27th

e.g. If I query about some default metrics vom July 14th there are now datapoints

Here you can see the border between data and no data in the very early minutes of July 27th

So the big question to me is - Why are there timestamps in the bucket from the July 13th or July 25th and Thanos can not display those data ? Is has to be connected with Downsampling, because older Data "disappears" after downsampling. But Downsampling is just the reason I would like to use Thanos for.

Does anyone have an idea or hint what might be going wrong in my case ?

Bye, Dirk

[sparkblaze]

If you change 'Only raw data' above the the graph, do you then see more data?

I think we're experiencing the same issue - it's similar to the linked issue, but in our case, we're not using rate functions, it just flat out isn't returning 1h samples when requested - even in the thanos UI.

As an example for us, we can see 'Only raw data' back 5 days, auto downsample goes back 7 days, max 5m about 14 days (even though it should go back 30) and max 1h goes back 7 days.

[DirkFries]

Hi @all,

I finally found out what went wrong in my case and of course it was a configuration/understanding error. I thought that downsampling was done during the compacting process. So I just compacted and had not downsampled first - So no wonder why my data was gone.

In the meantime I found out, that you have to use the "--query.auto-downsampling" Option in thanos query as mentioned here https://github.com/thanos-io/thanos/issues/601 if you want to query your data with Grafana.

If I do this I can see downsampled data in Grafana, but as soon as I zoom into a downsampled time range Grafana tells me there are no datapoints. I think this is this issue https://github.com/thanos-io/thanos/issues/1170 mentioned already by GiedriusS.

So in my usecase (long term storage and downsampling of Prometheus data und querying those data with Grafana) Thanos does not seem to be the perfect tool at the moment and I have to look for something else.

But I'll sure keep an eye on this project :-)

[bwplotka]

So in my usecase (long term storage and downsampling of Prometheus data und querying those data with Grafana) Thanos does not seem to be the perfect tool at the moment and I have to look for something else.

That's interesting as Thanos is exactly the project which solves this (:

Looks like downsampling does not work as intended without any aggregations. Rate is one thing (with big enough interval), but any sum/min/max/avg will work as well. I think we should explain that better AND ensure it's not surprising user. Maybe default aggregation or just absolute values would be nice to add then? There is not much sense in graphing your metrics over 5w without aggregations as you will see partial information every 1h. However in your case avg(promethues_sd_discovered_targets) would be more interesting to see over 5w.

[bwplotka]

Although avg is actually the default :thinking:

    // In the default case, we retrieve count and sum to compute an average.
    return []storepb.Aggr{storepb.Aggr_COUNT, storepb.Aggr_SUM}, resAggrAvg

https://github.com/thanos-io/thanos/blob/2c5f2cde11f5cd100f147ad2e5d4dbeccbd604c5/pkg/query/querier.go#L168

So it might be just a simple bug then ): Will look on that today, sorry for troubles.

[bwplotka]

@DirkFries can you confirm my assumptions are correct? E.g if you do any aggregation like sum sum_over_time or avg you have the data?

[DirkFries]

Hi bwplotka,

unfortunaly not.

Here's what I tried :

1. Setting up a query with avg_over_time with a range of 30m. Please note, that the Graph starts at July, 31th. On that day the data is downsampled to 5m.

2. I zoom in

3. After zooming in there are no data

If there are any more tests I can do to support you I'd be glad to help. So please feel free to ask. :-) Unfortunaly I can not give you access to this installation I am working on right now, but I could set up a similar one and give you access to that one if this would help.

Bye, Dirk

[bwplotka]

I would use Thanos UI for tests. It is only raw data by default so this result is expected. Can you check it from Querier with auto downsampling e.g? You can change the default as you mentioned in "--query.auto-downsampling"

[DirkFries]

If thanos query process is started with "--query.auto-downsampling" and "Auto Downsampling" chosen in Dropdown Menu I can see downsampled data both with an aggregation and without an aggregation in the query:

So I guess it's more an issue between Thanos and Grafana.

[bwplotka]

Ok, that looks good then, right? Just set this flag on querier:

--query.auto-downsampling  Enable automatic adjustment (step / 5) to what
                                 source of data should be used in store gateways
                                 if no max_source_resolution param is specified.

And then Grafana should use exactly the same thing Thanos UI uses - so it should give you the data.

[bwplotka]

We will make this by default true, once we will create more docs for downsampling as well as address the rate intervals mistake that is easy to make (:

[DirkFries]

Unforunaly I ran the thanos query process already with "--query.auto-downsampling" when I did this Test with Grafana

https://github.com/thanos-io/thanos/issues/1353#issuecomment-518141677

[bwplotka]

Awesome, let me check if there is anything suspicious in this path, but glad we narrowed down the issue.

Can you also play with step a bit? Essentially auto downsampling checks step first to tell what downsampling (if any to use). It might be that we miscalculate that. Or we should maybe fallback better to different resolution if one is not available. Especially if you zoom step is narrow so it asks Thanos for raw data strictly.

During Grafana test if you will grab the actual URL that was done to Thanos API (on chrome F12 > Network and refresh graph) it would be nice as well - it will show the step.

[bwplotka]

cc @GiedriusS and @povilasv as we discussed what we should fallback into in different downsampling cases (:

[bwplotka]

Essentially:

AC:

• Make the usage of downsampling smooth in the case of different retentions. I think auto should give what it has.

[DirkFries]

I'm not quite sure what you mean with "step". It is meantioned here https://improbable.io/blog/thanos-prometheus-at-scale but I could not find an example how to do it. Could please give me an example or show me where it is documented ?

[bwplotka]

Sure!

https://www.robustperception.io/step-and-query_range

Also Grafana allows to control resolution from here:

[DirkFries]

Ah thanks - I got it :-)

Well, if I choose a step size of 20m or below I get no datapoints, but with 25m and higher I do get datapoints.

Interestingly if I paste the URL Grafana send to Thanos in my Browser, I get no results with a step size of 20m, but I do get the results as json output with a step size of 25m.

[DirkFries]

Hi @all,

just to give some feedback. I just learned about a tool called Trickster that can be used as a reverse proxy for Proemtheus(... or Thanos) to accelerate dashboard rendering times.

One of it's featurs is described as :

"Step Boundary Normalization

When Trickster requests data from Prometheus, it adjusts the clients's requested time range slightly to ensure that all data points returned by Prometheus are aligned to normalized step boundaries. For example, if the step is 300s, all data points will fall on the clock 0's and 5's. This ensures that the data is highly cacheable, is conveyed visually to users in a more familiar way, and that all dashboard users see identical data on their screens."

I had hoped that Step Boundary Normalization would bypass my problems between Thanos and Grafana but unfortunaly the effect stays the same with Trickster.

[matejzero]

Well, if I choose a step size of 20m or below I get no datapoints, but with 25m and higher I do get datapoints.

I can confirm, I get the same behaviour. Query up{instance="host.example.tld",job="node"} wont show data if step is lower than 25m. Data for this metrics are in 5m downsampled buckets.

[seuf]

I've opened a PR on Grafana to add Thanos max_source_resolution option to thanos query : https://github.com/grafana/grafana/pull/19121 Like in thanos query UI the is a auto value which mean max_source_resolution="" but it only return raw data :(

[seuf]

the PR purpose has changed to a more generic "customQueryParameters" configuration in prometheus datasource. and now it's merged :)

[DirkFries]

Hi seuf,

this sounds like a great addition to Grafana, but I'm not sure if this really solves the problem or I maybe I understood somthing wrong or missing something.

e.g. I'm looking at a timerange that contains both steps of 15 seconds and 5 minutes I will have to adjust the value to 5m. If I zoom in to a range with only 15 seconds I have to manualy change the value to 15s and so on. So everytime I change the timerange I have to change the value.

On the other hand I could set the value to 1h, but I don't this that this would be appropriate.

Am I right with my assumption or have I missed something or got something wrong? (Hopyfully the latter one)

The missing Graphs in Grafana is the last issue that keeps me from using Thanos in production, so I'm pretty exited about this possible solution.

Bye, Dirk

[seuf]

No, the parameter max_source_resolution tell thanos query what is the maximum resolution allowed. Thanos query will always return the smallest resolution available. For example if you have a period with 15s and 5m resolution stored in thanos, it will return the smallest resolution available, even if you have set max_source_resolution=5m.

[DirkFries]

Hi seuf,

thanks a lot for your explanation. This sounds like a sweet dream to me. Just can't wait to test this feature in grafana :-)

[sepich]

Hello, i'm using v0.7.0 with compactor settings:

        - --retention.resolution-raw=14d
        - --retention.resolution-5m=30d
        - --retention.resolution-1h=0d

Grafana Last 90 days time span has $__interval = 2h Here is list of issues with it:

• Plain metric request like node_network_transmit_bytes_total transforms to &step=7200 and only last ~29 days are returned
• avg_over_time(node_network_transmit_bytes_total[$__interval]) gets avg_over_time(node_network_transmit_bytes_total[2h])&step=7200 and returns the same ~29days
• if i change in request &step=18000 - it returns whole 90days But the only way to change step in grafana is to reduce resolution from 1/1 to 1/3. I obviously do not want to lower it, as it is configuration for all time spans. The same results could be reproduced in thanos-query UI.

Grafana time span Last 6 month has $__interval = 6h (21600) and works fine.

Do i understand correctly that avg([2h])&span=18000 means i'm skipping the data, only drawing first 2h from each 5h(18000s)? Why there are so high step requirements for 1h averages in thanos?

[sepich]

Rechecked on another thanos installation with different bucket (AWS) and only first case with plain metric query fails. avg_over_time([2h])&step=7200 returns all 90 days.

So, it seems issue is somewhere inside of first bucket content and maybe &step=18000 just able to skip some corrupted data. Tried to check the bucket - no issues:

/ $ thanos bucket verify --objstore.config-file=/run/secrets/storage.yml
level=info ts=2019-09-26T18:42:37.867018361Z caller=main.go:168 msg="Tracing will be disabled"
level=info ts=2019-09-26T18:42:37.867105361Z caller=factory.go:39 msg="loading bucket configuration"
level=warn ts=2019-09-26T18:42:37.932374217Z caller=verify.go:49 msg="GLOBAL COMPACTOR SHOULD __NOT__ BE RUNNING ON THE SAME BUCKET" issues=2 repair=false
level=info ts=2019-09-26T18:42:37.932422918Z caller=index_issue.go:29 msg="started verifying issue" with-repair=false issue=index_issue
level=info ts=2019-09-26T18:43:11.233606686Z caller=index_issue.go:130 msg="verified issue" with-repair=false issue=index_issue
level=info ts=2019-09-26T18:43:11.233675686Z caller=overlapped_blocks.go:26 msg="started verifying issue" with-repair=false issue=overlapped_blocks
level=info ts=2019-09-26T18:43:14.413838542Z caller=verify.go:68 msg="verify completed" issues=2 repair=false
level=info ts=2019-09-26T18:43:14.414399145Z caller=main.go:216 msg=exiting

Restarted thanos-store with --log.level=debug and ran query (from thanos UI): api/v1/query_range?query=avg_over_time(node_network_transmit_bytes_total%7Bmonitor%3D%22srs-us4%22%2C%20device%3D%22eth0%22%7D%5B2h%5D)&dedup=true&partial_response=true&start=1561748490.034&end=1569524490.034&step=7200&max_source_resolution=&_=1569226210498

thanos-store logs with step=7200

``` k -n monitoring logs -f thanos-store-57cf77c4db-rw7tw level=debug ts=2019-09-26T18:52:28.842240471Z caller=main.go:120 msg="maxprocs: Leaving GOMAXPROCS=[4]: CPU quota undefined" level=info ts=2019-09-26T18:52:28.842439772Z caller=main.go:168 msg="Tracing will be disabled" level=info ts=2019-09-26T18:52:28.842504272Z caller=factory.go:39 msg="loading bucket configuration" level=debug ts=2019-09-26T18:52:28.842666473Z caller=azure.go:66 msg="creating new Azure bucket connection" component=store level=debug ts=2019-09-26T18:52:29.056512213Z caller=azure.go:85 msg="Getting connection to existing Azure blob container" container=thanos-store level=info ts=2019-09-26T18:52:29.061058036Z caller=cache.go:172 msg="created index cache" maxItemSizeBytes=5368709120 maxSizeBytes=10737418240 maxItems=math.MaxInt64 level=debug ts=2019-09-26T18:52:29.061303137Z caller=store.go:162 msg="initializing bucket store" level=debug ts=2019-09-26T18:52:29.069379576Z caller=azure.go:169 msg="getting blob" blob=01D1Z0ZFZJ36N55B4Q47X1880J/meta.json offset=0 length=0 level=debug ts=2019-09-26T18:52:29.069752478Z caller=azure.go:229 msg="check if blob exists" blob=01D1Z0ZFZJ36N55B4Q47X1880J/meta.json level=debug ts=2019-09-26T18:52:29.18931396Z caller=azure.go:169 msg="getting blob" blob=01D32QZJ1THDMBBAQDS0SG43B8/meta.json offset=0 length=0 level=debug ts=2019-09-26T18:52:29.18935766Z caller=azure.go:229 msg="check if blob exists" blob=01D32QZJ1THDMBBAQDS0SG43B8/meta.json level=debug ts=2019-09-26T18:52:29.189531661Z caller=azure.go:169 msg="getting blob" blob=01D1Z0ZFZJ36N55B4Q47X1880J/index.cache.json offset=0 length=0 level=debug ts=2019-09-26T18:52:29.189572661Z caller=azure.go:229 msg="check if blob exists" blob=01D1Z0ZFZJ36N55B4Q47X1880J/index.cache.json level=debug ts=2019-09-26T18:52:29.199065107Z caller=azure.go:169 msg="getting blob" blob=01D1Z0ZFZJ36N55B4Q47X1880J/index offset=0 length=0 level=debug ts=2019-09-26T18:52:29.199093507Z caller=azure.go:229 msg="check if blob exists" blob=01D1Z0ZFZJ36N55B4Q47X1880J/index level=debug ts=2019-09-26T18:52:29.237191993Z caller=azure.go:169 msg="getting blob" blob=01D46SW2YPJYYVD34RREFA27MD/meta.json offset=0 length=0 level=debug ts=2019-09-26T18:52:29.237237393Z caller=azure.go:229 msg="check if blob exists" blob=01D46SW2YPJYYVD34RREFA27MD/meta.json level=debug ts=2019-09-26T18:52:29.237707895Z caller=azure.go:169 msg="getting blob" blob=01D32QZJ1THDMBBAQDS0SG43B8/index.cache.json offset=0 length=0 level=debug ts=2019-09-26T18:52:29.237739295Z caller=azure.go:229 msg="check if blob exists" blob=01D32QZJ1THDMBBAQDS0SG43B8/index.cache.json level=debug ts=2019-09-26T18:52:29.245851235Z caller=azure.go:169 msg="getting blob" blob=01D32QZJ1THDMBBAQDS0SG43B8/index offset=0 length=0 level=debug ts=2019-09-26T18:52:29.245879435Z caller=azure.go:229 msg="check if blob exists" blob=01D32QZJ1THDMBBAQDS0SG43B8/index level=debug ts=2019-09-26T18:52:29.335796973Z caller=azure.go:169 msg="getting blob" blob=01D5AVFFB01TDNVA3QDENHGRXG/meta.json offset=0 length=0 level=debug ts=2019-09-26T18:52:29.335849273Z caller=azure.go:229 msg="check if blob exists" blob=01D5AVFFB01TDNVA3QDENHGRXG/meta.json level=debug ts=2019-09-26T18:52:29.336971378Z caller=azure.go:169 msg="getting blob" blob=01D46SW2YPJYYVD34RREFA27MD/index.cache.json offset=0 length=0 level=debug ts=2019-09-26T18:52:29.337005278Z caller=azure.go:229 msg="check if blob exists" blob=01D46SW2YPJYYVD34RREFA27MD/index.cache.json level=debug ts=2019-09-26T18:52:29.341708301Z caller=azure.go:169 msg="getting blob" blob=01D46SW2YPJYYVD34RREFA27MD/index offset=0 length=0 level=debug ts=2019-09-26T18:52:29.341757902Z caller=azure.go:229 msg="check if blob exists" blob=01D46SW2YPJYYVD34RREFA27MD/index level=debug ts=2019-09-26T18:52:29.463625295Z caller=azure.go:169 msg="getting blob" blob=01D6EX210CGQXG581PZ96NV5X5/meta.json offset=0 length=0 level=debug ts=2019-09-26T18:52:29.463678695Z caller=azure.go:229 msg="check if blob exists" blob=01D6EX210CGQXG581PZ96NV5X5/meta.json level=debug ts=2019-09-26T18:52:29.465229802Z caller=azure.go:169 msg="getting blob" blob=01D5AVFFB01TDNVA3QDENHGRXG/index.cache.json offset=0 length=0 level=debug ts=2019-09-26T18:52:29.465268703Z caller=azure.go:229 msg="check if blob exists" blob=01D5AVFFB01TDNVA3QDENHGRXG/index.cache.json level=debug ts=2019-09-26T18:52:29.470353327Z caller=azure.go:169 msg="getting blob" blob=01D5AVFFB01TDNVA3QDENHGRXG/index offset=0 length=0 level=debug ts=2019-09-26T18:52:29.470382428Z caller=azure.go:229 msg="check if blob exists" blob=01D5AVFFB01TDNVA3QDENHGRXG/index level=debug ts=2019-09-26T18:52:29.533001732Z caller=azure.go:169 msg="getting blob" blob=01D7JYK8ZZW83QHRJHHH9KQKWY/meta.json offset=0 length=0 level=debug ts=2019-09-26T18:52:29.533378134Z caller=azure.go:229 msg="check if blob exists" blob=01D7JYK8ZZW83QHRJHHH9KQKWY/meta.json level=debug ts=2019-09-26T18:52:29.534748741Z caller=azure.go:169 msg="getting blob" blob=01D6EX210CGQXG581PZ96NV5X5/index.cache.json offset=0 length=0 level=debug ts=2019-09-26T18:52:29.534778641Z caller=azure.go:229 msg="check if blob exists" blob=01D6EX210CGQXG581PZ96NV5X5/index.cache.json level=debug ts=2019-09-26T18:52:29.540442268Z caller=azure.go:169 msg="getting blob" blob=01D6EX210CGQXG581PZ96NV5X5/index offset=0 length=0 level=debug ts=2019-09-26T18:52:29.540620369Z caller=azure.go:229 msg="check if blob exists" blob=01D6EX210CGQXG581PZ96NV5X5/index level=debug ts=2019-09-26T18:52:29.585093886Z caller=azure.go:169 msg="getting blob" blob=01D8Q0761453ZNV447Z4BDCRJR/meta.json offset=0 length=0 level=debug ts=2019-09-26T18:52:29.585343987Z caller=azure.go:229 msg="check if blob exists" blob=01D8Q0761453ZNV447Z4BDCRJR/meta.json level=debug ts=2019-09-26T18:52:29.586692294Z caller=azure.go:169 msg="getting blob" blob=01D7JYK8ZZW83QHRJHHH9KQKWY/index.cache.json offset=0 length=0 level=debug ts=2019-09-26T18:52:29.586731294Z caller=azure.go:229 msg="check if blob exists" blob=01D7JYK8ZZW83QHRJHHH9KQKWY/index.cache.json level=debug ts=2019-09-26T18:52:29.590488712Z caller=azure.go:169 msg="getting blob" blob=01D7JYK8ZZW83QHRJHHH9KQKWY/index offset=0 length=0 level=debug ts=2019-09-26T18:52:29.590607113Z caller=azure.go:229 msg="check if blob exists" blob=01D7JYK8ZZW83QHRJHHH9KQKWY/index level=debug ts=2019-09-26T18:52:29.717978132Z caller=azure.go:169 msg="getting blob" blob=01D9V1DCTPNFQADXTAX8E6P6EA/meta.json offset=0 length=0 level=debug ts=2019-09-26T18:52:29.71949474Z caller=azure.go:229 msg="check if blob exists" blob=01D9V1DCTPNFQADXTAX8E6P6EA/meta.json level=debug ts=2019-09-26T18:52:29.724026162Z caller=azure.go:169 msg="getting blob" blob=01D8Q0761453ZNV447Z4BDCRJR/index.cache.json offset=0 length=0 level=debug ts=2019-09-26T18:52:29.724362664Z caller=azure.go:229 msg="check if blob exists" blob=01D8Q0761453ZNV447Z4BDCRJR/index.cache.json level=debug ts=2019-09-26T18:52:29.731240497Z caller=azure.go:169 msg="getting blob" blob=01D8Q0761453ZNV447Z4BDCRJR/index offset=0 length=0 level=debug ts=2019-09-26T18:52:29.731282097Z caller=azure.go:229 msg="check if blob exists" blob=01D8Q0761453ZNV447Z4BDCRJR/index level=debug ts=2019-09-26T18:52:29.78322635Z caller=azure.go:169 msg="getting blob" blob=01DAZ2V6HK8AM7FQ77GWB2Q5TN/meta.json offset=0 length=0 level=debug ts=2019-09-26T18:52:29.78328095Z caller=azure.go:229 msg="check if blob exists" blob=01DAZ2V6HK8AM7FQ77GWB2Q5TN/meta.json level=debug ts=2019-09-26T18:52:29.784790858Z caller=azure.go:169 msg="getting blob" blob=01D9V1DCTPNFQADXTAX8E6P6EA/index.cache.json offset=0 length=0 level=debug ts=2019-09-26T18:52:29.784819558Z caller=azure.go:229 msg="check if blob exists" blob=01D9V1DCTPNFQADXTAX8E6P6EA/index.cache.json level=debug ts=2019-09-26T18:52:29.790083383Z caller=azure.go:169 msg="getting blob" blob=01D9V1DCTPNFQADXTAX8E6P6EA/index offset=0 length=0 level=debug ts=2019-09-26T18:52:29.790247084Z caller=azure.go:229 msg="check if blob exists" blob=01D9V1DCTPNFQADXTAX8E6P6EA/index level=debug ts=2019-09-26T18:52:29.851263281Z caller=azure.go:169 msg="getting blob" blob=01DC34N0SYH60BD1B0HKM4MC7D/meta.json offset=0 length=0 level=debug ts=2019-09-26T18:52:29.851311481Z caller=azure.go:229 msg="check if blob exists" blob=01DC34N0SYH60BD1B0HKM4MC7D/meta.json level=debug ts=2019-09-26T18:52:29.852534387Z caller=azure.go:169 msg="getting blob" blob=01DAZ2V6HK8AM7FQ77GWB2Q5TN/index.cache.json offset=0 length=0 level=debug ts=2019-09-26T18:52:29.852561887Z caller=azure.go:229 msg="check if blob exists" blob=01DAZ2V6HK8AM7FQ77GWB2Q5TN/index.cache.json level=debug ts=2019-09-26T18:52:29.869181268Z caller=azure.go:169 msg="getting blob" blob=01DAZ2V6HK8AM7FQ77GWB2Q5TN/index offset=0 length=0 level=debug ts=2019-09-26T18:52:29.869217568Z caller=azure.go:229 msg="check if blob exists" blob=01DAZ2V6HK8AM7FQ77GWB2Q5TN/index level=debug ts=2019-09-26T18:52:29.994765679Z caller=azure.go:169 msg="getting blob" blob=01DD76942ZEV3CSK5H8RVPB16T/meta.json offset=0 length=0 level=debug ts=2019-09-26T18:52:29.999130101Z caller=azure.go:229 msg="check if blob exists" blob=01DD76942ZEV3CSK5H8RVPB16T/meta.json level=debug ts=2019-09-26T18:52:29.998432097Z caller=azure.go:169 msg="getting blob" blob=01DC34N0SYH60BD1B0HKM4MC7D/index.cache.json offset=0 length=0 level=debug ts=2019-09-26T18:52:29.999728004Z caller=azure.go:229 msg="check if blob exists" blob=01DC34N0SYH60BD1B0HKM4MC7D/index.cache.json level=debug ts=2019-09-26T18:52:30.358258849Z caller=azure.go:169 msg="getting blob" blob=01DEB7V9XA6V0C1C53RRJJEVC6/meta.json offset=0 length=0 level=debug ts=2019-09-26T18:52:30.358320049Z caller=azure.go:229 msg="check if blob exists" blob=01DEB7V9XA6V0C1C53RRJJEVC6/meta.json level=debug ts=2019-09-26T18:52:30.358838052Z caller=azure.go:169 msg="getting blob" blob=01DD76942ZEV3CSK5H8RVPB16T/index.cache.json offset=0 length=0 level=debug ts=2019-09-26T18:52:30.358939952Z caller=azure.go:229 msg="check if blob exists" blob=01DD76942ZEV3CSK5H8RVPB16T/index.cache.json level=debug ts=2019-09-26T18:52:30.663388434Z caller=azure.go:169 msg="getting blob" blob=01DFF9NPD5FMJV67QQKXQES9VG/meta.json offset=0 length=0 level=debug ts=2019-09-26T18:52:30.663453634Z caller=azure.go:229 msg="check if blob exists" blob=01DFF9NPD5FMJV67QQKXQES9VG/meta.json level=debug ts=2019-09-26T18:52:30.664496039Z caller=azure.go:169 msg="getting blob" blob=01DEB7V9XA6V0C1C53RRJJEVC6/index.cache.json offset=0 length=0 level=debug ts=2019-09-26T18:52:30.66470954Z caller=azure.go:229 msg="check if blob exists" blob=01DEB7V9XA6V0C1C53RRJJEVC6/index.cache.json level=debug ts=2019-09-26T18:52:31.0057044Z caller=azure.go:169 msg="getting blob" blob=01DGKB9BMYMSCC1V6X0DPB9HK7/meta.json offset=0 length=0 level=debug ts=2019-09-26T18:52:31.005803501Z caller=azure.go:229 msg="check if blob exists" blob=01DGKB9BMYMSCC1V6X0DPB9HK7/meta.json level=debug ts=2019-09-26T18:52:31.006792106Z caller=azure.go:169 msg="getting blob" blob=01DFF9NPD5FMJV67QQKXQES9VG/index.cache.json offset=0 length=0 level=debug ts=2019-09-26T18:52:31.006825706Z caller=azure.go:229 msg="check if blob exists" blob=01DFF9NPD5FMJV67QQKXQES9VG/index.cache.json level=debug ts=2019-09-26T18:52:31.066713297Z caller=azure.go:169 msg="getting blob" blob=01DHQCWD4AFE4JEFDZF0Q1CQFN/meta.json offset=0 length=0 level=debug ts=2019-09-26T18:52:31.066769198Z caller=azure.go:229 msg="check if blob exists" blob=01DHQCWD4AFE4JEFDZF0Q1CQFN/meta.json level=debug ts=2019-09-26T18:52:31.067759802Z caller=azure.go:169 msg="getting blob" blob=01DGKB9BMYMSCC1V6X0DPB9HK7/index.cache.json offset=0 length=0 level=debug ts=2019-09-26T18:52:31.067789103Z caller=azure.go:229 msg="check if blob exists" blob=01DGKB9BMYMSCC1V6X0DPB9HK7/index.cache.json level=debug ts=2019-09-26T18:52:31.144630477Z caller=azure.go:169 msg="getting blob" blob=01DJVEEVYA6ABF57ZVEQW57NWW/meta.json offset=0 length=0 level=debug ts=2019-09-26T18:52:31.144691777Z caller=azure.go:229 msg="check if blob exists" blob=01DJVEEVYA6ABF57ZVEQW57NWW/meta.json level=debug ts=2019-09-26T18:52:31.146041384Z caller=azure.go:169 msg="getting blob" blob=01DHQCWD4AFE4JEFDZF0Q1CQFN/index.cache.json offset=0 length=0 level=debug ts=2019-09-26T18:52:31.146068884Z caller=azure.go:229 msg="check if blob exists" blob=01DHQCWD4AFE4JEFDZF0Q1CQFN/index.cache.json level=debug ts=2019-09-26T18:52:31.267207374Z caller=azure.go:169 msg="getting blob" blob=01DKZFRCA5YH02S7TH5FM253PM/meta.json offset=0 length=0 level=debug ts=2019-09-26T18:52:31.267328674Z caller=azure.go:229 msg="check if blob exists" blob=01DKZFRCA5YH02S7TH5FM253PM/meta.json level=debug ts=2019-09-26T18:52:31.268151478Z caller=azure.go:169 msg="getting blob" blob=01DJVEEVYA6ABF57ZVEQW57NWW/index.cache.json offset=0 length=0 level=debug ts=2019-09-26T18:52:31.268192178Z caller=azure.go:229 msg="check if blob exists" blob=01DJVEEVYA6ABF57ZVEQW57NWW/index.cache.json level=debug ts=2019-09-26T18:52:31.358610119Z caller=azure.go:169 msg="getting blob" blob=01DKZG25YFZW3GM1PATHFS4D23/meta.json offset=0 length=0 level=debug ts=2019-09-26T18:52:31.358660119Z caller=azure.go:229 msg="check if blob exists" blob=01DKZG25YFZW3GM1PATHFS4D23/meta.json level=debug ts=2019-09-26T18:52:31.359826125Z caller=azure.go:169 msg="getting blob" blob=01DKZFRCA5YH02S7TH5FM253PM/index.cache.json offset=0 length=0 level=debug ts=2019-09-26T18:52:31.359855325Z caller=azure.go:229 msg="check if blob exists" blob=01DKZFRCA5YH02S7TH5FM253PM/index.cache.json level=debug ts=2019-09-26T18:52:31.544253122Z caller=azure.go:169 msg="getting blob" blob=01DN3GM2VFJREYJTT190AQ0EDZ/meta.json offset=0 length=0 level=debug ts=2019-09-26T18:52:31.544310723Z caller=azure.go:229 msg="check if blob exists" blob=01DN3GM2VFJREYJTT190AQ0EDZ/meta.json level=debug ts=2019-09-26T18:52:31.545644229Z caller=azure.go:169 msg="getting blob" blob=01DKZG25YFZW3GM1PATHFS4D23/index.cache.json offset=0 length=0 level=debug ts=2019-09-26T18:52:31.545671729Z caller=azure.go:229 msg="check if blob exists" blob=01DKZG25YFZW3GM1PATHFS4D23/index.cache.json level=debug ts=2019-09-26T18:52:31.619128887Z caller=azure.go:148 msg="requesting next iteration of listing blobs" last_entries=19 iteration=1 level=debug ts=2019-09-26T18:52:31.620417793Z caller=azure.go:169 msg="getting blob" blob=01DN3GM2VFJREYJTT190AQ0EDZ/index.cache.json offset=0 length=0 level=debug ts=2019-09-26T18:52:31.620450693Z caller=azure.go:229 msg="check if blob exists" blob=01DN3GM2VFJREYJTT190AQ0EDZ/index.cache.json level=debug ts=2019-09-26T18:52:31.665724714Z caller=azure.go:169 msg="getting blob" blob=01DN3H9NA1KGS4TKA4MYHY09MR/meta.json offset=0 length=0 level=debug ts=2019-09-26T18:52:31.665779114Z caller=azure.go:229 msg="check if blob exists" blob=01DN3H9NA1KGS4TKA4MYHY09MR/meta.json level=debug ts=2019-09-26T18:52:31.727442114Z caller=azure.go:169 msg="getting blob" blob=01DN3HMPA27V4QJBENN6GBVR7X/meta.json offset=0 length=0 level=debug ts=2019-09-26T18:52:31.727487015Z caller=azure.go:229 msg="check if blob exists" blob=01DN3HMPA27V4QJBENN6GBVR7X/meta.json level=debug ts=2019-09-26T18:52:31.728868021Z caller=azure.go:169 msg="getting blob" blob=01DN3H9NA1KGS4TKA4MYHY09MR/index.cache.json offset=0 length=0 level=debug ts=2019-09-26T18:52:31.728932922Z caller=azure.go:229 msg="check if blob exists" blob=01DN3H9NA1KGS4TKA4MYHY09MR/index.cache.json level=debug ts=2019-09-26T18:52:31.81495984Z caller=azure.go:169 msg="getting blob" blob=01DN8N123TCWZVXCGZ0CQ17M3E/meta.json offset=0 length=0 level=debug ts=2019-09-26T18:52:31.815286742Z caller=azure.go:229 msg="check if blob exists" blob=01DN8N123TCWZVXCGZ0CQ17M3E/meta.json level=debug ts=2019-09-26T18:52:31.817392652Z caller=azure.go:169 msg="getting blob" blob=01DN3HMPA27V4QJBENN6GBVR7X/index.cache.json offset=0 length=0 level=debug ts=2019-09-26T18:52:31.817426852Z caller=azure.go:229 msg="check if blob exists" blob=01DN3HMPA27V4QJBENN6GBVR7X/index.cache.json level=debug ts=2019-09-26T18:52:31.854501433Z caller=azure.go:169 msg="getting blob" blob=01DN8N4PJSNE9F07XJ78RZWN3G/meta.json offset=0 length=0 level=debug ts=2019-09-26T18:52:31.854559533Z caller=azure.go:229 msg="check if blob exists" blob=01DN8N4PJSNE9F07XJ78RZWN3G/meta.json level=debug ts=2019-09-26T18:52:31.854729534Z caller=azure.go:169 msg="getting blob" blob=01DN8N123TCWZVXCGZ0CQ17M3E/index.cache.json offset=0 length=0 level=debug ts=2019-09-26T18:52:31.854776634Z caller=azure.go:229 msg="check if blob exists" blob=01DN8N123TCWZVXCGZ0CQ17M3E/index.cache.json level=debug ts=2019-09-26T18:52:31.861765668Z caller=azure.go:148 msg="requesting next iteration of listing blobs" last_entries=3 iteration=1 level=debug ts=2019-09-26T18:52:31.91745814Z caller=azure.go:169 msg="getting blob" blob=01DNDSR1RXTY8GDXY0QGT02047/meta.json offset=0 length=0 level=debug ts=2019-09-26T18:52:31.91750264Z caller=azure.go:229 msg="check if blob exists" blob=01DNDSR1RXTY8GDXY0QGT02047/meta.json level=debug ts=2019-09-26T18:52:31.917684141Z caller=azure.go:169 msg="getting blob" blob=01DN8N4PJSNE9F07XJ78RZWN3G/index.cache.json offset=0 length=0 level=debug ts=2019-09-26T18:52:31.917763941Z caller=azure.go:229 msg="check if blob exists" blob=01DN8N4PJSNE9F07XJ78RZWN3G/index.cache.json level=debug ts=2019-09-26T18:52:31.986096974Z caller=azure.go:169 msg="getting blob" blob=01DNDSVHEHJKWMBHGFBEM0VBWZ/meta.json offset=0 length=0 level=debug ts=2019-09-26T18:52:31.986406675Z caller=azure.go:229 msg="check if blob exists" blob=01DNDSVHEHJKWMBHGFBEM0VBWZ/meta.json level=debug ts=2019-09-26T18:52:31.988564386Z caller=azure.go:169 msg="getting blob" blob=01DNDSR1RXTY8GDXY0QGT02047/index.cache.json offset=0 length=0 level=debug ts=2019-09-26T18:52:31.988603086Z caller=azure.go:229 msg="check if blob exists" blob=01DNDSR1RXTY8GDXY0QGT02047/index.cache.json level=debug ts=2019-09-26T18:52:32.094012399Z caller=azure.go:169 msg="getting blob" blob=01DNJYRW0PMCDRQW0NJ6YBZQ6F/meta.json offset=0 length=0 level=debug ts=2019-09-26T18:52:32.0940917Z caller=azure.go:229 msg="check if blob exists" blob=01DNJYRW0PMCDRQW0NJ6YBZQ6F/meta.json level=debug ts=2019-09-26T18:52:32.094311001Z caller=azure.go:169 msg="getting blob" blob=01DNDSVHEHJKWMBHGFBEM0VBWZ/index.cache.json offset=0 length=0 level=debug ts=2019-09-26T18:52:32.094375501Z caller=azure.go:229 msg="check if blob exists" blob=01DNDSVHEHJKWMBHGFBEM0VBWZ/index.cache.json level=debug ts=2019-09-26T18:52:32.382381604Z caller=azure.go:169 msg="getting blob" blob=01DNJYXKBWE6E3491G07X15PYF/meta.json offset=0 length=0 level=debug ts=2019-09-26T18:52:32.382435204Z caller=azure.go:229 msg="check if blob exists" blob=01DNJYXKBWE6E3491G07X15PYF/meta.json level=debug ts=2019-09-26T18:52:32.382617705Z caller=azure.go:169 msg="getting blob" blob=01DNJYRW0PMCDRQW0NJ6YBZQ6F/index.cache.json offset=0 length=0 level=debug ts=2019-09-26T18:52:32.382658805Z caller=azure.go:229 msg="check if blob exists" blob=01DNJYRW0PMCDRQW0NJ6YBZQ6F/index.cache.json level=debug ts=2019-09-26T18:52:32.570358419Z caller=azure.go:169 msg="getting blob" blob=01DNKT6J95BXE7H7EF1N969063/meta.json offset=0 length=0 level=debug ts=2019-09-26T18:52:32.570807521Z caller=azure.go:229 msg="check if blob exists" blob=01DNKT6J95BXE7H7EF1N969063/meta.json level=debug ts=2019-09-26T18:52:32.57065042Z caller=azure.go:169 msg="getting blob" blob=01DNJYXKBWE6E3491G07X15PYF/index.cache.json offset=0 length=0 level=debug ts=2019-09-26T18:52:32.570917222Z caller=azure.go:229 msg="check if blob exists" blob=01DNJYXKBWE6E3491G07X15PYF/index.cache.json level=debug ts=2019-09-26T18:52:32.604417885Z caller=azure.go:169 msg="getting blob" blob=01DNMNNG41J2SFYER3HVZYKNEW/meta.json offset=0 length=0 level=debug ts=2019-09-26T18:52:32.604825587Z caller=azure.go:229 msg="check if blob exists" blob=01DNMNNG41J2SFYER3HVZYKNEW/meta.json level=debug ts=2019-09-26T18:52:32.604538586Z caller=azure.go:169 msg="getting blob" blob=01DNKT6J95BXE7H7EF1N969063/index.cache.json offset=0 length=0 level=debug ts=2019-09-26T18:52:32.60550699Z caller=azure.go:229 msg="check if blob exists" blob=01DNKT6J95BXE7H7EF1N969063/index.cache.json level=debug ts=2019-09-26T18:52:32.683258169Z caller=azure.go:169 msg="getting blob" blob=01DNNH4DA8GQCVY23H4YQP8468/meta.json offset=0 length=0 level=debug ts=2019-09-26T18:52:32.683302469Z caller=azure.go:229 msg="check if blob exists" blob=01DNNH4DA8GQCVY23H4YQP8468/meta.json level=debug ts=2019-09-26T18:52:32.683382269Z caller=azure.go:169 msg="getting blob" blob=01DNMNNG41J2SFYER3HVZYKNEW/index.cache.json offset=0 length=0 level=debug ts=2019-09-26T18:52:32.68342217Z caller=azure.go:229 msg="check if blob exists" blob=01DNMNNG41J2SFYER3HVZYKNEW/index.cache.json level=debug ts=2019-09-26T18:52:32.73474142Z caller=azure.go:169 msg="getting blob" blob=01DNPCK9WXMKM7RX0P5RYGH26H/meta.json offset=0 length=0 level=debug ts=2019-09-26T18:52:32.734976721Z caller=azure.go:169 msg="getting blob" blob=01DNNH4DA8GQCVY23H4YQP8468/index.cache.json offset=0 length=0 level=debug ts=2019-09-26T18:52:32.735014121Z caller=azure.go:229 msg="check if blob exists" blob=01DNNH4DA8GQCVY23H4YQP8468/index.cache.json level=debug ts=2019-09-26T18:52:32.734991521Z caller=azure.go:229 msg="check if blob exists" blob=01DNPCK9WXMKM7RX0P5RYGH26H/meta.json level=debug ts=2019-09-26T18:52:33.078947496Z caller=azure.go:169 msg="getting blob" blob=01DNQ633CRJ2YR6YKH75NV24N9/meta.json offset=0 length=0 level=debug ts=2019-09-26T18:52:33.079028596Z caller=azure.go:229 msg="check if blob exists" blob=01DNQ633CRJ2YR6YKH75NV24N9/meta.json level=debug ts=2019-09-26T18:52:33.079379898Z caller=azure.go:169 msg="getting blob" blob=01DNPCK9WXMKM7RX0P5RYGH26H/index.cache.json offset=0 length=0 level=debug ts=2019-09-26T18:52:33.079404498Z caller=azure.go:229 msg="check if blob exists" blob=01DNPCK9WXMKM7RX0P5RYGH26H/index.cache.json level=debug ts=2019-09-26T18:52:33.280552778Z caller=azure.go:169 msg="getting blob" blob=01DNQ6YJ9Q8RXZ7ZQG3JJG7XGX/meta.json offset=0 length=0 level=debug ts=2019-09-26T18:52:33.280603678Z caller=azure.go:229 msg="check if blob exists" blob=01DNQ6YJ9Q8RXZ7ZQG3JJG7XGX/meta.json level=debug ts=2019-09-26T18:52:33.28086568Z caller=azure.go:169 msg="getting blob" blob=01DNQ633CRJ2YR6YKH75NV24N9/index.cache.json offset=0 length=0 level=debug ts=2019-09-26T18:52:33.28088328Z caller=azure.go:229 msg="check if blob exists" blob=01DNQ633CRJ2YR6YKH75NV24N9/index.cache.json level=debug ts=2019-09-26T18:52:33.284409997Z caller=azure.go:169 msg="getting blob" blob=01DNQ633CRJ2YR6YKH75NV24N9/index offset=0 length=0 level=debug ts=2019-09-26T18:52:33.284434097Z caller=azure.go:229 msg="check if blob exists" blob=01DNQ633CRJ2YR6YKH75NV24N9/index level=debug ts=2019-09-26T18:52:33.465852281Z caller=azure.go:169 msg="getting blob" blob=01DNQ7T16QVXT5FF46JT25ZRAS/meta.json offset=0 length=0 level=debug ts=2019-09-26T18:52:33.465906581Z caller=azure.go:229 msg="check if blob exists" blob=01DNQ7T16QVXT5FF46JT25ZRAS/meta.json level=debug ts=2019-09-26T18:52:33.465935981Z caller=azure.go:169 msg="getting blob" blob=01DNQ6YJ9Q8RXZ7ZQG3JJG7XGX/index.cache.json offset=0 length=0 level=debug ts=2019-09-26T18:52:33.465957581Z caller=azure.go:229 msg="check if blob exists" blob=01DNQ6YJ9Q8RXZ7ZQG3JJG7XGX/index.cache.json level=debug ts=2019-09-26T18:52:33.469651199Z caller=azure.go:169 msg="getting blob" blob=01DNQ6YJ9Q8RXZ7ZQG3JJG7XGX/index offset=0 length=0 level=debug ts=2019-09-26T18:52:33.469676799Z caller=azure.go:229 msg="check if blob exists" blob=01DNQ6YJ9Q8RXZ7ZQG3JJG7XGX/index level=debug ts=2019-09-26T18:52:33.629790279Z caller=azure.go:169 msg="getting blob" blob=01DNQ828623R6EKK99A09RJECB/meta.json offset=0 length=0 level=debug ts=2019-09-26T18:52:33.629840979Z caller=azure.go:229 msg="check if blob exists" blob=01DNQ828623R6EKK99A09RJECB/meta.json level=debug ts=2019-09-26T18:52:33.62986628Z caller=azure.go:169 msg="getting blob" blob=01DNQ7T16QVXT5FF46JT25ZRAS/index.cache.json offset=0 length=0 level=debug ts=2019-09-26T18:52:33.62988788Z caller=azure.go:229 msg="check if blob exists" blob=01DNQ7T16QVXT5FF46JT25ZRAS/index.cache.json level=debug ts=2019-09-26T18:52:33.635747908Z caller=azure.go:169 msg="getting blob" blob=01DNQ7T16QVXT5FF46JT25ZRAS/index offset=0 length=0 level=debug ts=2019-09-26T18:52:33.635798208Z caller=azure.go:229 msg="check if blob exists" blob=01DNQ7T16QVXT5FF46JT25ZRAS/index level=debug ts=2019-09-26T18:52:33.68126933Z caller=azure.go:169 msg="getting blob" blob=01DNQ8NG3QYNVQ22QC3T0Z31KE/meta.json offset=0 length=0 level=debug ts=2019-09-26T18:52:33.68131513Z caller=azure.go:229 msg="check if blob exists" blob=01DNQ8NG3QYNVQ22QC3T0Z31KE/meta.json level=debug ts=2019-09-26T18:52:33.681530231Z caller=azure.go:169 msg="getting blob" blob=01DNQ828623R6EKK99A09RJECB/index.cache.json offset=0 length=0 level=debug ts=2019-09-26T18:52:33.681561731Z caller=azure.go:229 msg="check if blob exists" blob=01DNQ828623R6EKK99A09RJECB/index.cache.json level=debug ts=2019-09-26T18:52:33.746002645Z caller=azure.go:169 msg="getting blob" blob=01DNQ9GZ0Q3S1D1FJJ0H1YJ0HT/meta.json offset=0 length=0 level=debug ts=2019-09-26T18:52:33.746418147Z caller=azure.go:229 msg="check if blob exists" blob=01DNQ9GZ0Q3S1D1FJJ0H1YJ0HT/meta.json level=debug ts=2019-09-26T18:52:33.746066146Z caller=azure.go:169 msg="getting blob" blob=01DNQ8NG3QYNVQ22QC3T0Z31KE/index.cache.json offset=0 length=0 level=debug ts=2019-09-26T18:52:33.746584848Z caller=azure.go:229 msg="check if blob exists" blob=01DNQ8NG3QYNVQ22QC3T0Z31KE/index.cache.json level=debug ts=2019-09-26T18:52:33.760941418Z caller=azure.go:169 msg="getting blob" blob=01DNQ8NG3QYNVQ22QC3T0Z31KE/index offset=0 length=0 level=debug ts=2019-09-26T18:52:33.760981918Z caller=azure.go:229 msg="check if blob exists" blob=01DNQ8NG3QYNVQ22QC3T0Z31KE/index level=debug ts=2019-09-26T18:52:33.841591011Z caller=azure.go:169 msg="getting blob" blob=01DNQACDYQSNE1KGSP6R7XNCD1/meta.json offset=0 length=0 level=debug ts=2019-09-26T18:52:33.841662911Z caller=azure.go:229 msg="check if blob exists" blob=01DNQACDYQSNE1KGSP6R7XNCD1/meta.json level=debug ts=2019-09-26T18:52:33.841986713Z caller=azure.go:169 msg="getting blob" blob=01DNQ9GZ0Q3S1D1FJJ0H1YJ0HT/index.cache.json offset=0 length=0 level=debug ts=2019-09-26T18:52:33.842022013Z caller=azure.go:229 msg="check if blob exists" blob=01DNQ9GZ0Q3S1D1FJJ0H1YJ0HT/index.cache.json level=debug ts=2019-09-26T18:52:33.855224777Z caller=azure.go:169 msg="getting blob" blob=01DNQ9GZ0Q3S1D1FJJ0H1YJ0HT/index offset=0 length=0 level=debug ts=2019-09-26T18:52:33.855280278Z caller=azure.go:229 msg="check if blob exists" blob=01DNQ9GZ0Q3S1D1FJJ0H1YJ0HT/index level=debug ts=2019-09-26T18:52:33.945476117Z caller=azure.go:169 msg="getting blob" blob=01DNQACDYQSNE1KGSP6R7XNCD1/index.cache.json offset=0 length=0 level=debug ts=2019-09-26T18:52:33.945600218Z caller=azure.go:229 msg="check if blob exists" blob=01DNQACDYQSNE1KGSP6R7XNCD1/index.cache.json level=debug ts=2019-09-26T18:52:33.945540817Z caller=azure.go:169 msg="getting blob" blob=01DNQB7WTRXK03AVWS10BBQ9YT/meta.json offset=0 length=0 level=debug ts=2019-09-26T18:52:33.945809319Z caller=azure.go:229 msg="check if blob exists" blob=01DNQB7WTRXK03AVWS10BBQ9YT/meta.json level=debug ts=2019-09-26T18:52:33.951138445Z caller=azure.go:169 msg="getting blob" blob=01DNQACDYQSNE1KGSP6R7XNCD1/index offset=0 length=0 level=debug ts=2019-09-26T18:52:33.951165745Z caller=azure.go:229 msg="check if blob exists" blob=01DNQACDYQSNE1KGSP6R7XNCD1/index level=debug ts=2019-09-26T18:52:34.05491555Z caller=azure.go:169 msg="getting blob" blob=01DNQC3BQQQ4NXH5R4570DN23X/meta.json offset=0 length=0 level=debug ts=2019-09-26T18:52:34.05496435Z caller=azure.go:229 msg="check if blob exists" blob=01DNQC3BQQQ4NXH5R4570DN23X/meta.json level=debug ts=2019-09-26T18:52:34.05499385Z caller=azure.go:169 msg="getting blob" blob=01DNQB7WTRXK03AVWS10BBQ9YT/index.cache.json offset=0 length=0 level=debug ts=2019-09-26T18:52:34.055016751Z caller=azure.go:229 msg="check if blob exists" blob=01DNQB7WTRXK03AVWS10BBQ9YT/index.cache.json level=debug ts=2019-09-26T18:52:34.058430467Z caller=azure.go:169 msg="getting blob" blob=01DNQB7WTRXK03AVWS10BBQ9YT/index offset=0 length=0 level=debug ts=2019-09-26T18:52:34.058460667Z caller=azure.go:229 msg="check if blob exists" blob=01DNQB7WTRXK03AVWS10BBQ9YT/index level=debug ts=2019-09-26T18:52:34.166973596Z caller=azure.go:169 msg="getting blob" blob=01DNQCYTMQBY6BSF9DMRY79D2Q/meta.json offset=0 length=0 level=debug ts=2019-09-26T18:52:34.167025896Z caller=azure.go:229 msg="check if blob exists" blob=01DNQCYTMQBY6BSF9DMRY79D2Q/meta.json level=debug ts=2019-09-26T18:52:34.167026896Z caller=azure.go:169 msg="getting blob" blob=01DNQC3BQQQ4NXH5R4570DN23X/index.cache.json offset=0 length=0 level=debug ts=2019-09-26T18:52:34.167049396Z caller=azure.go:229 msg="check if blob exists" blob=01DNQC3BQQQ4NXH5R4570DN23X/index.cache.json level=debug ts=2019-09-26T18:52:34.172323722Z caller=azure.go:169 msg="getting blob" blob=01DNQC3BQQQ4NXH5R4570DN23X/index offset=0 length=0 level=debug ts=2019-09-26T18:52:34.172351722Z caller=azure.go:229 msg="check if blob exists" blob=01DNQC3BQQQ4NXH5R4570DN23X/index level=debug ts=2019-09-26T18:52:34.28679478Z caller=azure.go:169 msg="getting blob" blob=01DNQDT9HQTG98KYWWS329K7EB/meta.json offset=0 length=0 level=debug ts=2019-09-26T18:52:34.286952081Z caller=azure.go:229 msg="check if blob exists" blob=01DNQDT9HQTG98KYWWS329K7EB/meta.json level=debug ts=2019-09-26T18:52:34.28684508Z caller=azure.go:169 msg="getting blob" blob=01DNQCYTMQBY6BSF9DMRY79D2Q/index.cache.json offset=0 length=0 level=debug ts=2019-09-26T18:52:34.287096381Z caller=azure.go:229 msg="check if blob exists" blob=01DNQCYTMQBY6BSF9DMRY79D2Q/index.cache.json level=debug ts=2019-09-26T18:52:34.290371997Z caller=azure.go:169 msg="getting blob" blob=01DNQCYTMQBY6BSF9DMRY79D2Q/index offset=0 length=0 level=debug ts=2019-09-26T18:52:34.290398897Z caller=azure.go:229 msg="check if blob exists" blob=01DNQCYTMQBY6BSF9DMRY79D2Q/index level=debug ts=2019-09-26T18:52:34.449419672Z caller=azure.go:169 msg="getting blob" blob=01DNQDT9HQTG98KYWWS329K7EB/index.cache.json offset=0 length=0 level=debug ts=2019-09-26T18:52:34.449470472Z caller=azure.go:229 msg="check if blob exists" blob=01DNQDT9HQTG98KYWWS329K7EB/index.cache.json level=debug ts=2019-09-26T18:52:34.453709593Z caller=azure.go:169 msg="getting blob" blob=01DNQDT9HQTG98KYWWS329K7EB/index offset=0 length=0 level=debug ts=2019-09-26T18:52:34.453737493Z caller=azure.go:229 msg="check if blob exists" blob=01DNQDT9HQTG98KYWWS329K7EB/index level=debug ts=2019-09-26T18:52:34.667613335Z caller=store.go:166 msg="bucket store ready" init_duration=5.606308898s level=info ts=2019-09-26T18:52:34.668000037Z caller=main.go:288 msg="disabled TLS, key and cert must be set to enable" level=info ts=2019-09-26T18:52:34.668081038Z caller=store.go:209 msg="starting store node" level=info ts=2019-09-26T18:52:34.668351239Z caller=store.go:199 msg="Listening for StoreAPI gRPC" address=0.0.0.0:10901 level=info ts=2019-09-26T18:52:34.668401339Z caller=main.go:341 msg="Listening for metrics" address=0.0.0.0:10902 level=debug ts=2019-09-26T18:52:34.694816768Z caller=azure.go:148 msg="requesting next iteration of listing blobs" last_entries=19 iteration=1 level=debug ts=2019-09-26T18:55:01.378395494Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=20 MaximumResolution=1440000 mint=1561740900657 maxt=1569524100657 lset="{environment=\"Prod\",monitor=\"srs-us4\"}" spans="Range: 1566432000000-1569369600000 Resolution: 300000\nRange: 1569369600000-1569522600000 Resolution: 0" level=debug ts=2019-09-26T18:55:01.378596595Z caller=azure.go:169 msg="getting blob" blob=01DNQDT9HQTG98KYWWS329K7EB/index offset=3517572 length=294240 level=debug ts=2019-09-26T18:55:01.378643995Z caller=azure.go:229 msg="check if blob exists" blob=01DNQDT9HQTG98KYWWS329K7EB/index level=debug ts=2019-09-26T18:55:01.378675495Z caller=azure.go:169 msg="getting blob" blob=01DN3H9NA1KGS4TKA4MYHY09MR/index offset=21400932 length=704 level=debug ts=2019-09-26T18:55:01.378709095Z caller=azure.go:229 msg="check if blob exists" blob=01DN3H9NA1KGS4TKA4MYHY09MR/index level=debug ts=2019-09-26T18:55:01.378864196Z caller=azure.go:169 msg="getting blob" blob=01DN8N4PJSNE9F07XJ78RZWN3G/index offset=6200360 length=332828 level=debug ts=2019-09-26T18:55:01.378897096Z caller=azure.go:229 msg="check if blob exists" blob=01DN8N4PJSNE9F07XJ78RZWN3G/index level=debug ts=2019-09-26T18:55:01.379015997Z caller=azure.go:169 msg="getting blob" blob=01DNDSVHEHJKWMBHGFBEM0VBWZ/index offset=6152836 length=328984 level=debug ts=2019-09-26T18:55:01.379045997Z caller=azure.go:229 msg="check if blob exists" blob=01DNDSVHEHJKWMBHGFBEM0VBWZ/index level=debug ts=2019-09-26T18:55:01.379386699Z caller=azure.go:169 msg="getting blob" blob=01DNJYXKBWE6E3491G07X15PYF/index offset=10239408 length=844 level=debug ts=2019-09-26T18:55:01.379424299Z caller=azure.go:229 msg="check if blob exists" blob=01DNJYXKBWE6E3491G07X15PYF/index level=debug ts=2019-09-26T18:55:01.379472099Z caller=azure.go:169 msg="getting blob" blob=01DNNH4DA8GQCVY23H4YQP8468/index offset=13732932 length=295148 level=debug ts=2019-09-26T18:55:01.379508999Z caller=azure.go:229 msg="check if blob exists" blob=01DNNH4DA8GQCVY23H4YQP8468/index level=debug ts=2019-09-26T18:55:01.3795966Z caller=azure.go:169 msg="getting blob" blob=01DNKT6J95BXE7H7EF1N969063/index offset=13843324 length=308720 level=debug ts=2019-09-26T18:55:01.3796285Z caller=azure.go:229 msg="check if blob exists" blob=01DNKT6J95BXE7H7EF1N969063/index level=debug ts=2019-09-26T18:55:01.3796619Z caller=azure.go:169 msg="getting blob" blob=01DNPCK9WXMKM7RX0P5RYGH26H/index offset=13978688 length=331224 level=debug ts=2019-09-26T18:55:01.3796915Z caller=azure.go:229 msg="check if blob exists" blob=01DNPCK9WXMKM7RX0P5RYGH26H/index level=debug ts=2019-09-26T18:55:01.3797304Z caller=azure.go:169 msg="getting blob" blob=01DKZFRCA5YH02S7TH5FM253PM/index offset=21362256 length=64 level=debug ts=2019-09-26T18:55:01.3797539Z caller=azure.go:229 msg="check if blob exists" blob=01DKZFRCA5YH02S7TH5FM253PM/index level=debug ts=2019-09-26T18:55:01.379816801Z caller=azure.go:169 msg="getting blob" blob=01DNQ828623R6EKK99A09RJECB/index offset=15047992 length=450472 level=debug ts=2019-09-26T18:55:01.379841101Z caller=azure.go:229 msg="check if blob exists" blob=01DNQ828623R6EKK99A09RJECB/index level=debug ts=2019-09-26T18:55:01.379864301Z caller=azure.go:169 msg="getting blob" blob=01DN3H9NA1KGS4TKA4MYHY09MR/index offset=20719072 length=64 level=debug ts=2019-09-26T18:55:01.379885101Z caller=azure.go:229 msg="check if blob exists" blob=01DN3H9NA1KGS4TKA4MYHY09MR/index level=debug ts=2019-09-26T18:55:01.379951501Z caller=azure.go:169 msg="getting blob" blob=01DNQ633CRJ2YR6YKH75NV24N9/index offset=3506704 length=292752 level=debug ts=2019-09-26T18:55:01.379981901Z caller=azure.go:229 msg="check if blob exists" blob=01DNQ633CRJ2YR6YKH75NV24N9/index level=debug ts=2019-09-26T18:55:01.379972501Z caller=azure.go:169 msg="getting blob" blob=01DNJYXKBWE6E3491G07X15PYF/index offset=9581284 length=76 level=debug ts=2019-09-26T18:55:01.379995601Z caller=azure.go:229 msg="check if blob exists" blob=01DNJYXKBWE6E3491G07X15PYF/index level=debug ts=2019-09-26T18:55:01.380095802Z caller=azure.go:169 msg="getting blob" blob=01DNQ6YJ9Q8RXZ7ZQG3JJG7XGX/index offset=3505424 length=292536 level=debug ts=2019-09-26T18:55:01.380126202Z caller=azure.go:229 msg="check if blob exists" blob=01DNQ6YJ9Q8RXZ7ZQG3JJG7XGX/index level=debug ts=2019-09-26T18:55:01.380139002Z caller=azure.go:169 msg="getting blob" blob=01DNQ7T16QVXT5FF46JT25ZRAS/index offset=3502796 length=292076 level=debug ts=2019-09-26T18:55:01.380162002Z caller=azure.go:229 msg="check if blob exists" blob=01DNQ7T16QVXT5FF46JT25ZRAS/index level=debug ts=2019-09-26T18:55:01.380264503Z caller=azure.go:169 msg="getting blob" blob=01DNQC3BQQQ4NXH5R4570DN23X/index offset=3513036 length=293840 level=debug ts=2019-09-26T18:55:01.380296303Z caller=azure.go:229 msg="check if blob exists" blob=01DNQC3BQQQ4NXH5R4570DN23X/index level=debug ts=2019-09-26T18:55:01.380301403Z caller=azure.go:169 msg="getting blob" blob=01DNQ8NG3QYNVQ22QC3T0Z31KE/index offset=3500852 length=291940 level=debug ts=2019-09-26T18:55:01.380335103Z caller=azure.go:229 msg="check if blob exists" blob=01DNQ8NG3QYNVQ22QC3T0Z31KE/index level=debug ts=2019-09-26T18:55:01.380407903Z caller=azure.go:169 msg="getting blob" blob=01DNQCYTMQBY6BSF9DMRY79D2Q/index offset=3515244 length=293988 level=debug ts=2019-09-26T18:55:01.380438504Z caller=azure.go:229 msg="check if blob exists" blob=01DNQCYTMQBY6BSF9DMRY79D2Q/index level=debug ts=2019-09-26T18:55:01.380449904Z caller=azure.go:169 msg="getting blob" blob=01DNQ9GZ0Q3S1D1FJJ0H1YJ0HT/index offset=3522288 length=294932 level=debug ts=2019-09-26T18:55:01.380473804Z caller=azure.go:229 msg="check if blob exists" blob=01DNQ9GZ0Q3S1D1FJJ0H1YJ0HT/index level=debug ts=2019-09-26T18:55:01.380525904Z caller=azure.go:169 msg="getting blob" blob=01DNMNNG41J2SFYER3HVZYKNEW/index offset=13748800 length=298076 level=debug ts=2019-09-26T18:55:01.380551504Z caller=azure.go:229 msg="check if blob exists" blob=01DNMNNG41J2SFYER3HVZYKNEW/index level=debug ts=2019-09-26T18:55:01.380576804Z caller=azure.go:169 msg="getting blob" blob=01DNQACDYQSNE1KGSP6R7XNCD1/index offset=3530744 length=296188 level=debug ts=2019-09-26T18:55:01.380608004Z caller=azure.go:229 msg="check if blob exists" blob=01DNQACDYQSNE1KGSP6R7XNCD1/index level=debug ts=2019-09-26T18:55:01.378619395Z caller=azure.go:169 msg="getting blob" blob=01DKZFRCA5YH02S7TH5FM253PM/index offset=22160124 length=704 level=debug ts=2019-09-26T18:55:01.380636904Z caller=azure.go:229 msg="check if blob exists" blob=01DKZFRCA5YH02S7TH5FM253PM/index level=debug ts=2019-09-26T18:55:01.380101602Z caller=azure.go:169 msg="getting blob" blob=01DNQB7WTRXK03AVWS10BBQ9YT/index offset=3515276 length=293760 level=debug ts=2019-09-26T18:55:01.380734405Z caller=azure.go:229 msg="check if blob exists" blob=01DNQB7WTRXK03AVWS10BBQ9YT/index level=debug ts=2019-09-26T18:55:01.567807883Z caller=azure.go:169 msg="getting blob" blob=01DNQC3BQQQ4NXH5R4570DN23X/index offset=2518352 length=65776 level=debug ts=2019-09-26T18:55:01.567878283Z caller=azure.go:229 msg="check if blob exists" blob=01DNQC3BQQQ4NXH5R4570DN23X/index level=debug ts=2019-09-26T18:55:01.568964889Z caller=azure.go:169 msg="getting blob" blob=01DNDSVHEHJKWMBHGFBEM0VBWZ/index offset=4674288 length=65920 level=debug ts=2019-09-26T18:55:01.568996389Z caller=azure.go:169 msg="getting blob" blob=01DNQ6YJ9Q8RXZ7ZQG3JJG7XGX/index offset=2511392 length=65776 level=debug ts=2019-09-26T18:55:01.569026489Z caller=azure.go:229 msg="check if blob exists" blob=01DNQ6YJ9Q8RXZ7ZQG3JJG7XGX/index level=debug ts=2019-09-26T18:55:01.569008989Z caller=azure.go:229 msg="check if blob exists" blob=01DNDSVHEHJKWMBHGFBEM0VBWZ/index level=debug ts=2019-09-26T18:55:01.574594315Z caller=azure.go:169 msg="getting blob" blob=01DNMNNG41J2SFYER3HVZYKNEW/index offset=10815616 length=66896 level=debug ts=2019-09-26T18:55:01.574632315Z caller=azure.go:229 msg="check if blob exists" blob=01DNMNNG41J2SFYER3HVZYKNEW/index level=debug ts=2019-09-26T18:55:01.575123017Z caller=azure.go:169 msg="getting blob" blob=01DNQ9GZ0Q3S1D1FJJ0H1YJ0HT/index offset=2526784 length=65776 level=debug ts=2019-09-26T18:55:01.575154118Z caller=azure.go:229 msg="check if blob exists" blob=01DNQ9GZ0Q3S1D1FJJ0H1YJ0HT/index level=debug ts=2019-09-26T18:55:01.57781773Z caller=azure.go:169 msg="getting blob" blob=01DNKT6J95BXE7H7EF1N969063/index offset=10917760 length=66896 level=debug ts=2019-09-26T18:55:01.57789013Z caller=azure.go:229 msg="check if blob exists" blob=01DNKT6J95BXE7H7EF1N969063/index level=debug ts=2019-09-26T18:55:01.578476633Z caller=azure.go:169 msg="getting blob" blob=01DNQ7T16QVXT5FF46JT25ZRAS/index offset=2508976 length=65776 level=debug ts=2019-09-26T18:55:01.578509733Z caller=azure.go:229 msg="check if blob exists" blob=01DNQ7T16QVXT5FF46JT25ZRAS/index level=debug ts=2019-09-26T18:55:01.582180451Z caller=azure.go:169 msg="getting blob" blob=01DNQ633CRJ2YR6YKH75NV24N9/index offset=2512528 length=65776 level=debug ts=2019-09-26T18:55:01.582218451Z caller=azure.go:229 msg="check if blob exists" blob=01DNQ633CRJ2YR6YKH75NV24N9/index level=debug ts=2019-09-26T18:55:01.591780396Z caller=azure.go:169 msg="getting blob" blob=01DNDSVHEHJKWMBHGFBEM0VBWZ/chunks/000001 offset=125714288 length=75484 level=debug ts=2019-09-26T18:55:01.591817396Z caller=azure.go:229 msg="check if blob exists" blob=01DNDSVHEHJKWMBHGFBEM0VBWZ/chunks/000001 level=debug ts=2019-09-26T18:55:01.593895006Z caller=azure.go:169 msg="getting blob" blob=01DNQ828623R6EKK99A09RJECB/index offset=11623504 length=66896 level=debug ts=2019-09-26T18:55:01.593927606Z caller=azure.go:229 msg="check if blob exists" blob=01DNQ828623R6EKK99A09RJECB/index level=debug ts=2019-09-26T18:55:01.597463222Z caller=azure.go:169 msg="getting blob" blob=01DKZFRCA5YH02S7TH5FM253PM/index offset=17174640 length=66752 level=debug ts=2019-09-26T18:55:01.597498823Z caller=azure.go:229 msg="check if blob exists" blob=01DKZFRCA5YH02S7TH5FM253PM/index level=debug ts=2019-09-26T18:55:01.599486332Z caller=azure.go:169 msg="getting blob" blob=01DN8N4PJSNE9F07XJ78RZWN3G/index offset=4712160 length=65920 level=debug ts=2019-09-26T18:55:01.599521932Z caller=azure.go:229 msg="check if blob exists" blob=01DN8N4PJSNE9F07XJ78RZWN3G/index level=debug ts=2019-09-26T18:55:01.600145235Z caller=azure.go:169 msg="getting blob" blob=01DNQ6YJ9Q8RXZ7ZQG3JJG7XGX/chunks/000001 offset=4559042 length=17634 level=debug ts=2019-09-26T18:55:01.600209135Z caller=azure.go:229 msg="check if blob exists" blob=01DNQ6YJ9Q8RXZ7ZQG3JJG7XGX/chunks/000001 level=debug ts=2019-09-26T18:55:01.603951753Z caller=azure.go:169 msg="getting blob" blob=01DNQDT9HQTG98KYWWS329K7EB/index offset=2522320 length=65776 level=debug ts=2019-09-26T18:55:01.603988753Z caller=azure.go:229 msg="check if blob exists" blob=01DNQDT9HQTG98KYWWS329K7EB/index level=debug ts=2019-09-26T18:55:01.609394278Z caller=azure.go:169 msg="getting blob" blob=01DNQ8NG3QYNVQ22QC3T0Z31KE/index offset=2507296 length=65776 level=debug ts=2019-09-26T18:55:01.609431279Z caller=azure.go:229 msg="check if blob exists" blob=01DNQ8NG3QYNVQ22QC3T0Z31KE/index level=debug ts=2019-09-26T18:55:01.609522279Z caller=azure.go:169 msg="getting blob" blob=01DN3H9NA1KGS4TKA4MYHY09MR/index offset=16740544 length=66752 level=debug ts=2019-09-26T18:55:01.609552079Z caller=azure.go:229 msg="check if blob exists" blob=01DN3H9NA1KGS4TKA4MYHY09MR/index level=debug ts=2019-09-26T18:55:01.611227187Z caller=azure.go:169 msg="getting blob" blob=01DNJYXKBWE6E3491G07X15PYF/index offset=6509648 length=66016 level=debug ts=2019-09-26T18:55:01.611263087Z caller=azure.go:229 msg="check if blob exists" blob=01DNJYXKBWE6E3491G07X15PYF/index level=debug ts=2019-09-26T18:55:01.612914095Z caller=azure.go:169 msg="getting blob" blob=01DNPCK9WXMKM7RX0P5RYGH26H/index offset=11035264 length=66896 level=debug ts=2019-09-26T18:55:01.612983395Z caller=azure.go:229 msg="check if blob exists" blob=01DNPCK9WXMKM7RX0P5RYGH26H/index level=debug ts=2019-09-26T18:55:01.613550398Z caller=azure.go:169 msg="getting blob" blob=01DNQB7WTRXK03AVWS10BBQ9YT/index offset=2520272 length=65776 level=debug ts=2019-09-26T18:55:01.613591698Z caller=azure.go:229 msg="check if blob exists" blob=01DNQB7WTRXK03AVWS10BBQ9YT/index level=debug ts=2019-09-26T18:55:01.613809899Z caller=azure.go:169 msg="getting blob" blob=01DNQACDYQSNE1KGSP6R7XNCD1/index offset=2534352 length=65776 level=debug ts=2019-09-26T18:55:01.613839799Z caller=azure.go:229 msg="check if blob exists" blob=01DNQACDYQSNE1KGSP6R7XNCD1/index level=debug ts=2019-09-26T18:55:01.62040633Z caller=azure.go:169 msg="getting blob" blob=01DNQ633CRJ2YR6YKH75NV24N9/chunks/000001 offset=4545685 length=17642 level=debug ts=2019-09-26T18:55:01.62044063Z caller=azure.go:229 msg="check if blob exists" blob=01DNQ633CRJ2YR6YKH75NV24N9/chunks/000001 level=debug ts=2019-09-26T18:55:01.621964937Z caller=azure.go:169 msg="getting blob" blob=01DNKT6J95BXE7H7EF1N969063/chunks/000001 offset=143840147 length=70012 level=debug ts=2019-09-26T18:55:01.621999538Z caller=azure.go:229 msg="check if blob exists" blob=01DNKT6J95BXE7H7EF1N969063/chunks/000001 level=debug ts=2019-09-26T18:55:01.629188471Z caller=azure.go:169 msg="getting blob" blob=01DNQ828623R6EKK99A09RJECB/chunks/000001 offset=144753016 length=75424 level=debug ts=2019-09-26T18:55:01.629261572Z caller=azure.go:229 msg="check if blob exists" blob=01DNQ828623R6EKK99A09RJECB/chunks/000001 level=debug ts=2019-09-26T18:55:01.63114808Z caller=azure.go:169 msg="getting blob" blob=01DNQ9GZ0Q3S1D1FJJ0H1YJ0HT/chunks/000001 offset=4819539 length=17699 level=debug ts=2019-09-26T18:55:01.631181381Z caller=azure.go:229 msg="check if blob exists" blob=01DNQ9GZ0Q3S1D1FJJ0H1YJ0HT/chunks/000001 level=debug ts=2019-09-26T18:55:01.634051394Z caller=azure.go:169 msg="getting blob" blob=01DNQC3BQQQ4NXH5R4570DN23X/chunks/000001 offset=4668287 length=17645 level=debug ts=2019-09-26T18:55:01.634085294Z caller=azure.go:229 msg="check if blob exists" blob=01DNQC3BQQQ4NXH5R4570DN23X/chunks/000001 level=debug ts=2019-09-26T18:55:01.635601501Z caller=azure.go:169 msg="getting blob" blob=01DNQCYTMQBY6BSF9DMRY79D2Q/index offset=2520272 length=65776 level=debug ts=2019-09-26T18:55:01.635640802Z caller=azure.go:229 msg="check if blob exists" blob=01DNQCYTMQBY6BSF9DMRY79D2Q/index level=debug ts=2019-09-26T18:55:01.637844012Z caller=azure.go:169 msg="getting blob" blob=01DNNH4DA8GQCVY23H4YQP8468/index offset=10804960 length=66896 level=debug ts=2019-09-26T18:55:01.637877412Z caller=azure.go:229 msg="check if blob exists" blob=01DNNH4DA8GQCVY23H4YQP8468/index level=debug ts=2019-09-26T18:55:01.64588325Z caller=azure.go:169 msg="getting blob" blob=01DNQB7WTRXK03AVWS10BBQ9YT/chunks/000001 offset=4716040 length=17721 level=debug ts=2019-09-26T18:55:01.64602375Z caller=azure.go:229 msg="check if blob exists" blob=01DNQB7WTRXK03AVWS10BBQ9YT/chunks/000001 level=debug ts=2019-09-26T18:55:01.646094051Z caller=azure.go:169 msg="getting blob" blob=01DNPCK9WXMKM7RX0P5RYGH26H/chunks/000001 offset=144696142 length=73998 level=debug ts=2019-09-26T18:55:01.646155451Z caller=azure.go:229 msg="check if blob exists" blob=01DNPCK9WXMKM7RX0P5RYGH26H/chunks/000001 level=debug ts=2019-09-26T18:55:01.647739258Z caller=azure.go:169 msg="getting blob" blob=01DNMNNG41J2SFYER3HVZYKNEW/chunks/000001 offset=144038714 length=72258 level=debug ts=2019-09-26T18:55:01.647773659Z caller=azure.go:229 msg="check if blob exists" blob=01DNMNNG41J2SFYER3HVZYKNEW/chunks/000001 level=debug ts=2019-09-26T18:55:01.649273466Z caller=azure.go:169 msg="getting blob" blob=01DNJYXKBWE6E3491G07X15PYF/chunks/000001 offset=132864560 length=84053 level=debug ts=2019-09-26T18:55:01.649310666Z caller=azure.go:229 msg="check if blob exists" blob=01DNJYXKBWE6E3491G07X15PYF/chunks/000001 level=debug ts=2019-09-26T18:55:01.657760405Z caller=azure.go:169 msg="getting blob" blob=01DNQ7T16QVXT5FF46JT25ZRAS/chunks/000001 offset=4621704 length=17622 level=debug ts=2019-09-26T18:55:01.657801406Z caller=azure.go:229 msg="check if blob exists" blob=01DNQ7T16QVXT5FF46JT25ZRAS/chunks/000001 level=debug ts=2019-09-26T18:55:01.659002411Z caller=azure.go:169 msg="getting blob" blob=01DN3H9NA1KGS4TKA4MYHY09MR/chunks/000002 offset=295578691 length=439843 level=debug ts=2019-09-26T18:55:01.659039111Z caller=azure.go:229 msg="check if blob exists" blob=01DN3H9NA1KGS4TKA4MYHY09MR/chunks/000002 level=debug ts=2019-09-26T18:55:01.670513265Z caller=azure.go:169 msg="getting blob" blob=01DNQDT9HQTG98KYWWS329K7EB/chunks/000001 offset=4695586 length=17697 level=debug ts=2019-09-26T18:55:01.670546365Z caller=azure.go:229 msg="check if blob exists" blob=01DNQDT9HQTG98KYWWS329K7EB/chunks/000001 level=debug ts=2019-09-26T18:55:01.675354588Z caller=azure.go:169 msg="getting blob" blob=01DNQCYTMQBY6BSF9DMRY79D2Q/chunks/000001 offset=4682168 length=17683 level=debug ts=2019-09-26T18:55:01.675388988Z caller=azure.go:229 msg="check if blob exists" blob=01DNQCYTMQBY6BSF9DMRY79D2Q/chunks/000001 level=debug ts=2019-09-26T18:55:01.6778898Z caller=azure.go:169 msg="getting blob" blob=01DKZFRCA5YH02S7TH5FM253PM/chunks/000002 offset=286686920 length=460494 level=debug ts=2019-09-26T18:55:01.6779249Z caller=azure.go:229 msg="check if blob exists" blob=01DKZFRCA5YH02S7TH5FM253PM/chunks/000002 level=debug ts=2019-09-26T18:55:01.688853951Z caller=azure.go:169 msg="getting blob" blob=01DNNH4DA8GQCVY23H4YQP8468/chunks/000001 offset=145982535 length=75694 level=debug ts=2019-09-26T18:55:01.688894452Z caller=azure.go:229 msg="check if blob exists" blob=01DNNH4DA8GQCVY23H4YQP8468/chunks/000001 level=debug ts=2019-09-26T18:55:01.705788631Z caller=azure.go:169 msg="getting blob" blob=01DNQ8NG3QYNVQ22QC3T0Z31KE/chunks/000001 offset=4681407 length=17646 level=debug ts=2019-09-26T18:55:01.705833731Z caller=azure.go:229 msg="check if blob exists" blob=01DNQ8NG3QYNVQ22QC3T0Z31KE/chunks/000001 level=debug ts=2019-09-26T18:55:01.710663454Z caller=azure.go:169 msg="getting blob" blob=01DNQACDYQSNE1KGSP6R7XNCD1/chunks/000001 offset=4847223 length=17685 level=debug ts=2019-09-26T18:55:01.710701854Z caller=azure.go:229 msg="check if blob exists" blob=01DNQACDYQSNE1KGSP6R7XNCD1/chunks/000001 level=debug ts=2019-09-26T18:55:01.790433428Z caller=azure.go:169 msg="getting blob" blob=01DN8N4PJSNE9F07XJ78RZWN3G/chunks/000001 offset=124056629 length=76827 level=debug ts=2019-09-26T18:55:01.790497429Z caller=azure.go:229 msg="check if blob exists" blob=01DN8N4PJSNE9F07XJ78RZWN3G/chunks/000001 level=debug ts=2019-09-26T18:55:02.312749481Z caller=bucket.go:871 msg="stats query processed" stats="&{blocksQueried:20 postingsTouched:40 postingsTouchedSizeSum:17792 postingsToFetch:0 postingsFetched:40 postingsFetchedSizeSum:5284160 postingsFetchCount:23 postingsFetchDurationSum:4994210545 seriesTouched:116 seriesTouchedSizeSum:14585 seriesFetched:116 seriesFetchedSizeSum:1323600 seriesFetchCount:20 seriesFetchDurationSum:1218364121 chunksTouched:1383 chunksTouchedSizeSum:1367134 chunksFetched:1383 chunksFetchedSizeSum:1680761 chunksFetchCount:20 chunksFetchDurationSum:1695743958 getAllDuration:531936997 mergedSeriesCount:6 mergedChunksCount:1383 mergeDuration:402209188}" err=null ```

But only data till Aug 22 has been returned.

The same query with step=18000 yields 90 days (till 28Jun):

thanos-store logs with step=18000

``` level=debug ts=2019-09-26T19:12:43.939045116Z caller=bucket.go:771 msg="Blocks source resolutions" blocks=19 MaximumResolution=3600000 mint=1561741963338 maxt=1569525163338 lset="{environment=\"Prod\",monitor=\"srs-us4\"}" spans="Range: 1561593600000-1568851200000 Resolution: 3600000\nRange: 1568851200000-1569369600000 Resolution: 300000\nRange: 1569369600000-1569524400000 Resolution: 0" level=debug ts=2019-09-26T19:12:43.939245217Z caller=azure.go:169 msg="getting blob" blob=01DNQFH7BQQH376PAQH6T7P7E9/index offset=3539616 length=297072 level=debug ts=2019-09-26T19:12:43.939287117Z caller=azure.go:229 msg="check if blob exists" blob=01DNQFH7BQQH376PAQH6T7P7E9/index level=debug ts=2019-09-26T19:12:43.939384018Z caller=azure.go:169 msg="getting blob" blob=01DNJYXKBWE6E3491G07X15PYF/chunks/000001 offset=132864560 length=84053 level=debug ts=2019-09-26T19:12:43.939496818Z caller=azure.go:229 msg="check if blob exists" blob=01DNJYXKBWE6E3491G07X15PYF/chunks/000001 level=debug ts=2019-09-26T19:12:43.93971422Z caller=azure.go:169 msg="getting blob" blob=01DFF9NPD5FMJV67QQKXQES9VG/index offset=11838732 length=424 level=debug ts=2019-09-26T19:12:43.93974552Z caller=azure.go:229 msg="check if blob exists" blob=01DFF9NPD5FMJV67QQKXQES9VG/index level=debug ts=2019-09-26T19:12:43.939904921Z caller=azure.go:169 msg="getting blob" blob=01DGKB9BMYMSCC1V6X0DPB9HK7/index offset=7250428 length=480460 level=debug ts=2019-09-26T19:12:43.940066321Z caller=azure.go:169 msg="getting blob" blob=01DNKT6J95BXE7H7EF1N969063/chunks/000001 offset=143840147 length=70012 level=debug ts=2019-09-26T19:12:43.940099522Z caller=azure.go:229 msg="check if blob exists" blob=01DNKT6J95BXE7H7EF1N969063/chunks/000001 level=debug ts=2019-09-26T19:12:43.940349523Z caller=azure.go:169 msg="getting blob" blob=01DNMNNG41J2SFYER3HVZYKNEW/chunks/000001 offset=144038714 length=72258 level=debug ts=2019-09-26T19:12:43.940376623Z caller=azure.go:229 msg="check if blob exists" blob=01DNMNNG41J2SFYER3HVZYKNEW/chunks/000001 level=debug ts=2019-09-26T19:12:43.940639224Z caller=azure.go:169 msg="getting blob" blob=01DNNH4DA8GQCVY23H4YQP8468/chunks/000001 offset=145982535 length=75694 level=debug ts=2019-09-26T19:12:43.940664024Z caller=azure.go:229 msg="check if blob exists" blob=01DNNH4DA8GQCVY23H4YQP8468/chunks/000001 level=debug ts=2019-09-26T19:12:43.939981921Z caller=azure.go:229 msg="check if blob exists" blob=01DGKB9BMYMSCC1V6X0DPB9HK7/index level=debug ts=2019-09-26T19:12:43.940913926Z caller=azure.go:169 msg="getting blob" blob=01DNPCK9WXMKM7RX0P5RYGH26H/chunks/000001 offset=144696142 length=73998 level=debug ts=2019-09-26T19:12:43.940946926Z caller=azure.go:229 msg="check if blob exists" blob=01DNPCK9WXMKM7RX0P5RYGH26H/chunks/000001 level=debug ts=2019-09-26T19:12:43.941175827Z caller=azure.go:169 msg="getting blob" blob=01DHQCWD4AFE4JEFDZF0Q1CQFN/index offset=8568068 length=704 level=debug ts=2019-09-26T19:12:43.941214227Z caller=azure.go:229 msg="check if blob exists" blob=01DHQCWD4AFE4JEFDZF0Q1CQFN/index level=debug ts=2019-09-26T19:12:43.941315228Z caller=azure.go:169 msg="getting blob" blob=01DNQ828623R6EKK99A09RJECB/chunks/000001 offset=144753016 length=75424 level=debug ts=2019-09-26T19:12:43.941355528Z caller=azure.go:229 msg="check if blob exists" blob=01DNQ828623R6EKK99A09RJECB/chunks/000001 level=debug ts=2019-09-26T19:12:43.941348328Z caller=azure.go:169 msg="getting blob" blob=01DJVEEVYA6ABF57ZVEQW57NWW/index offset=10434208 length=704 level=debug ts=2019-09-26T19:12:43.941442028Z caller=azure.go:229 msg="check if blob exists" blob=01DJVEEVYA6ABF57ZVEQW57NWW/index level=debug ts=2019-09-26T19:12:43.941591729Z caller=azure.go:169 msg="getting blob" blob=01DNQEXMKJCBD6G7NZK4DJ536X/chunks/000001 offset=37458871 length=31648 level=debug ts=2019-09-26T19:12:43.941624829Z caller=azure.go:229 msg="check if blob exists" blob=01DNQEXMKJCBD6G7NZK4DJ536X/chunks/000001 level=debug ts=2019-09-26T19:12:43.94173003Z caller=azure.go:169 msg="getting blob" blob=01DKZG25YFZW3GM1PATHFS4D23/index offset=12277548 length=704 level=debug ts=2019-09-26T19:12:43.94184373Z caller=azure.go:229 msg="check if blob exists" blob=01DKZG25YFZW3GM1PATHFS4D23/index level=debug ts=2019-09-26T19:12:43.941969931Z caller=azure.go:169 msg="getting blob" blob=01DNQDT9HQTG98KYWWS329K7EB/chunks/000001 offset=4695586 length=17697 level=debug ts=2019-09-26T19:12:43.942082031Z caller=azure.go:229 msg="check if blob exists" blob=01DNQDT9HQTG98KYWWS329K7EB/chunks/000001 level=debug ts=2019-09-26T19:12:43.942237832Z caller=azure.go:169 msg="getting blob" blob=01DN3HMPA27V4QJBENN6GBVR7X/index offset=11327304 length=704 level=debug ts=2019-09-26T19:12:43.942342833Z caller=azure.go:229 msg="check if blob exists" blob=01DN3HMPA27V4QJBENN6GBVR7X/index level=debug ts=2019-09-26T19:12:43.942460333Z caller=azure.go:169 msg="getting blob" blob=01DHQCWD4AFE4JEFDZF0Q1CQFN/index offset=8000188 length=64 level=debug ts=2019-09-26T19:12:43.942495934Z caller=azure.go:229 msg="check if blob exists" blob=01DHQCWD4AFE4JEFDZF0Q1CQFN/index level=debug ts=2019-09-26T19:12:43.942715235Z caller=azure.go:169 msg="getting blob" blob=01DN8N4PJSNE9F07XJ78RZWN3G/chunks/000001 offset=124056629 length=76827 level=debug ts=2019-09-26T19:12:43.942745235Z caller=azure.go:229 msg="check if blob exists" blob=01DN8N4PJSNE9F07XJ78RZWN3G/chunks/000001 level=debug ts=2019-09-26T19:12:43.942967936Z caller=azure.go:169 msg="getting blob" blob=01DNDSVHEHJKWMBHGFBEM0VBWZ/chunks/000001 offset=125714288 length=75484 level=debug ts=2019-09-26T19:12:43.942992636Z caller=azure.go:229 msg="check if blob exists" blob=01DNDSVHEHJKWMBHGFBEM0VBWZ/chunks/000001 level=debug ts=2019-09-26T19:12:43.943154637Z caller=azure.go:169 msg="getting blob" blob=01DFF9NPD5FMJV67QQKXQES9VG/index offset=11015808 length=40 level=debug ts=2019-09-26T19:12:43.943187937Z caller=azure.go:229 msg="check if blob exists" blob=01DFF9NPD5FMJV67QQKXQES9VG/index level=debug ts=2019-09-26T19:12:43.943152937Z caller=azure.go:169 msg="getting blob" blob=01DJVEEVYA6ABF57ZVEQW57NWW/index offset=9747316 length=64 level=debug ts=2019-09-26T19:12:43.94183713Z caller=azure.go:169 msg="getting blob" blob=01DNQCYTMQBY6BSF9DMRY79D2Q/chunks/000001 offset=4682168 length=17683 level=debug ts=2019-09-26T19:12:43.942301233Z caller=azure.go:169 msg="getting blob" blob=01DNQENREQPDS015JV9582ABRH/chunks/000001 offset=4882397 length=17727 level=debug ts=2019-09-26T19:12:43.943404538Z caller=azure.go:229 msg="check if blob exists" blob=01DJVEEVYA6ABF57ZVEQW57NWW/index level=debug ts=2019-09-26T19:12:43.943404238Z caller=azure.go:169 msg="getting blob" blob=01DKZG25YFZW3GM1PATHFS4D23/index offset=11479680 length=64 level=debug ts=2019-09-26T19:12:43.94377274Z caller=azure.go:229 msg="check if blob exists" blob=01DKZG25YFZW3GM1PATHFS4D23/index level=debug ts=2019-09-26T19:12:43.943429438Z caller=azure.go:169 msg="getting blob" blob=01DN3HMPA27V4QJBENN6GBVR7X/index offset=10645444 length=64 level=debug ts=2019-09-26T19:12:43.94388494Z caller=azure.go:229 msg="check if blob exists" blob=01DN3HMPA27V4QJBENN6GBVR7X/index level=debug ts=2019-09-26T19:12:43.943515839Z caller=azure.go:229 msg="check if blob exists" blob=01DNQCYTMQBY6BSF9DMRY79D2Q/chunks/000001 level=debug ts=2019-09-26T19:12:43.943618239Z caller=azure.go:229 msg="check if blob exists" blob=01DNQENREQPDS015JV9582ABRH/chunks/000001 level=debug ts=2019-09-26T19:12:44.115150299Z caller=azure.go:169 msg="getting blob" blob=01DKZG25YFZW3GM1PATHFS4D23/index offset=9023600 length=65904 level=debug ts=2019-09-26T19:12:44.115240799Z caller=azure.go:229 msg="check if blob exists" blob=01DKZG25YFZW3GM1PATHFS4D23/index level=debug ts=2019-09-26T19:12:44.120705127Z caller=azure.go:169 msg="getting blob" blob=01DFF9NPD5FMJV67QQKXQES9VG/index offset=8726240 length=65728 level=debug ts=2019-09-26T19:12:44.121064829Z caller=azure.go:229 msg="check if blob exists" blob=01DFF9NPD5FMJV67QQKXQES9VG/index level=debug ts=2019-09-26T19:12:44.128493266Z caller=azure.go:169 msg="getting blob" blob=01DN3HMPA27V4QJBENN6GBVR7X/index offset=8671920 length=65920 level=debug ts=2019-09-26T19:12:44.128529266Z caller=azure.go:229 msg="check if blob exists" blob=01DN3HMPA27V4QJBENN6GBVR7X/index level=debug ts=2019-09-26T19:12:44.138257415Z caller=azure.go:169 msg="getting blob" blob=01DJVEEVYA6ABF57ZVEQW57NWW/index offset=7799856 length=65920 level=debug ts=2019-09-26T19:12:44.138294415Z caller=azure.go:229 msg="check if blob exists" blob=01DJVEEVYA6ABF57ZVEQW57NWW/index level=debug ts=2019-09-26T19:12:44.141744732Z caller=azure.go:169 msg="getting blob" blob=01DHQCWD4AFE4JEFDZF0Q1CQFN/index offset=6441648 length=65888 level=debug ts=2019-09-26T19:12:44.141781732Z caller=azure.go:229 msg="check if blob exists" blob=01DHQCWD4AFE4JEFDZF0Q1CQFN/index level=debug ts=2019-09-26T19:12:44.14329904Z caller=azure.go:169 msg="getting blob" blob=01DGKB9BMYMSCC1V6X0DPB9HK7/index offset=5681280 length=65728 level=debug ts=2019-09-26T19:12:44.14333714Z caller=azure.go:229 msg="check if blob exists" blob=01DGKB9BMYMSCC1V6X0DPB9HK7/index level=debug ts=2019-09-26T19:12:44.148481466Z caller=azure.go:169 msg="getting blob" blob=01DKZG25YFZW3GM1PATHFS4D23/chunks/000001 offset=98674414 length=59024 level=debug ts=2019-09-26T19:12:44.148520266Z caller=azure.go:229 msg="check if blob exists" blob=01DKZG25YFZW3GM1PATHFS4D23/chunks/000001 level=debug ts=2019-09-26T19:12:44.162238535Z caller=azure.go:169 msg="getting blob" blob=01DHQCWD4AFE4JEFDZF0Q1CQFN/chunks/000001 offset=88634267 length=45439 level=debug ts=2019-09-26T19:12:44.162274235Z caller=azure.go:229 msg="check if blob exists" blob=01DHQCWD4AFE4JEFDZF0Q1CQFN/chunks/000001 level=debug ts=2019-09-26T19:12:44.170404476Z caller=azure.go:169 msg="getting blob" blob=01DJVEEVYA6ABF57ZVEQW57NWW/chunks/000001 offset=94347127 length=56734 level=debug ts=2019-09-26T19:12:44.170441576Z caller=azure.go:229 msg="check if blob exists" blob=01DJVEEVYA6ABF57ZVEQW57NWW/chunks/000001 level=debug ts=2019-09-26T19:12:44.176458606Z caller=azure.go:169 msg="getting blob" blob=01DN3HMPA27V4QJBENN6GBVR7X/chunks/000001 offset=97926212 length=56129 level=debug ts=2019-09-26T19:12:44.176495006Z caller=azure.go:229 msg="check if blob exists" blob=01DN3HMPA27V4QJBENN6GBVR7X/chunks/000001 level=debug ts=2019-09-26T19:12:44.187456761Z caller=azure.go:169 msg="getting blob" blob=01DGKB9BMYMSCC1V6X0DPB9HK7/chunks/000001 offset=82839235 length=41002 level=debug ts=2019-09-26T19:12:44.187493961Z caller=azure.go:229 msg="check if blob exists" blob=01DGKB9BMYMSCC1V6X0DPB9HK7/chunks/000001 level=debug ts=2019-09-26T19:12:44.274160696Z caller=azure.go:169 msg="getting blob" blob=01DFF9NPD5FMJV67QQKXQES9VG/chunks/000001 offset=81072119 length=41935 level=debug ts=2019-09-26T19:12:44.274227096Z caller=azure.go:229 msg="check if blob exists" blob=01DFF9NPD5FMJV67QQKXQES9VG/chunks/000001 level=debug ts=2019-09-26T19:12:44.373579294Z caller=azure.go:169 msg="getting blob" blob=01DNQFH7BQQH376PAQH6T7P7E9/index offset=2533568 length=65776 level=debug ts=2019-09-26T19:12:44.373640094Z caller=azure.go:229 msg="check if blob exists" blob=01DNQFH7BQQH376PAQH6T7P7E9/index level=debug ts=2019-09-26T19:12:44.40874477Z caller=azure.go:169 msg="getting blob" blob=01DNQFH7BQQH376PAQH6T7P7E9/chunks/000001 offset=4603575 length=17686 level=debug ts=2019-09-26T19:12:44.408801171Z caller=azure.go:229 msg="check if blob exists" blob=01DNQFH7BQQH376PAQH6T7P7E9/chunks/000001 level=debug ts=2019-09-26T19:12:44.444432649Z caller=bucket.go:871 msg="stats query processed" stats="&{blocksQueried:19 postingsTouched:38 postingsTouchedSizeSum:15656 postingsToFetch:0 postingsFetched:14 postingsFetchedSizeSum:781068 postingsFetchCount:12 postingsFetchDurationSum:2440876535 seriesTouched:102 seriesTouchedSizeSum:12548 seriesFetched:32 seriesFetchedSizeSum:460864 seriesFetchCount:7 seriesFetchDurationSum:365501431 chunksTouched:1203 chunksTouchedSizeSum:709390 chunksFetched:1203 chunksFetchedSizeSum:1006454 chunksFetchCount:19 chunksFetchDurationSum:3243238652 getAllDuration:503944626 mergedSeriesCount:6 mergedChunksCount:1203 mergeDuration:1279906}" err=null ```

What else should I check for?

[sepich]

Just can't wait to test this feature in grafana :-)

You can test it already in grafana/grafana:master image. I've tried to set Custom query parameters: max_source_resolution=1h&partial_response=true It starts to display 90d interval, but then i have issues with 1d interval. This also is reproducible in thanos-query UI: Max 1h downsampling and Max 5m downsampling both leads to the same broken result for 1d interval. Auto downsampling and Only raw data returns whole day.

[wbh1]

Experiencing the same issue in Query UI & Grafana with different results depending on the downsampling selected. Step/Resolution don't have an effect. I do have different retention periods set for different resolutions (30, 180, infinite for raw, 5m, 1h respectively). No errors in store or query 🤔

[bwplotka]

It looks like this issue is actually duplicate of https://github.com/thanos-io/thanos/issues/1115, closing in favor of older one.

[rayjcwu]

I'm using sum function summing over gauge metrics and encounter the same issue. Need to set step to more than 25m to see 5m resolution downsampled data. Doesn't the explaination in that old github issue only apply to rate function?