Thanos Sharding Cache Issue

[Lukiido]

Name and Version

bitnami/thanos 15.5.0

What architecture are you using?

amd64

What steps will reproduce the bug?

If you enable sharding and have an indexcacheconfig or bucketcacheconfig set the storegateway shards do not come up, they crashloop with an error saying the file cannot be found. Wondering if the sts yaml for the shards needs to be updated possibly?

Are you using any custom parameters or values?

indexCacheConfig: |
type: IN-MEMORY
config:
max_size: 512MB
max_item_size: 125MB

What is the expected behavior?

The sharded storegateway is to come up.

What do you see instead?

This error:

{"caller":"main.go:145","err":"open /conf/cache/index-cache.yml: no such file or directory
loading file /conf/cache/index-cache.yml for index-cache.config-file\http:/
github.com/efficientgo/tools/extkingpin.(*PathOrContent).Content
    /bitnami/blacksmith-sandox/thanos-0.35.0/pkg/mod/github.com/efficientgo/tools/extkingpin@v0.0.0-20220817170617-6c25e3b627dd/pathorcontent.go:79
main.runStore
    /bitnami/blacksmith-sandox/thanos-0.35.0/src/github.com/thanos-io/thanos/cmd/thanos/store.go:341
main.registerStore.func1
    /bitnami/blacksmith-sandox/thanos-0.35.0/src/github.com/thanos-io/thanos/cmd/thanos/store.go:250
main.main
    /bitnami/blacksmith-sandox/thanos-0.35.0/src/github.com/thanos-io/thanos/cmd/thanos/main.go:143
runtime.main
    /opt/bitnami/go/src/runtime/proc.go:267
runtime.goexit
    /opt/bitnami/go/src/runtime/asm_amd64.s:1650
get content of index cache configuration
main.runStore
    /bitnami/blacksmith-sandox/thanos-0.35.0/src/github.com/thanos-io/thanos/cmd/thanos/store.go:343
main.registerStore.func1
    /bitnami/blacksmith-sandox/thanos-0.35.0/src/github.com/thanos-io/thanos/cmd/thanos/store.go:250
main.main
    /bitnami/blacksmith-sandox/thanos-0.35.0/src/github.com/thanos-io/thanos/cmd/thanos/main.go:143
runtime.main
    /opt/bitnami/go/src/runtime/proc.go:267
runtime.goexit
    /opt/bitnami/go/src/runtime/asm_amd64.s:1650
preparing store command failed
main.main
    /bitnami/blacksmith-sandox/thanos-0.35.0/src/github.com/thanos-io/thanos/cmd/thanos/main.go:145
runtime.main
    /opt/bitnami/go/src/runtime/proc.go:267
runtime.goexit
    /opt/bitnami/go/src/runtime/asm_amd64.s:1650","level":"error","ts":"2024-05-24T14:22:52.795611102Z"}

[fmulero]

Hi @Lukiido, thanks for using Bitnami charts.

I am trying to reproduce your issue but I didn't have luck. Are you sure that you are using the right indentation?:

indexCacheConfig: |
  type: IN-MEMORY
  config:
  max_size: 512MB
  max_item_size: 125MB

Could you share concrete steps to reproduce it? Please include the whole values.yaml file?

[Lukiido]

Hey sure thing!

Basically this issue only occurs when using this values.yaml file and sharding is enabled. If we disable sharding then the storegateway will come up without issue.

I am using argocd to deploy any changes, so we update the values file and push into our repo.

thanos-values.txt

[sgreben]

@Lukiido This looks a lot like the issue addressed by my two PRs here (#26490 (buggy fix) and #27101 (fixing the buggy fix)). When v15.7.6 is released this should just work again..

[sgreben]

oh right, note that you'll need to use storegateway.config instead of the toplevel indexCacheConfig key to configure the index cache. Otherwise it tries to set the config files from both keys as the index cache config and the pods fail to start.

[Lukiido]

oh right, note that you'll need to use storegateway.config instead of the toplevel indexCacheConfig key to configure the index cache. Otherwise it tries to set the config files from both keys as the index cache config and the pods fail to start.

Oh thank you! do you happen to have a link to the doc or an example, I must have missed it when I was digging through the chart.

[sgreben]

It's really just

storegateway:
  config: |
    type: IN-MEMORY
    config:
    max_size: 512MB
    max_item_size: 125MB

I don't think I saw this in the docs at all. Here's where I found out about this: https://github.com/bitnami/charts/issues/24678#issuecomment-2042559131

[sgreben]

The chart could really use another follow-up PR to deconflict indexCacheConfig and storegateway.config somehow. Tbh I'm really not sure what the point of additionally having storegateway.config is, given that it's passed to the index-cache config thanos CLI flag..

[Lukiido]

Hey, I removed my last comment after finding this issue here off of the link provided above. This seems to work to get the sharded gateways up! Thanks for the help!

I think one improvement would be to no need to do this and just define them earlier in the values like we would if it isnt sharded, or to add this to the documentation some place.

Thanks again!

[Lukiido]

Sorry, I did get these confused it does look like the issue I linked doesnt fix it for sharding it only fixes the statefulset.yaml. If I am wanting to define a bucketcache and indexcache for the shards how do I do that under the config spec using the example you provided @sgreben ?

Thanks again!

[Lukiido]

Hey, Just wanted to touch back on this issue to see if anyone is able to provide me with example config or some docs on how to set index and bucket cache within the storage-gateway shareded setting? Currently when trying to set both it indicates I have duplicate options and crashloops.