search

thanos-io / thanos

Highly available Prometheus setup with long term storage capabilities. A CNCF Incubating project.
https://thanos.io
Apache License 2.0
13.1k stars 2.1k forks source link

Sidecar: no supported bucket was configured, uploads will be disabled #1195

Closed MaxDiOrio closed 5 years ago

MaxDiOrio commented 5 years ago

Using latest releases of Prometheus-Operator and Thanos Sidecar (v0.4.0

What happened - Launched Prom Op with a Thanos Sidecar and receive the following error in the sidecar: caller=sidecar.go:123 msg="no supported bucket was configured, uploads will be disabled"

What you expected to happen. The sidecar would connect to the S3 compatible Object Storage

How to reproduce it (as minimally and precisely as possible):

Have the following in the Prometheus Operator values yaml file:

    thanos:
      baseImage: improbable/thanos
      version: v0.4.0
      peers: thanos-peers.prod.cluster:10900
      objectStorageConfig:
        key: bucket.yaml
        name: thanos-objstore-config

I have a secret name thanos-objstore-config in the same namespace with a key bucket.yaml and value:

type: S3
config:
  endpoint: "minioingress.rook-minio.10.85.166.33.xip.io"
  bucket: "thanos"
  access_key: "MINIO_Access_Key"
  secret_key: "MINIO_Secret_Key"
  insecure: true
  signature_version2: false

The actual Thanos Store container has no problems with the bucket config used to access this same bucket.

bwplotka commented 5 years ago

I am pretty sure this is something with how operator configures Thanos. Can you double check what the actual Sidecar and Prometheus YAML looks like when it is generated and running on Kubernetes?

MaxDiOrio commented 5 years ago

That sounds right - the yaml doesn't have any reference to the argument. 

args:
        - sidecar
        - --prometheus.url=http://127.0.0.1:9090/
        - --tsdb.path=/prometheus
        - --cluster.address=[$(POD_IP)]:10900
        - --grpc-address=[$(POD_IP)]:10901
        - --cluster.peers=thanos-peers.prod.cluster:10900
MaxDiOrio commented 5 years ago

It's strange - doing a dry-run in debug mode in helm shows that it is using the config:

thanos:
      baseImage: improbable/thanos
      objectStorageConfig:
        key: bucket.yaml
        name: thanos-objstore-config
      peers: thanos-peers.prod.cluster:10900
      version: v0.4.0
bwplotka commented 5 years ago

Hm, maybe then just not deployed yet?

MaxDiOrio commented 5 years ago

It's all definitely been deployed - the whole operator itself is working except for the bucket config. It's really strange.

FUSAKLA commented 5 years ago

Hi, what is the status of this? Did you managet to get the sidecar uploading the blocks? From the discussion it sounds more related to the prometheus operator than actual thanos issue.

MaxDiOrio commented 5 years ago

I gave up on Thanos. Far too many moving pieces for the size of our environment. Especially when a core piece doesn’t work. I switched to remote writing our Prometheus instances to a VictoriaMetrics server and couldn’t be happier.

FUSAKLA commented 5 years ago

Ok, sorry to hear that but whatever works for you is fine. So I'm closing this fo now.

Radoslaw-Kutylowski commented 10 months ago

If anyone else stumbles upon this issue like I did - I believe kube-prometheus-stack changed the format of values for thanos-sidecar's objectStore config: Screenshot from 2023-12-27 11-52-43 I have not tested it thoroughly yet, but at least the generated configuration looks right and the "no supported bucket was configured" message disappeared from logs

pjuri commented 10 months ago

If anyone else stumbles upon this issue like I did - I believe kube-prometheus-stack changed the format of values for thanos-sidecar's objectStore config: Screenshot from 2023-12-27 11-52-43 I have not tested it thoroughly yet, but at least the generated configuration looks right and the "no supported bucket was configured" message disappeared from logs

This helped, thanks!

Elter71 commented 10 months ago

If anyone else stumbles upon this issue like I did - I believe kube-prometheus-stack changed the format of values for thanos-sidecar's objectStore config: Screenshot from 2023-12-27 11-52-43 I have not tested it thoroughly yet, but at least the generated configuration looks right and the "no supported bucket was configured" message disappeared from logs

Works thanks! <3

pduchnovsky commented 9 months ago

If anyone else stumbles upon this issue like I did - I believe kube-prometheus-stack changed the format of values for thanos-sidecar's objectStore config: Screenshot from 2023-12-27 11-52-43 I have not tested it thoroughly yet, but at least the generated configuration looks right and the "no supported bucket was configured" message disappeared from logs

If anybody is looking when this happened: https://github.com/prometheus-community/helm-charts/pull/3918