"The specified bucket does not exist" but all buckets are present in minio

[kandeshvari]

Describe the bug

I'm getting the error for all services

ts=2024-03-25T21:35:58.095869334Z caller=sanity_check.go:115 level=warn msg="Unable to successfully connect to configured object storage (will retry)" err="blocks storage: unable to successfully send a request to object storage: The specified bucket does not exist."

All needed buckets exist in minio. root credentials for minio are correctly passed, and mimir sees it. I can list all created buckets via mc utility with those credentials

part of the values.yaml

mimir:
  structuredConfig:
    common:
      storage:
        backend: s3
        s3:
          insecure: true
          http:
            insecure_skip_verify: true
          endpoint: minio.minio-tenant-mimir.svc.cluster.local
          access_key_id: PA...
          secret_access_key: hXy...

    alertmanager_storage:
      s3:
        insecure: true
        bucket_name: mimir-alertmanager

    ruler_storage:
      s3:
        insecure: true
        bucket_name: mimir-ruler

    blocks_storage:
      s3:
        insecure: true
        bucket_name: mimir-blocks

How can I debug this behavior and figure out the reason why mimir does not see the buckets?

I spent the whole day trying, and I'm close to despair :(

Environment

• Infrastructure: bare-metal Kubernetes 1.27 on Virtual Machines
• Deployment tool: helm, argocd
• versions:
  • grafana/mimir-distributed, 5.2.3
  • minio/operator, 5.0.11
  • minio/tenant, 5.0.11

[andyasp]

This may be because there is a minio section in the values file. If you're setting up your minio seperately then at the top-level you could counter-intuitively set

minio:
  enabled: false

One way you could verify if this is the interaction you're experiencing is by looking at the config that is actually being generated with:

kubectl describe configmap/mimir-config

For instance, if you see mimir-tsdb in the config and not your mimir-blocks, then that's the issue.

[kandeshvari]

Thanks for your reply, @andyasp!

Yes, I tried to use embedded minio, and it works fine. It seems like some policies are not set properly. Is there a guide to set up the junction of mimir and minio?

[andyasp]

It seems like some policies are not set properly.

I'm not sure what this means, but it sounds like maybe you figured it out?

Is there a guide to set up the junction of mimir and minio?

Not that I'm aware of. There's the object storage configuration documentation, but it's not specifically about minio. From Mimir's perspective it is only aware of the fact that it is talking to an s3 backend. There's nothing special about minio in this sense. As long as it knows the buckets and how to talk to it/how to auth to it that's basically it.

The helm chart has that embedded minio included by default so it can work out of the box, but it's suggested to override it for production use cases. You can see an artifact of that in some of the sample value files, like here.

[kandeshvari]

@andyasp . I just found a root cause: that was a confusing error message.

my settings are:

mimir:
  structuredConfig:
    common:
      storage:
        s3:
          insecure: true
          http:
            insecure_skip_verify: true
          endpoint: minio.minio-tenant-mimir.svc.cluster.local
          ...

key point here is insecure: true. This endpoint receives only HTTPS. For some reason, I got the error message

blocks storage: unable to successfully send a request to object storage: The specified bucket does not exist.

instead of

blocks storage: unable to successfully send a request to object storage: HTTP request made to HTTPS server

I removed insecure: true, and everything started to work fine.

Is it a bug?

[andyasp]

Interesting. I don't think that's a bug with the sanity check. It's rather simple and creates the bucket client and GETs a single object (code reference). Here the error it's getting from minio has the code NoSuchBucket and it's reflecting that back.

If there was a simple reproduction procedure (like with curl or minio-go directly) to verify that swapping HTTP/HTTPS is all that's going on then filing an issue with minio may help improve that.

[narqo]

blocks storage: unable to successfully send a request to object storage: The specified bucket does not exist.

I think this is a "non-ideal" behaviour in mimio-go (the s3 API client, we use through the objstore package): they treat all responses with 404 as "bucket not exists" (ref the code in mimio-go).

[andyasp]

they treat all responses with 404 as "bucket not exists"

That isn't true and would be very incorrect behavior. Requesting an object that doesn't exist results in 404 and a NoSuchKey code. The code you linked shows that in the else case. That's what objstore looks for too: objstore ref

[narqo]

they treat all responses with 404 as "bucket not exists"

That isn't true and would be very incorrect behavior. Requesting an object that doesn't exist results in 404 and a NoSuchKey code. The code you linked shows that in the else case

What I wanted to say is that a request to an object, needs to locate the object's bucket, first, by calling to S3's GetBucketLocation API (ref to this chunk in minio-go).

I suspect, the error reported in the issue came from such request, as it returned 404 due to wrong protocol.

[andyasp]

Ah, gotcha.

[HamzaKhait]

Hello @kandeshvari

I have the exact same problem and I would be extremely glad if you could share your Mimir and MinIO configuration 🙏

I tried to use the same configuration as suggested in the official tutorial, but I get this error :

Jul  2 09:50:00 mimir1 mimir[57769]: ts=2024-07-02T09:50:00.753945562Z caller=sanity_check.go:115 level=warn msg="Unable to successfully connect to configured object storage (will retry)" err="blocks storage: unable to successfully send a request to object storage: The specified bucket does not exist"

When I check on MinIO, I can see that my bucket 'mimir' exists

My mimir and minio are hosted on the same server. and MinIO is not using HTTPS.

---

EDIT : problem solved

My bad, I had a bad configuration parameter. Here's my mimir configuration file for anyone facing the same issue :

# Run Mimir in single process mode, with all components running in 1 process.
target: all,alertmanager,overrides-exporter

# Configure Mimir to use Minio as object storage backend.
common:
  storage:
    backend: s3
    s3:
      endpoint: 10.103.0.174:9000
      access_key_id: mimir
      secret_access_key: supersecret
      insecure: true
      bucket_name: mimir

# Blocks storage requires a prefix when using a common object storage bucket.
blocks_storage:
  storage_prefix: blocks

# Blocks storage requires a prefix when using a common object storage bucket.

# Alertmanager storage requires a prefix when using a common object storage bucket.

# Use memberlist, a gossip-based protocol, to enable the 3 Mimir replicas to communicate.

ruler:
  rule_path: /var/lib/mimir/ruler
  alertmanager_url: http://127.0.0.1:8080/alertmanager
  ring:
    heartbeat_period: 5s
    heartbeat_timeout: 10s

alertmanager:
  data_dir: /var/lib/mimir/alertmanager
  external_url: http://localhost:8080/alertmanager

server:
  http_listen_port: 8080
  http_listen_address: 0.0.0.0
  log_level: warn

limits:
  compactor_blocks_retention_period: 6m