storegateway,compactor,bucketweb: SSE-C encryption causes Bad Request errors on reading data from bucket

[jabbrwcky]

Thanos, Prometheus and Golang version used:

thanos:0.36.1

Object Storage Provider: S3

What happened:

I configured Thanos to store metrics data in OpenStack Ceph via S3 protocol using SSE-C encryption. The sidecar sucessfully writes data to the bucket, but all components reading from the buckets fail to stat the objects in the bucket, returning a 400 Bad Request error.

What you expected to happen:

Data written to the bucket using SSE-C encryption should have been read back without error.

How to reproduce it (as minimally and precisely as possible):

Configure objectstorage with S3 setting encryption to SSE-C:

type: S3
config:
  bucket: metrics
  endpoint: objectstorage.fes.cloud.syseleven.net
  region: fes
  access_key: <key>
  secret_key: <secret>
  sse_config:
    type: SSE-C
    encryption_key: /sse-c-key

The sidecar should be able to store data. All reading components fail.

Anything else we need to know:

I identified thanos-io/objstore as the culprit, because the mino..StatObject() call also requires (at least for SSE-C!) an encryption key to be passed in the request options, which is only provided in Upload and Get resp. GetRange calls there, but not for Exists and Attributes.

As a consequence these calls fail with an 400 Bad Request error.

This may not be the case for SSE-KMS and default SSE methods, because the server handles these transparently.

I opened an issue and an PR over at thanos-io/objstore so this issue is mostly a marker for the release fixing it by using the fixed version of objstore: https://github.com/thanos-io/objstore/issues/141

[dosubot[bot]]

Hey @jabbrwcky! I'm here to help you squash those bugs, answer your burning questions, and guide you on your journey to becoming a top-notch contributor. Let's tackle this together! 🤖

I found a similar unsolved discussion that might be relevant to your issue: Thanos store and compactor fail to load and sync blocks from Linode Object Storage uploaded via sidecar using sse-c encryption ^[1].

To resolve the 400 Bad Request error when Thanos components read from an OpenStack Ceph bucket using SSE-C encryption with the S3 protocol, ensure that the sse_config section in your Thanos configuration is correctly set up for SSE-C. Specifically, you need to provide the path to the encryption key using the encryption_key field.

Here is an example configuration snippet for SSE-C:

type: S3
config:
  bucket: "<your-bucket-name>"
  endpoint: "<your-endpoint>"
  access_key: "<your-access-key>"
  secret_key: "<your-secret-key>"
  sse_config:
    type: "SSE-C"
    encryption_key: "<path-to-your-encryption-key>"

Make sure that the encryption_key points to the correct path where your encryption key is stored. This should resolve the 400 Bad Request error related to SSE-C encryption ^[2].

To continue talking to Dosu, mention @dosu.

[jabbrwcky]

@dosu The symptoms found by dosubot are similar, but the encryption key is configured correctly in my case (I can access the files using a test script using minio SDK directly).

I have linked the issue and PR in thanos-io/bjstore that should fix this issue above.

[dosubot[bot]]

To resolve the 400 Bad Request error when Thanos components read from an OpenStack Ceph bucket using SSE-C encryption, ensure that the encryption key is passed correctly in the Exists() and Attributes() methods. This can be done by configuring the sse_config section in your Thanos configuration file to include the encryption_key for SSE-C.

Here is an example configuration for an S3 bucket with SSE-C encryption:

type: S3
config:
  bucket: "<your-bucket-name>"
  endpoint: "<your-endpoint>"
  access_key: "<your-access-key>"
  secret_key: "<your-secret-key>"
  sse_config:
    type: "SSE-C"
    encryption_key: "<path-to-your-encryption-key>"

Make sure that the encryption_key is correctly specified and accessible by Thanos components. This should ensure that the encryption key is passed during the Exists() and Attributes() method calls, resolving the 400 Bad Request error ^[1].

To continue talking to Dosu, mention @dosu.