search

cortexproject / cortex

A horizontally scalable, highly available, multi-tenant, long term Prometheus.
https://cortexmetrics.io/
Apache License 2.0
5.47k stars 797 forks source link

Cloudflare R2 Header 'x-amz-checksum-algorithm' with value 'CRC32C' not implemented #5829

Closed mpavlovicbb closed 6 months ago

mpavlovicbb commented 7 months ago

Describe the bug I am running my Cortex deployment in combination with Cloudflare R2 bucket. In Ingesters we have following logs:

ts=2024-03-25T11:25:24.379913283Z caller=shipper.go:325 level=error org_id=rxgqq9xn96 msg="shipping failed" block=01HST6JB4PKRJ2NEGRPQN3JSS9 err="upload chunks: upload file /data/rxgqq9xn96/thanos/upload/01HST6JB4PKRJ2NEGRPQN3JSS9/chunks/000001 as 01HST6JB4PKRJ2NEGRPQN3JSS9/chunks/000001: upload s3 object: Header 'x-amz-checksum-algorithm' with value 'CRC32C' not implemented" |  
ts=2024-03-25T11:25:24.377531977Z caller=bucket_client.go:182 level=warn msg="skip upload retry as reader is not seekable" file=c95nsmcjh9/01HSTK3QGYGVBECS5GP0KZBAP3/chunks/000001 err="upload s3 object: Header 'x-amz-checksum-algorithm' with value 'CRC32C' not implemented" |  
ts=2024-03-25T11:25:24.336389491Z caller=bucket_client.go:182 level=warn msg="skip upload retry as reader is not seekable" file=j67mpw76kd/01HST6JGATCSY9CXR0JHYSMHF9/chunks/000001 err="upload s3 object: Header 'x-amz-checksum-algorithm' with value 'CRC32C' not implemented" |  
ts=2024-03-25T11:25:24.315175576Z caller=bucket_client.go:182 level=warn msg="skip upload retry as reader is not seekable" file=d66xdbw5mf/01HSTK3T0N2PERDWMVJNFKH18J/chunks/000001 err="upload s3 object: Header 'x-amz-checksum-algorithm' with value 'CRC32C' not implemented" |  
ts=2024-03-25T11:25:24.313750159Z caller=bucket_client.go:182 level=warn msg="skip upload retry as reader is not seekable" file=w6a16f481/01HSSZPD1TQMHR44YXBHBHVFW6/chunks/000001 err="upload s3 object: Header 'x-amz-checksum-algorithm' with value 'CRC32C' not implemented" |  
ts=2024-03-25T11:25:24.302724654Z caller=shipper.go:325 level=error org_id=c95nsmcjh9 msg="shipping failed" block=01HSTDDV6KCMCQTK7PRZB8RJZ0 err="upload chunks: upload file /data/c95nsmcjh9/thanos/upload/01HSTDDV6KCMCQTK7PRZB8RJZ0/chunks/000001 as 01HSTDDV6KCMCQTK7PRZB8RJZ0/chunks/000001: upload s3 object: Header 'x-amz-checksum-algorithm' with value 'CRC32C' not implemented" 
ts=2024-03-25T11:25:24.262010696Z caller=bucket_client.go:182 level=warn msg="skip upload retry as reader is not seekable" file=xnpxx8mnsz/01HSTK3STNQKV3182KGPE8X4HQ/chunks/000001 err="upload s3 object: Header 'x-amz-checksum-algorithm' with value 'CRC32C' not implemented"

To Reproduce Steps to reproduce the behavior:

  1. Distributed Cortex installation in Kubernetes with 5+ tenants
  2. Cloudflare R2 bucket to send metrics to

Expected behavior Without errors in logs.

Environment:

friedrichg commented 7 months ago

to implement this, we just need to expose https://github.com/cortexproject/cortex/blob/99d8c3e2f741c2c740fac598c48c003263b56721/vendor/github.com/thanos-io/objstore/providers/s3/s3.go#L116 to the configuration, so users can disable this