search

grafana / tempo

Grafana Tempo is a high volume, minimal dependency distributed tracing backend.
https://grafana.com/oss/tempo/
GNU Affero General Public License v3.0
3.93k stars 510 forks source link

Tenant error when S3 bucket analytics turned on for S3 backend storage bucket #3913

Open dgeorges opened 2 months ago

dgeorges commented 2 months ago

Describe the bug

When S3 bucket analytics is enable a folder is create at the root of the bucket for the analytics to be written to. In our case it's named the folder is named_AWSBucketAnalytics/ see. https://docs.aws.amazon.com/AmazonS3/latest/userguide/analytics-storage-class.html

This bucket appears in log and many of the tenant metrics such as tempodb_blocklist_tenant_index_errors_total, tempodb_blocklist_tenant_index_age_seconds where is looks like the folder is being treated like a tenant.

This is the log statement. 

2024-07-25T19:15:14.297265084Z stderr F level=error ts=2024-07-25T19:15:14.297179018Z caller=poller.go:225 msg=\"failed to pull bucket index for tenant. falling back to polling\" tenant=_AWSBucketAnalytics err=\"does not exist\"

To Reproduce Steps to reproduce the behavior:

  1. Start Tempo (2.4.1)
  2. Enable S3 bucket analytics
  3. Perform Operations (Read/Write/Others)

Expected behavior

Tempo should be able to determine if a folder is a tenant or not and if not should not log it as such in metrics and logs.

Environment:

Additional Context

joe-elliott commented 2 months ago

Tempo considers any "folder" in the root of its bucket to be a tenant. There are virtually no restrictions on tenant name so it's quite possible for "_AWSBucketAnalytics" to be a tenant.

One option to work around this would be to use the s3 prefix option:

https://github.com/grafana/tempo/blob/d0f0a5fddb7b3234e70f4bd39bb592a8e0b5df50/tempodb/backend/s3/config.go#L17

github-actions[bot] commented 6 days ago

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed after 15 days if there is no new activity. Please apply keepalive label to exempt this Issue.