grafana / tempo

Grafana Tempo is a high volume, minimal dependency distributed tracing backend.
https://grafana.com/oss/tempo/
GNU Affero General Public License v3.0
3.88k stars 504 forks source link

Grafana Tempoo with Storage at S3 #2099

Closed JalisDiehl closed 1 year ago

JalisDiehl commented 1 year ago

We are installing our Grafana Tempo in a local kubernetes cluster and trying to storage jaeger traces to s3 to query longer traces and use TraceQL. However when we try to request to out s3 we got some error the first one was: │ level=error ts=2023-02-09T21:42:42.950354002Z caller=main.go:117 msg="error running Tempo" err="failed to init module services error initialising module: store: failed to create store un │ │ expected error from ListObjects on /plataform-monit-xptoxxxxxxxxxxxxxx-nprod-4/nprod/traces-parquet-tempo/: Bucket name cannot be longer than 63 characters"

Already we try to shortcut our bucket a bit with:

/plataform-monit-xptoxxxxxxxxxxxxxx-nprod-4/nprod/parquet/ but we got error like this:

caller=main.go:117 msg="error running Tempo" err="failed to init module services error initialising module: store: failed to create store un │ │ expected error from ListObjects on /plataform-monit-xptoxxxxxxxxxxxxxx-nprod-4/nprod/parquet/: Bucket name contains invalid characters"

This s3 is local and with many application using different path for this reason we need to put / That's our settings for storage:

> storage:
>   trace:
>     backend: "s3"
>     s3:
>       access_key: <access-key>
>       bucket: /plataform-monit-xptoxxxxxxxxxxxxxx-nprod-4/nprod/parquet/
>       endpoint: s3.local.net
>       insecure_skip_verify: true
>       secret_key: <secret-key>
mapno commented 1 year ago

Hi! Bucket names with / are not allowed per S3's bucket naming rules.

If you are trying to access a subfolder within a bucket the proper syntax to do so would be to specify the subfolder within the key value. Unfortunately, from what I've checked, it's not possible to configure Tempo to access a specific folder within an S3 bucket to use as its root folder. I'll open an issue for it, it'll be nice to have.

mapno commented 1 year ago

Here is the issue: https://github.com/grafana/tempo/issues/2100

joe-elliott commented 1 year ago

Closing this since it seems to be covered by #2100. Thanks for filing the issue @JalisDiehl