S3 Exporter in ADOT

[anish-aws]

Is your feature request related to a problem? Please describe. I am working with a team that wants to export application traces into long term storage. The team has needs to perform custom analysis (things not provided by services like DataDog) over the long term and also analyse traces data in conjunction with other data sets from various sources. They want the ability to move the trace data into cheap and durable storage like S3 and use tools like Athena, EMR, SageMaker, etc to perform analysis.

Describe the solution you'd like The solution requested would be the ability to configure an S3 exporter, similiar to other exporters like X-Ray exporter or Prometheus Exporter. The exporter would take in configuration like the bucket name, path, etc and the ADOT collector will ship trace information into the S3 bucket. From there on downstream systems can consume it as they like. Current use case for the team is to export traces, but future use cases exist for similiar functionality with logs and metrics

Describe alternatives you've considered Other options like Kinesis Exporter and X-Ray exporter was considered. this adds additional step in the data movement to downstream systems and cost. This also requires a substantial engineering effort to standup the solution and maintain it. Analysis inside current services like DataDog or Jaeger is limited as performing long term query analysis with multi source data sets is either technically impossible (such as exporting data from DataDog) or cost prohibitive. Open-source alternatives like Parquet exporter are currently in beta and would also require the engineering time to build a solution to move data from the container system into S3. The expectation is for the S3 exporter and the collector to do this activity, if built

[mhausenblas]

Thanks @anish-aws, we will have a look into it and update this feature request within the next couple of days.

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[anish-aws]

Commenting for bot. but also, any update on this request?

[mhausenblas]

@anish-aws we're in the review process, gathering input. No confirmed ECD yet.

[anish-aws]

Linking open issue from Upstream that seems to have a sponsor as of last week https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/2835

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[kevinslin]

in case folks were curious - there is now alpha support for s3 exporter in opentelemetry-collector-contrib

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[github-actions[bot]]

This issue was closed because it has been marked as stale for 30 days with no activity.

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[odeke-em]

@Aneurysm9 @mhausenblas howdy howdy? I can take on this issue and bring in the exporter that was implemented in https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/awss3exporter

[mhausenblas]

Thanks for your offer @odeke-em, much appreciated. However, this component is not on our short term roadmap, yet.

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.