search

opensearch-project / data-prepper

Data Prepper is a component of the OpenSearch project that accepts, filters, transforms, enriches, and routes data at scale.
https://opensearch.org/docs/latest/clients/data-prepper/index/
Apache License 2.0
238 stars 176 forks source link

Store Logs at S3 bucket #4651

Open adecchi-2inno opened 2 weeks ago

adecchi-2inno commented 2 weeks ago

Describe the bug I am not able to find a link or document where explain how to store data at S3 bucket after some period of time.

Expected behavior I am expected to store logs to s3 bucket after some period of time.

Chart Name opensearch 2.20.0

prudhvigodithi commented 1 week ago

[Triage] Hey @adecchi-2inno, you can use S3 to store snapshots of old logs. I'm not sure if its possible from Data Prepper to extract data from OpenSearch and customize to push to S3. Thank you @getsaurabh02 @peterzhuamazon @gaiksaya

dlvenable commented 1 week ago

@adecchi-2inno , Can you clarify your request here?

Data Prepper can read from an OpenSearch source and write to S3. So if you are looking to move documents, you can create a pipeline to do that.

Here are some links that might help:

https://opensearch.org/docs/latest/data-prepper/pipelines/configuration/sources/opensearch/

https://opensearch.org/docs/latest/data-prepper/pipelines/configuration/sinks/s3/