Ingest existing object in AWS_S3 source

[ambroserb3]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

Vector is a great alternative to logstash in ELK stack. But there doesn't seem to be anyway to send existing data from an s3 bucket to Elasticsearch with vector, making migration harder.

This wouldn't just benefit an EVK stack, there are multiple use patterns where it would be useful to ingest existing s3 objects.

Attempted Solutions

I haven't tried this yet, but I was thinking I could just tag the existing objects in s3 and change my SQS rules to include created tags.

Proposal

No response

References

No response

Version

vector 0.26.0

[jszwedko]

Hi @stephanrb3 !

Thanks for this feature request! We actually have a parent issue where we are collecting ETL-like use-cases like this one, https://github.com/vectordotdev/vector/issues/11095, so I'll close this one and link it there. Feel free to add any additional details there.

As a workaround, you could publish SQS messages in the same format as the bucket notifications to trigger Vector's reading of the S3 files.