Add Azure Blob Storage source

vectordotdev / vector

A high-performance observability data pipeline.

https://vector.dev

Mozilla Public License 2.0

18.25k stars 1.6k forks source link

Add Azure Blob Storage source #13882

Open spencergilbert opened 2 years ago

spencergilbert commented 2 years ago

A note for the community

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

Similar use-case to our s3 and gcs sources, and the mirror of the azure_blob_storage sink we have today.

Attempted Solutions

N/A

Proposal

Implement azure_blob_storage source akin to gcs and s3 sources.

References

No response

Version

master@ 461130

kwapik commented 6 months ago

I'd like to work on this issue together with @ldrozdz93.

Xuanwo commented 1 month ago

cc @jszwedko, Similar to what I mentioned in https://github.com/vectordotdev/vector/issues/7501#issuecomment-2395498188, I'm willing to start an implementation for azblob based on opendal.

After these implementations, I'm eager to add documentation on how to add more sources based on opendal and work on improving existing sources to better align with others.

Maybe we could establish a tracking issue for those tasks. Do you think it's a good direction?

ldrozdz93 commented 1 month ago

FYI: me and @kwapik implemented the azblob source on our client's fork of v0.38 and it's been working on prod for the last few months. Soon I'll start porting that to vector upstream.

Xuanwo commented 1 month ago

FYI: me and @kwapik implemented the azblob source on our client's fork of v0.38 and it's been working on prod for the last few months. Soon I'll start porting that to vector upstream.

That will be great. Looking forward to future collaboration, we can build on your work.