Open brianpham opened 3 years ago
Is there anything on the roadmap for this source?
Not yet, but we have been experimenting with OpenDAL, which was recently used to add a WebHDFS sink, and does have support for GCS. It could be an avenue to experiment with if anyone wants to take a shot at this.
@jszwedko Would you accept a PR that implements this in roughly the same way that the aws_s3
source is implemented, i.e. via event notifications in a PubSub topic?
@jszwedko Would you accept a PR that implements this in roughly the same way that the
aws_s3
source is implemented, i.e. via event notifications in a PubSub topic?
Hey! Yes, I think that would make sense as the initial implementation to match the behavior of the aws_s3
source.
Not yet, but we have been experimenting with OpenDAL, which was recently used to add a WebHDFS sink, and does have support for GCS. It could be an avenue to experiment with if anyone wants to take a shot at this.
Hi, @jszwedko. I'm willing to help implement the GCS source, but I might not have time to complete the full documentation. Do you think it's a good idea to start the implementation first? For example, all content under src/sinks/webhdfs
but not website/**/webhdfs
.
As a replacement for logstash, we will need vector to support google cloud storage as a source similar to AWS S3 (https://vector.dev/docs/reference/configuration/sources/aws_s3/)
We do something similar in logstash