Template the `bucket` field for `GCP Cloud Storage` sink

FlorianGD commented 2 years ago

A note for the community

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

We want to select the bucket to write to when using the GCP Cloud Storage sink based on the information in the logs. The field is not templated at the moment, so it is not possible.

Attempted Solutions

No response

Proposal

Allowing this syntax would be very useful to us:

gcs:
    type: gcp_cloud_storage
    inputs:
      - my_input
    bucket: "bucket-{{context.project.guid}}"

References

No response

Version

vector 0.20.0 (x86_64-unknown-linux-gnu 2a706a3 2022-02-11)

jerome-kleinen-kbc-be commented 2 years ago

I have a similar requirement but for the AWS S3 sink. We are using an S3 compatible file storage which struggles with large buckets, hence we want to make buckets per time period. Ideally we would like to do this in a templated way instead of updating our configs periodically.

clotildeguinard commented 2 years ago

I have a similar requirement but for the influxdb_metrics sink. I want to send metrics to different buckets depending on some metric field, my goal is to segregate data for security purpose.

vectordotdev / vector