feat(amqp sink): Add ability to use message persistence

[andrejbaran]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

🙋 I am more than happy to work on this as it seems to be an easy task to do and since no one else (to my knowledge) raised this "issue" yet

Use Cases

In order to ensure logs really make it to processing/end storage while using AMQP sinks it would be really great to have an option to mark the message as persistent. This way logs (marked as such, like audit logs for instance) would survive restart of AMQP component (e.g. RabbitMQ).

Attempted Solutions

Unfortunately persistence in RabbitMQ can be setup only at the producer level and not via policies so there is really no other option how to achieve this than adding this to vector itself in one way or another.

Proposal

There are several approaches to tackle this:

Add an optional AMQP sink configuration field (like mark_persistent for instance)

• Use a boolean
  • that would quite limiting: either none or all of the messages would be marked as such for a particular sink
  • to work around that limitation several sinks would have to be declared in case only certain messages need to be persisted
• Use a template that vould evaluate to an event/metadata path expr having values true or false
  • this feels like kind of misusing the template type since it always results in string and in this case the value would need to be coerced into a boolean and in case that path doesn't exist the event would be dropped but there is always the possibility to create a transform that would ensure the existence of the key 🤷 as is suggested in the docs
• Force the user (on a "use it if you want" basis) to use a metadata key path having values true or false (like: %amqp.persistent)
  • this doesn't feel right in respect to configurability of the rest of vector

References

No response

Version

vector 0.28.1 (x86_64-unknown-linux-musl ff15924 2023-03-06)

[egorovd]

Can not understand... )) Who does not need such a useful thing?

[jszwedko]

Can not understand... )) Who does not need such a useful thing?

We are open to a PR here :) It should be a relatively straight-forward change to expose.

[egorovd]

We are open to a PR here :) It should be a relatively straight-forward change to expose.

I decided to use vector in one of my projects... And rested on the fact that I can’t do it because of this thing. Now looking for alternative solutions... Maybe fluentd, haven't decided yet. But as soon as this thing appears, you will definitely pass on it. Pretty cool product. Taking off my hat. Thank you.

P.S. If today it was added, I would build vector from the source. ))

[andrejbaran]

@jszwedko great. I will find some time to implement it. Sorry for the late reply.

[andrejbaran]

@jszwedko which solution of those mentioned would you guys prefer?

[jszwedko]

@jszwedko which solution of those mentioned would you guys prefer?

Is it common to only want certain messages to be persistent? If it is, than the template approach makes sense. Otherwise I would say just go with a boolean.

I would not go with the third approach since we don't have any precedent for that pattern in Vector at the moment.

[andrejbaran]

Is it common to only want certain messages to be persistent? If it is, than the template approach makes sense. Otherwise I would say just go with a boolean.

Well, that depends I guess. I'd say it could be a valid scenario, where for example you absolutely don't want to loose audit log messages or certain levels of but you don't mind if you loose some extent of others (e.g. some levels of infrastructure or application logs)

I don't think there is a "common theme" here since that might vary a lot 🤷 so I would say the template way is "safer" ?

[andrejbaran]

@jszwedko Now that I think of it I would go for the simple boolean. One can always create separate sinks (with same/similar configuration) and route logs to respective sinks if needed.

Since there wasn't a real need for this feature anyway (4 👍 including mine 😁 ) I will go for the boolean.