Support http body data validation for http_server source

[IFtech-A]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

We use vector agents as event receivers from different devices, each developed by separate teams. Adding HTTP body validation ensures data integrity for events. It allows us to respond with a bad request status for events that do not match the receiver's custom data validation.

Attempted Solutions

No response

Proposal

It would be possible, if we add a field for http_server with a VRL source, and execute it after body decoding completes.

References

No response

Version

No response

[jszwedko]

What sorts of validations are you thinking of @IFtech-A ?

[IFtech-A]

Thank you for your response!

Validation of fields’ existence, data type checks, fields’ data type dependent validations(min max values, date format checking, etc.). If validations fail respond with bad request status.

[jszwedko]

Gotcha! As a workaround, you could consider spinning up a separate HTTP server receive the data and do this validation before forwarding the data to Vector. I suggest this because I think it'll be a while before Vector supports something like this natively 🤔

[IFtech-A]

What if I try it and make a pull request? Would that be okay ?

[jszwedko]

What if I try it and make a pull request? Would that be okay ?

I think we'd be open to this as a general concept, but this is the first time we'd be introducing this sort of functionality into Vector and so it would likely involve a decent amount of discussion around what the correct user experience should be. I think the simplest and probably least contentious option would be to add JSON schema validation to the json decoding. Would that satisfy your use case?

[IFtech-A]

That won't be enough for our case, we will put another API for now then. Thank you for help. You may close the issue if you think it won't be helpful.

[jszwedko]

Gotcha, yeah, if JSON schema isn't sufficient, I think we'd need to pursue an RFC proposing another approach since there is likely to be more discussion. I'll mark this as an "idea" and we can see if we get additional use-cases from others.