[question] message validation

[SpareShade]

Hey team Buf, we are loving what you are building, thank you sooo much!!!!

It was a breeze switching from @improbable-eng/grpc-web to connect-web. We are looking to transitioning to connect-go in the future (still using @improbable-eng's grpcweb).

The question here is regarding message validation. We are wondering if you are at all considering implementing message validation as a part of the connect ecosystem?

(Just for the background, we are early in the development process, and are currently evaluating envoyproxy/protoc-gen-validate for the backend and looking into colinhacks/zod with fabien0102/ts-to-zod for our frontend.)

[timostamm]

Hey @SpareShade, it looks like we are going to take up maintenance of protoc-gen-validate. See envoyproxy/protoc-gen-validate#616 for context. I don't think we would want to tie connect and PGV closely together, but we definitely want to make sure connect-go and connect-web work well with PGV.

connect-web uses the bases types generated by protobuf-es. Currently, there is no code generator for PGV and protobuf-es, but it should be feasible to implement one either with @bufbuild/protoplugin (you can see it in use here), or with PGVs code generators. There is also a third option: Instead of generating individual validation functions, there could be a single validation function that uses metadata in the generated code.

It's too early in the project to tell what's best, but we appreciate input from you! If you think one of the options above is preferable, let us know.

[fubhy]

I came here looking for the same thing (typescript-first runtime message validation with zod or similar).

I'll have a bit of time by the end of this week and meant to play around with connect-web and protobuf-es (currently, we are using protobuf-ts and have a bespoke zod based validation solution in place there). I'll give protoplugin a go then and see how far I can get. Will share any relevant progress here in case that's useful for others.

FYI: I'm thinking of generating the code with PGV in a way that further refines the generated types (if possible, haven't looked into it yet) according to the validation schema.

[timostamm]

Sounds great, Sebastian!

Please note that we're going change the plugin API a bit. The goal is that plugin authors only need to generate TypeScript, and we transpile JavaScript and TypeScript declaration files for you (unless you opt out by bringing your own). See https://github.com/bufbuild/protobuf-es/pull/228.

Also, reading custom options is a very manual process right now. We are going to provide a convenience function for this. Happy to help you get started in a discussion in this repo, or via Slack.

further refines the generated types

Letting the compiler know that a required message field is set after a validate(someMessage)? Yes, that would be really neat!

[derekperkins]

cc @tannerlinsley

[ghost]

Hello everyone!

@fubhy, if you need/want some help, I would be glad to work with you on that, I think it's a really good approach.

[fubhy]

Hey @johynpapin . I'm close to releaseing a first iteration. Will put it here in a few hours. Still got about ~100 test cases from the protoc-gen-validate test harness to go (likely not going to cover all of them initially). Will put it up here shortly: https://github.com/fubhy/protobuf-zod

[ghost]

OK that's really cool! Thanks!

[fubhy]

@johynpapin I pushed a first "work in progress" to that repository. Things are likely going to change quite a bit as I continue to experiment so be aware of that if you want to play around with it already.

I also haven't implemented any runtime bindings yet. Not sure how I want to integrate it with connect-web (client side validation) and connect-node (server side validation) yet (interceptors?).

I might add another generator that uses grpc service definitions to generate a schema registry and generic validation function (that can validate any message).

For now you have to do that manually (MyMessageSchema.parse(message)).

I am not particularly fond of the string based code generation and how I used it here ... I think I would rather like to print some kind of abstract, composed AST. Particularly with something like the zod API that is designed for chaining and composing function calls that would probably make the generator code quite a bit more digestable.

Anyways, it's a start ...

/cc @timostamm

[ghost]

Nice work @fubhy! I think that using some kind of IR is a good idea, this would potentially allow to integrate another validation library than Zod or to do something manually.

[fubhy]

Nice work @fubhy! I think that using some kind of IR is a good idea, this would potentially allow to integrate another validation library than Zod or to do something manually.

Thanks. Yeah. I'm working on that atm. ... Translating all the nuances & combinations of rules and how they are expected to behave into an intermediary structure. Eliminating invalid combinations and condensing overlapping rules / interpreting combinations of rules and handling implicit logic (e.g. combining lt, gt, gte and lte rules, etc.) ... And all other behavioral rules expected by PVG as detailed in the docs here for insstance: https://github.com/bufbuild/protoc-gen-validate/blob/main/validate/validate.proto#L81-L83

So that the following implementation then just has to iterate over a set of constraints and pretty much only concern itself with printing the output.

[rustysys-dev]

@fubhy Awesome work!!! And good to see that PGV is being extended for this usecase too!!! I am stopping by hoping to see if this issue has stalled or if there are still plans to get this out? Right now we are evaluating whether to use connect-web, and this is one of the pain points against something like trpc which offers integration with Zod.

[fubhy]

@rustysys-dev Hey. I still plan on getting this production ready at some point but after my experimenting with connect-web I'm now again working on some other things internally for a few weeks. I'll get back to PGV & Zod early February.