search

dialohq / ocaml-grpc

gRPC library for OCaml
https://dialohq.github.io/ocaml-grpc
BSD 3-Clause "New" or "Revised" License
62 stars 9 forks source link

Support for gRPC compression #46

Open tmcgilchrist opened 1 year ago

tmcgilchrist commented 1 year ago

Provide a minimal implementation of compression using gzip (which is supported in Go providing a popular interoperability target). There is a general spec outlining how compression should work https://github.com/grpc/grpc/blob/master/doc/compression.md. The decompress library looks promising, it provides gzip and zlib plus it is fairly popular and implemented in pure OCaml.

Bonus feature would be to design an API such that compression is extensible and library users can provide their own compression algorithms.

joprice commented 9 months ago

I tried using a service with grpcurl but the underlying library it uses by default sends a grpc-accept-encoding: gzip header. This results in the error 406 (Not Acceptable); malformed header: missing HTTP content-type due to the check here https://github.com/dialohq/ocaml-grpc/blob/e066812702d4c2221fe85d39d1a78305ea7d7361/lib/grpc-eio/server.ml#L39.

According to https://grpc.io/docs/guides/compression/#compression-method-asymmetry-between-peers, in the case of an unsupported compression type, an unimplemented error should be returned. https://github.com/grpc/grpc-go/blob/5051eeae537cb2839dd499e1a63a141098a3a03a/server.go#L1316.

As for the compression itself, I took a stab at it and got it partly working, decompressing an inbound body https://github.com/dialohq/ocaml-grpc/compare/main...joprice:ocaml-grpc:gzipSupport?expand=1#diff-2592ab6cc5a62541ce6f428cc52bed5bc738dbbdfbb6be87042a0fb5bfe390b2R56. That branch also has the Unimplemented response on it, along with some other local changes I had made for error handling that I can clean up if this general approach makes sense.

tmcgilchrist commented 9 months ago

That would be great @joprice this library should interop correctly against other implementations. Have a look at the Typed RPC PR https://github.com/dialohq/ocaml-grpc/pull/55 that should be the new API going forward.

You might also be interested in https://github.com/tmcgilchrist/ocaml-grpc/tree/fixes which has the initial support for running tests against the Go gRPC bindings. That is what I would eventually like to have but as your found out there are plenty of corner cases where ocaml-grpc doesn't do the right thing.

joprice commented 9 months ago

I'll check both of those out. I actually just got the full end to end working https://github.com/dialohq/ocaml-grpc/compare/main...joprice:ocaml-grpc:gzipSupport?expand=1#diff-2592ab6cc5a62541ce6f428cc52bed5bc738dbbdfbb6be87042a0fb5bfe390b2R49-R69.

I added a codec type and the negotiation of client and server encodings. It combines the two together into a new codec to support asymmetric encoding. I added a parameter to the server v builder function that takes in a list of supported compression implementations. The implementations could be moved to separate libraries to avoid having to build things unnecessarily, and since it's just a record of functions, you can provide your own implementation easily.

joprice commented 9 months ago

Here's some more docs I found on compression. The last one looks especially useful as it has some nice flowcharts that can be used to validate the implementation.