Non-Binary Message Encoding (e.g. JSON)

[remko]

Hi,

The roadmap mentions non-binary message encoding as a roadmap item. Are there any further thoughts about this somewhere, apart from what is mentioned in the roadmap? Is there still interest in this, or do you think a better direction for this use case is to use gRPC JSON transcoding?

[sampajano]

Hi! Thanks for the question!

I've checked with the team and it seems that there's no concrete plans yet to add non-binary encodings.

gRPC JSON should be a possibility but again there's no active work on this yet :)

Sorry for the lack of progress but hope this answers your question. :)

[xxgreg]

I've been experimenting the connect web GRPC protocol. It's really nice. Perhaps there could be some collaboration on this?

See https://connect.build/docs/introduction/

And https://connect.build/docs/protocol/

Connect supports its own protocol: a straightforward, POST-only protocol that works over HTTP/1.1 and HTTP/2. It takes the best parts of gRPC and gRPC-Web, including streaming, and packages them into a protocol that's equally at home in browsers, monoliths, and microservices. The Connect protocol is what we think the gRPC protocol should be. By default, JSON- and binary-encoded Protobuf is supported. You can call our live demo service with cURL:

curl \
    --header "Content-Type: application/json" \
    --data '{"sentence": "I feel happy."}' \
    https://demo.connect.build/buf.connect.demo.eliza.v1.ElizaService/Say

[sampajano]

@xxgreg Thanks for sharing the pointers! We're aware of the solutions from connect.build, and i think a json encoding wouldn't be out of the question for us in the future either. We'll keep this in mind thanks again for sharing!

[dimo414]

I'm pretty surprised to discover that there's no support for a non-binary encoding, nor a clear plan to implement one. The gRPC-web spec specifically calls out JSON as a valid content type, and makes no mention of the fact that application/grpc-web+json is entirely hypothetical.

It would be hugely helpful to refresh both the gRPC-web specification and the roadmap (which mentions a JSON-like format but not JSON itself) to clarify what exists today and what may be implemented in the future. In particular if real JSON is not actually expected to ever be implemented application/grpc-web+json should be removed/deprecated.

[sampajano]

@dimo414 Thank you for the feedback!

I'm pretty surprised to discover that there's no support for a non-binary encoding, nor a clear plan to implement one.

Do you mind sharing what motivates you to look for a non-binary encoding?

It would be hugely helpful to refresh both the gRPC-web specification and the roadmap

Indeed it should be updated in the near term. Given the fact that protobuf-javascript is nearing sunset, we will need to find an alternative protobuf solution (e.g. protobuf.js), and would likely be adding more native JSON support. (Although that may or may not be related to wire encodings.)

[dimo414]

Do you mind sharing what motivates you to look for a non-binary encoding?

I was hoping to make debugging in the browser easier by being able to configure requests to use JSON instead of binary format. I had assumed, given the content type mentioned in the spec, that this would be something I could flip on/off fairly easily.

But I'm also just going off what's documented in the specification. I wouldn't have even considered this approach if it wasn't specified.

protobuf-javascript is nearing sunset

Wow, that's the first I've heard of that too. Could you resolve https://github.com/protocolbuffers/protobuf-javascript/issues/175 and update the project status to reflect this plan? There's no mention on the repo page that this library is being turned down.

we will need to find an alternative protobuf solution (e.g. protobuf.js)

I do not think protobuf.js is a viable alternative because it is non-compliant with the proto spec. Going off Buf's conformance tests my impression is that protobuf-ts / protobuf-ES are the best available options.

[sampajano]

I was hoping to make debugging in the browser easier by being able to configure requests to use JSON instead of binary format.

Aha thanks for explaining! It would be a good feature to have indeed :)

Wow, that's the first I've heard of that too. Could you resolve https://github.com/protocolbuffers/protobuf-javascript/issues/175 and update the project status to reflect this plan? There's no mention on the repo page that this library is being turned down.

Sorry, I didn't mean the project is planned to be turned down, as AFAIK there's no plan to do so. I was referring to how the project was getting more outdated and some major usability issues (e.g. code size) that are not being fixed.

Overall, our Closure dependency is getting outdated and hence we're looking for ways to modernize the project soon, and protobuf is just one major part of the equation.

I do not think protobuf.js is a viable alternative because it is non-compliant with the proto spec. Going off Buf's conformance tests my impression is that protobuf-ts / protobuf-ES are the best available options.

Yeah we're aware of that test. Although i think the test has previously put protobuf.js at a very low compatibility score (37%), but my colleague has recently found a bug in the test and now the score has increased to 75% already. For that reason we don't really put too much confidence on that score alone :)

Another reason to consider protobuf.js is because it's used by gRPC Node, and we're motivated to make the 2 solutions converge more in the future.

[dimo414]

I know this is tangential to the issue, but since we're talking about it I'd like to suggest not using protobuf.js. Since November we have migrated our application off protobuf.js and onto protobuf-ts and the difference is night and day. I can go into more detail in a discussion if it would be helpful, but we have resolved numerous longstanding bugs that were due to broken or incompatible behavior in protobuf.js.