search

connectrpc / connect-es

The TypeScript implementation of Connect: Protobuf RPC that works.
https://connectrpc.com/
Apache License 2.0
1.39k stars 81 forks source link

Eliminate protobuf dependencies when using `useBinaryFormat: false` #1287

Closed kjvalencik closed 3 weeks ago

kjvalencik commented 4 weeks ago

Is your feature request related to a problem? Please describe.

I would like a smaller package when not using protobuf serialization.

Both the ConnectRPC protocol and grpc-web (application/grpc-web+json) support JSON encoding as an alternative to protobuf. In these schemes, most of the protobuf logic is not necessary. These libraries take up a significant portion of a package (67% of the 55kb in this example).

   ├ node_modules/@bufbuild/protobuf/dist/esm/private/json-format.js                    7.6kb   14.2%
   ├ node_modules/@bufbuild/protobuf/dist/esm/private/binary-format.js                  4.5kb    8.5%
   ├ node_modules/@bufbuild/protobuf/dist/esm/binary-encoding.js                        3.8kb    7.2%
   ├ node_modules/@bufbuild/protobuf/dist/esm/private/util-common.js                    2.6kb    4.8%
   ├ node_modules/@bufbuild/protobuf/dist/esm/google/protobuf/any_pb.js                 2.4kb    4.5%
   ├ node_modules/@bufbuild/protobuf/dist/esm/google/varint.js                          2.1kb    3.8%
   ├ node_modules/@bufbuild/protobuf/dist/esm/proto-int64.js                            1.4kb    2.7%
   ├ node_modules/@bufbuild/protobuf/dist/esm/extension-accessor.js                     1.1kb    2.1%
   ├ node_modules/@bufbuild/protobuf/dist/esm/message.js                                1.0kb    1.9%
   ├ node_modules/@bufbuild/protobuf/dist/esm/proto-base64.js                           950b     1.7%
   ├ node_modules/@bufbuild/protobuf/dist/esm/private/extensions.js                     794b     1.5%
   ├ node_modules/@bufbuild/protobuf/dist/esm/private/scalars.js                        720b     1.3%
   ├ node_modules/@bufbuild/protobuf/dist/esm/private/field-normalize.js                688b     1.3%
   ├ node_modules/@bufbuild/protobuf/dist/esm/private/field-list.js                     664b     1.2%
   ├ node_modules/@bufbuild/protobuf/dist/esm/private/enum.js                           631b     1.2%
   ├ node_modules/@bufbuild/protobuf/dist/esm/private/names.js                          619b     1.1%
   ├ node_modules/@bufbuild/protobuf/dist/esm/private/reflect.js                        618b     1.1%
   ├ node_modules/@bufbuild/protobuf/dist/esm/private/assert.js                         608b     1.1%
   ├ node_modules/@bufbuild/protobuf/dist/esm/private/message-type.js                   489b     0.9%
   ├ node_modules/@bufbuild/protobuf/dist/esm/scalar.js                                 461b     0.8%
   ├ node_modules/@bufbuild/protobuf/dist/esm/private/field.js                          442b     0.8%
   ├ node_modules/@bufbuild/protobuf/dist/esm/private/field-wrapper.js                  424b     0.8%
   ├ node_modules/@bufbuild/protobuf/dist/esm/proto3.js                                 322b     0.6%
   ├ node_modules/@bufbuild/protobuf/dist/esm/is-message.js                             285b     0.5%
   ├ node_modules/@bufbuild/protobuf/dist/esm/service-type.js                           275b     0.5%
   ├ node_modules/@bufbuild/protobuf/dist/esm/private/proto-runtime.js                  261b     0.5%

Describe the solution you'd like

I would like to rely solely on JSON.parse and JSON.stringify and eliminate protobuf from the dependency tree. Unfortunately, the protobuf package is scattered around the tree. This would likely require changes to the generated code.

While this is relevant to both Node and Web client implementations, it is more important for web.

Describe alternatives you've considered

I have tried various tree shaking and deadcode elimination techniques but have not had much success. Snipping code (i.e., replacing with stubs) is effective but cumbersome and prone to regressions.

Additional context

This is very similar--and possibly identical--to https://github.com/connectrpc/connect-es/issues/1030.

Thanks!

timostamm commented 3 weeks ago

This is not as simple as it may seem. A simple message serializes as expected with JSON.stringify, but bytes (Uint8Array), int64 (BigInt) and several other types and constructs do not. To serialize or parse between the Protobuf JSON format and the generated class, schema information is required.

Protobuf-ES V2 supports JSON types. They match the Protobuf JSON format, which allows us to potentially provide clients that accept and return these JSON types, and do not require schema information.

In the upcoming version 2 of Connect-ES (see release candidate here), we are updating the Connect packages to support Protobuf-ES version 2, but we do not decouple from the Protobuf JSON and binary formats yet (more details in https://github.com/connectrpc/connect-es/issues/1233#issuecomment-2355138379).

I'm closing this to consolidate in #1030. Thanks for the issue!

kjvalencik commented 3 weeks ago

Thanks for the quick reply! That makes perfect sense that it would need a pass to recursively transform all the typed arrays (plus any serialized i64). It would still end up being much simpler without all of the varint machinery, etc., but definitely not trivial! It makes sense to close in favor of that issue.

Thanks for the information on the RC. Cheers!