Update Protobuf Content Types

[bluemods]

Adds 'application/x-protobuffer' and 'application/grpc'.

I commonly see these content-types in the wild, especially on iOS and Android devices. It would be nice to have the protobuf context menu option for these content types.

[CLAassistant]

All committers have signed the CLA.

[pimterry]

Thanks for the contribution @bluemods!

application/x-protobuffer is fine, that's a great find, but unfortunately application/grpc is not. It's been a while since I looked into this, but I think the situation is that Grpc does use protobuf, but the messages themselves aren't just raw protobuf: there's a grpc framing structure around the protobuf content, and we can't just render the whole message as protobuf without first parsing that part to extract the actual protobuf inside.

To handle that we'd need a separate formatter for grpc, which handles that outer layer. PRs welcome for that of course if you're interested (let me know and I can share some pointers) but it will be a fair bit more work.

Does that make sense? Happy to merge this anyway to handle that protobuffer case if you drop the grpc line.

[bluemods]

Upon doing some research, looks like you're right about application/grpc having a slightly different format. But it seems simple to fix:

Here's a sample GRPC request body in hex format:

00 00 00 00 21 08 57 12 02 08 02 1a 19 0a 17 62 61 72 64 2d 6e 6f 72 65 70 6c 79 40 67 6f 6f 67 6c 65 2e 63 6f 6d

To parse GRPC messages, there can be something added that discards the first 5 bytes then we are left with:

08 57 12 02 08 02 1a 19 0a 17 62 61 72 64 2d 6e 6f 72 65 70 6c 79 40 67 6f 6f 67 6c 65 2e 63 6f 6d

Which should then parse correctly as a normal protobuf message.

The reason I thought it was simple is because https://protobuf-decoder.netlify.app/ automatically handles this, and I had been pasting the requests there.

They solved it this way: https://github.com/pawitp/protobuf-decoder/blob/0f419b32f85a5f1f6ea9e9d982ef79bc37099e85/src/protobufDecoder.js#L25

[bluemods]

To fix it, looks like you need to modify the input that goes to the protobuf parser by first removing the first 5 bytes or so. I don't use Javascript much so I'm not initially seeing the best place where I can do that. Let me know if you want to try that or if I should remove the application/grpc for now.

[pimterry]

Thanks, that's good info. I found the actual spec for grpc encoding here: https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-HTTP2.md, and read around the topic a bit more generally too. I think adding proper support is actually not too tough, so let's do it! There's a few important points:

• GRPC can be protobuf (application/grpc+proto), but also comes in a few other formats (e.g. application/grpc+json) - so we need to check that (but I think assuming literally application/grpc without a suffix is protobuf-based is reasonable)
• The 5 bytes at the start are actually an "is this compressed" 1/0 byte plus a 4-byte length. To handle this, we'd need to check that first byte as well, and either decompress body if required (possible but mildly complicated) or just throw an error to start with like "Compressed gRPC is not supported". We should probably also parse and check the length just try to catch other obvious issues (just the same way as in the code you linked above).

Does that make sense?

To fix it, looks like you need to modify the input that goes to the protobuf parser by first removing the first 5 bytes or so. I don't use Javascript much so I'm not initially seeing the best place where I can do that. Let me know if you want to try that or if I should remove the application/grpc for now.

To implement this, you'll need a new grpc formatter (very very similar to the protobuf formatter, but with this extra logic to handle the header bytes) and then we'll need the content types logic to select that formatter when it's appropriate.

The protobuf formatter is defined here: https://github.com/httptoolkit/httptoolkit-ui/blob/0989258cbdf0b9974e5a88bed4b97e5350faf715/src/services/ui-worker-formatters.ts#L76-L95

The actual protobuf logic that that uses is defined in a separate file here: https://github.com/httptoolkit/httptoolkit-ui/blob/main/src/util/protobuf.ts.

I would suggest:

• Add a stripGrpcHeader method to that protobuf.ts file, which drops those bytes (but checks if compression is set or the length is obviously wrong) and then returns the remaining part of the buffer ready for parsing.
• Add a new grpc property to the formatters, which calls your grpc-strip method first, and then does the same parsing as for normal protobuf (you might want to move any shared code into protobuf.ts en route)
• Modify content-types to add the new grpc content type, and select it when appropriate

Does that make sense?

[bluemods]

Ok I made the changes and so the way I did it is to specifically look for the application/grpc header (which is standard for the iOS and Android mobile clients as well as many others like Golang) and it will parse the message in the special manner as required by reading the first 5 bytes to get the length of the message. for me it is working as intended and parsing the messages.

[pimterry]

Nice work @bluemods! This looks great. I've got a couple of other big changes just about to land in the UI right now, so I'm not going to merge this immediately, but I should be able to in just a few days. Watch this space :smile: