Closed bluemods closed 1 month ago
Thanks for the contribution @bluemods!
application/x-protobuffer
is fine, that's a great find, but unfortunately application/grpc
is not. It's been a while since I looked into this, but I think the situation is that Grpc does use protobuf, but the messages themselves aren't just raw protobuf: there's a grpc framing structure around the protobuf content, and we can't just render the whole message as protobuf without first parsing that part to extract the actual protobuf inside.
To handle that we'd need a separate formatter for grpc, which handles that outer layer. PRs welcome for that of course if you're interested (let me know and I can share some pointers) but it will be a fair bit more work.
Does that make sense? Happy to merge this anyway to handle that protobuffer
case if you drop the grpc line.
Upon doing some research, looks like you're right about application/grpc having a slightly different format. But it seems simple to fix:
Here's a sample GRPC request body in hex format:
00 00 00 00 21 08 57 12 02 08 02 1a 19 0a 17 62 61 72 64 2d 6e 6f 72 65 70 6c 79 40 67 6f 6f 67 6c 65 2e 63 6f 6d
To parse GRPC messages, there can be something added that discards the first 5 bytes then we are left with:
08 57 12 02 08 02 1a 19 0a 17 62 61 72 64 2d 6e 6f 72 65 70 6c 79 40 67 6f 6f 67 6c 65 2e 63 6f 6d
Which should then parse correctly as a normal protobuf message.
The reason I thought it was simple is because https://protobuf-decoder.netlify.app/ automatically handles this, and I had been pasting the requests there.
They solved it this way: https://github.com/pawitp/protobuf-decoder/blob/0f419b32f85a5f1f6ea9e9d982ef79bc37099e85/src/protobufDecoder.js#L25
To fix it, looks like you need to modify the input that goes to the protobuf parser by first removing the first 5 bytes or so. I don't use Javascript much so I'm not initially seeing the best place where I can do that. Let me know if you want to try that or if I should remove the application/grpc for now.
Thanks, that's good info. I found the actual spec for grpc encoding here: https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-HTTP2.md, and read around the topic a bit more generally too. I think adding proper support is actually not too tough, so let's do it! There's a few important points:
application/grpc+proto
), but also comes in a few other formats (e.g. application/grpc+json
) - so we need to check that (but I think assuming literally application/grpc
without a suffix is protobuf-based is reasonable)Does that make sense?
To fix it, looks like you need to modify the input that goes to the protobuf parser by first removing the first 5 bytes or so. I don't use Javascript much so I'm not initially seeing the best place where I can do that. Let me know if you want to try that or if I should remove the application/grpc for now.
To implement this, you'll need a new grpc
formatter (very very similar to the protobuf
formatter, but with this extra logic to handle the header bytes) and then we'll need the content types logic to select that formatter when it's appropriate.
The protobuf formatter is defined here: https://github.com/httptoolkit/httptoolkit-ui/blob/0989258cbdf0b9974e5a88bed4b97e5350faf715/src/services/ui-worker-formatters.ts#L76-L95
The actual protobuf logic that that uses is defined in a separate file here: https://github.com/httptoolkit/httptoolkit-ui/blob/main/src/util/protobuf.ts.
I would suggest:
stripGrpcHeader
method to that protobuf.ts file, which drops those bytes (but checks if compression is set or the length is obviously wrong) and then returns the remaining part of the buffer ready for parsing.grpc
property to the formatters, which calls your grpc-strip method first, and then does the same parsing as for normal protobuf (you might want to move any shared code into protobuf.ts en route)grpc
content type, and select it when appropriateDoes that make sense?
Ok I made the changes and so the way I did it is to specifically look for the application/grpc header (which is standard for the iOS and Android mobile clients as well as many others like Golang) and it will parse the message in the special manner as required by reading the first 5 bytes to get the length of the message. for me it is working as intended and parsing the messages.
Nice work @bluemods! This looks great. I've got a couple of other big changes just about to land in the UI right now, so I'm not going to merge this immediately, but I should be able to in just a few days. Watch this space :smile:
Adds 'application/x-protobuffer' and 'application/grpc'.
I commonly see these content-types in the wild, especially on iOS and Android devices. It would be nice to have the protobuf context menu option for these content types.