how to decrypt grpc-web-text ?

[invisibleDesigner]

[problem] I send a grpc-web-text to server successfully, but payload has encrypted, I want to decrypt this so that I can do api test [try] I have used base64 decoding, but I failed [expected] the encrypt function and the decrypt function

[stanley-cheung]

The payload is base64-encoded. So the first step is to base64-decode it. After that, you get a series of bytes that's arranged in the "grpc-web" wire format, which is spec'ed out here: https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-WEB.md#protocol-differences-vs-grpc-over-http2.

So in general it goes "marker" "4 bytes denoting length" "X bytes of data / trailer", and repeat.

[invisibleDesigner]

I have base64-decode, but I can't continue decode, what should I do next? @stanley-cheung

[stanley-cheung]

Let me give an example. So let's say I made a grpc-web request in my app and now I look at the response from the developer console

You see this gibberish string that starts with "AAAAAAcKBWhlbGxvgAAAAehncnBjLX....". That was the entire grpc-web response.

As I mentioned in the previous reply, the grpc-web response is base64-encoded. So let's base64 decode it to see what that is (and since the result of base64-decode, in this case, are binary bytes, let's pass it to the utility xxd to see exactly the value of each bytes):

echo "AAAAAAcKBWhlbGxvgAAAAehncnBjLXN0YXR1czowDQpncnBjLW1lc3NhZ2U6T0sNCnNlYy1mZXRjaC1tb2RlOmNvcnMNCngtdXNlci1hZ2VudDpncnBjLXdlYi1qYXZhc2NyaXB0LzAuMQ0KY3VzdG9tLWhlYWRlci0xOnZhbHVlMQ0KdXNlci1hZ2VudDpNb3ppbGxhLzUuMCAoWDExOyBMaW51eCB4ODZfNjQpIEFwcGxlV2ViS2l0LzUzNy4zNiAoS0hUTUwsIGxpa2UgR2Vja28pIENocm9tZS83Ny4wLjM4NjUuOTAgU2FmYXJpLzUzNy4zNg0KYWNjZXB0OmFwcGxpY2F0aW9uL2dycGMtd2ViLXRleHQNCngtZ3JwYy13ZWI6MQ0Kb3JpZ2luOmh0dHA6Ly9sb2NhbGhvc3Q6ODA4MQ0Kc2VjLWZldGNoLXNpdGU6c2FtZS1zaXRlDQpyZWZlcmVyOmh0dHA6Ly9sb2NhbGhvc3Q6ODA4MS9lY2hvdGVzdC5odG1sDQphY2NlcHQtbGFuZ3VhZ2U6ZW4tVVMsZW47cT0wLjkNCngtZm9yd2FyZGVkLXByb3RvOmh0dHANCngtcmVxdWVzdC1pZDo5OTM2YmVjMS0zZjVmLTRkZGMtYjg1OS1iZDAxZTdjOGQ1YjUNCg==" | base64 -d | xxd
00000000: 0000 0000 070a 0568 656c 6c6f 8000 0001  .......hello....
00000010: e867 7270 632d 7374 6174 7573 3a30 0d0a  .grpc-status:0..
00000020: 6772 7063 2d6d 6573 7361 6765 3a4f 4b0d  grpc-message:OK.
00000030: 0a73 6563 2d66 6574 6368 2d6d 6f64 653a  .sec-fetch-mode:
00000040: 636f 7273 0d0a 782d 7573 6572 2d61 6765  cors..x-user-age
00000050: 6e74 3a67 7270 632d 7765 622d 6a61 7661  nt:grpc-web-java
00000060: 7363 7269 7074 2f30 2e31 0d0a 6375 7374  script/0.1..cust
00000070: 6f6d 2d68 6561 6465 722d 313a 7661 6c75  om-header-1:valu
00000080: 6531 0d0a 7573 6572 2d61 6765 6e74 3a4d  e1..user-agent:M
00000090: 6f7a 696c 6c61 2f35 2e30 2028 5831 313b  ozilla/5.0 (X11;
000000a0: 204c 696e 7578 2078 3836 5f36 3429 2041   Linux x86_64) A
000000b0: 7070 6c65 5765 624b 6974 2f35 3337 2e33  ppleWebKit/537.3
000000c0: 3620 284b 4854 4d4c 2c20 6c69 6b65 2047  6 (KHTML, like G
000000d0: 6563 6b6f 2920 4368 726f 6d65 2f37 372e  ecko) Chrome/77.
000000e0: 302e 3338 3635 2e39 3020 5361 6661 7269  0.3865.90 Safari
000000f0: 2f35 3337 2e33 360d 0a61 6363 6570 743a  /537.36..accept:
00000100: 6170 706c 6963 6174 696f 6e2f 6772 7063  application/grpc
00000110: 2d77 6562 2d74 6578 740d 0a78 2d67 7270  -web-text..x-grp
00000120: 632d 7765 623a 310d 0a6f 7269 6769 6e3a  c-web:1..origin:
00000130: 6874 7470 3a2f 2f6c 6f63 616c 686f 7374  http://localhost
00000140: 3a38 3038 310d 0a73 6563 2d66 6574 6368  :8081..sec-fetch
00000150: 2d73 6974 653a 7361 6d65 2d73 6974 650d  -site:same-site.
00000160: 0a72 6566 6572 6572 3a68 7474 703a 2f2f  .referer:http://
00000170: 6c6f 6361 6c68 6f73 743a 3830 3831 2f65  localhost:8081/e
00000180: 6368 6f74 6573 742e 6874 6d6c 0d0a 6163  chotest.html..ac
00000190: 6365 7074 2d6c 616e 6775 6167 653a 656e  cept-language:en
000001a0: 2d55 532c 656e 3b71 3d30 2e39 0d0a 782d  -US,en;q=0.9..x-
000001b0: 666f 7277 6172 6465 642d 7072 6f74 6f3a  forwarded-proto:
000001c0: 6874 7470 0d0a 782d 7265 7175 6573 742d  http..x-request-
000001d0: 6964 3a39 3933 3662 6563 312d 3366 3566  id:9936bec1-3f5f
000001e0: 2d34 6464 632d 6238 3539 2d62 6430 3165  -4ddc-b859-bd01e
000001f0: 3763 3864 3562 350d 0a                   7c8d5b5..

It still has a whole bunch of stuff in it. But let's break it down into the "marker" "4 bytes denoting length" "X bytes of data / trailer" format I mentioned in the previous comment.

The response starts with a marker byte 00. This means what's coming next is a "data" frame.

The next 4 bytes are 00 00 00 07. So that means the length of the frame is 7 bytes.

So let's grab the next 7 bytes: 0a 05 68 65 6c 6c 6f. This is effectively a protobuf message encoded in the binary format. We can use protoc --decode_raw to see what the protobuf message actually is

~$ echo -n -e '\x0a\x05\x68\x65\x6c\x6c\x6f' | protoc --decode_raw
1: "hello"

So it means that 0a 05 68 65 6c 6c 6f is a protobuf message with 1 field, field number = 1, and it's a string with a value "hello".

So we are done with the first frame.

Now the next byte is 80. This means what's coming next is a "trailer" frame.

The next 4 bytes are 00 00 01 e8. 1e8 in hex = 488 in dec. So that means, the next 488 bytes are all content of the "trailer" frame. I am not going to go into the details of the trailer frame.

But you see that's how, in general, the grpc-web wire protocol actually looks like. And as I have mentioned, these are all covered in the grpc-web spec

[stanley-cheung]

By the way, in your original message, here's how the response decoded to:

$ echo "AAAAAAwIARDABxoFL3Rlc3Q=" | base64 -d | xxd                         
00000000: 0000 0000 0c08 0110 c007 1a05 2f74 6573  ............/tes                                           
00000010: 74                                       t

So there are 0c = 12 bytes for the data frame

Grabbing the next 12 bytes: 08 01 10 c0 07 1a 05 2f 74 65 73 74, decoded to:

$ echo -n -e '\x08\x01\x10\xc0\x07\x1a\x05\x2f\x74\x65\x73\x74' | protoc --decode_raw                                                                                                    
1: 1                                                                                                          
2: 960                                                                                                        
3: "/test"

[poriam]

@stanley-cheung Excuse me for asking questions here. Can we opt-out of using base64? What is wrong with simply sending/receiving normal text? The doc says support text streams (e.g. base64) in order to provide cross-browser support (e.g. IE-10)

What if we don't need that text streams?

[avbenavides]

Try this to decode the whole payload in one go. Have noticed that request offset is 4 and reponse is 6.

$ echo "AAAAAAwIARDABxoFL3Rlc3Q=" | base64 -d | tail -c +6 | protoc --decode_raw
1: 1
2: 960
3: "/test"

[floydjones1]

@stanley-cheung Can you explain how the encoding process would work? Like how would we go from the output example

1: 1
2: 960
3: "/test"

to "AAAAAAwIARDABxoFL3Rlc3Q="

[floydjones1]

FYI I built a tool to help decode grpc web text into a human readable format. I would love to implement the reverse from human readable into grpc-web-text. Looking forward to your response! https://github.com/floydjones1/grpcwebtext-parser

[efossvold]

FYI I built a tool to help decode grpc web text into a human readable format. I would love to implement the reverse from human readable into grpc-web-text. Looking forward to your response! https://github.com/floydjones1/grpcwebtext-parser

@floydjones1 Thank you for creating the grpcwebtext-parser, it's been very helpful for me. I forked it to grpcwebtext-decoder which also parses trailing headers.

If you want to convert from protobuf objects to grpc-web-text (base64) you can use the encodeProtobuf function below.

const toBytesInt32 = (num: number): Uint8Array => {
  // an Int32 takes 4 bytes
  const arr = new ArrayBuffer(4)
  const view = new DataView(arr)
  // byteOffset = 0; litteEndian = false
  view.setUint32(0, num, false)
  return new Uint8Array(arr)
}

const createHeader = (name: string, value: string): Buffer => {
  const buffers: Array<number> = []
  for (const char of name) {
    buffers.push(char.charCodeAt(0))
  }
  buffers.push(':'.charCodeAt(0))
  for (const char of value) {
    buffers.push(char.charCodeAt(0))
  }
  buffers.push('\r'.charCodeAt(0))
  buffers.push('\n'.charCodeAt(0))
  return Buffer.from(buffers)
}

export const encodeProtobuf = <
  GRPCReply extends { serializeBinary: () => Uint8Array },
>(
  reply: GRPCReply,
): string => {
  const replyBuffer = reply.serializeBinary()
  const replyLength = toBytesInt32(replyBuffer.length)

  const headers = Buffer.from([
    ...createHeader('grpc-status', '0'),
    ...createHeader('grpc-message', 'OK'),
  ])
  const headersLength = toBytesInt32(headers.length)

  const response = Buffer.from([
    0, // mark response start ("data" frame coming next)
    ...replyLength,
    ...replyBuffer,
    128, // end of "data" frame, trailer frame coming next
    ...headersLength,
    ...headers,
  ])

  return response.toString('base64')
}

function main(): void {
  // UserPB is protobuf object created with protoc-gen-grpc
  // Any protobuf object with a serializeBinary function 
  // which returns an Uint8Array will do
  const reply = new UserPB.SetUserReply()
  reply.setUserId('mock-user-id')
  reply.setFirstName('foo')
  reply.setLastName('bar')
  const encoded = encodeProtobuf(reply)
  console.log(encoded)
}

main()

[Andrew713]

@stanley-cheung sorry for bothering you in a closed issue, but you mentioned:

The response starts with a marker byte 00. This means what's coming next is a "data" frame.

Now the next byte is 80. This means what's coming next is a "trailer" frame.

Where can I get this information?

I followed the link you provided at the bottom of your reply but didn't manage to find any explicit "legend".

[sampajano]

@stanley-cheung sorry for bothering you in a closed issue, but you mentioned:

The response starts with a marker byte 00. This means what's coming next is a "data" frame.

Now the next byte is 80. This means what's coming next is a "trailer" frame.

Where can I get this information?

I followed the link you provided at the bottom of your reply but didn't manage to find any explicit "legend".

@Andrew713 Hey Andrew if you search for "Message framing" in the page you linked, you should see the following info:

8th (MSB) bit of the 1st gRPC frame byte

0: data
1: trailers
  10000000b: an uncompressed trailer (as part of the body)
  10000001b: a compressed trailer

where 10000000b translates to 80 as a hex number.

Hope that answers your question :)

[Andrew713]

@sampajano Hi! Thanks, that makes sense now, however I am still stuck with decoding

My hex representation of the message starts as:

00000000: 0100 0001 4d1f efbf bd08 0000 0000 0000  ....M...........
00000010: 006d 50ef bfbd 4e02 4114 efbf bd3e 04ef  .mP...N.A....>..

etc.

From what I managed to find out, the first byte 01 (aka compressed-flag) means that the message was compressed -> I need to refer to grpc-encoding header (based on my yet poor knowledge it also seems to explain the "......M.......mP...N.A....>.." nonsense).

After that 00 00 01 4d indicates the length of 333 bytes of actual data, starting with 1f. But I still have to decompress it somehow to proceed.

If you could confirm my understanding or point out mistakes, it would be great!

[sampajano]

@Andrew713 Hi Andrew, since i haven't worked closely with the hex encodings, i won't be able to answer your questions easily.

My hex representation of the message starts as:

I'm not sure if this is the full response you're getting. If you could do the following like in the example above, maybe it would be easier to follow.

echo "[YOUR_FULL_RESPONSE_TEXT]" | base64 -d | xxd

---

Also, i'm wondering the goal of your effort. Are you trying to debug some bugs in the grpc-web client? Or only for the interest of understanding the details?

The best reference would be the grpc-web source code, which you can try to debug using the echo example (you can debug better when using "development" mode in the webpack config).

You could also consider using the gRPC-Web Dev Tools plugin (github), which many found useful for debugging the grpc-web responses.

Hope this helps :)

[djsubstance]

thx folks you are all helpful!