Questions about the DecryptedDataSent event

[aizazadi1]

I encountered a problem when using Https related agents of Titanium-Web-Proxy products. The data obtained in the DecryptedDataSent event is as follows (hexadecimal):

82 8A AD 47 F9 88 AD 47 F9 82 AD 47 FA 79 AD 46

It seems that this kind of data is not decrypted, or the decryption is wrong

The correct data obtained in DecryptedDataReceived is as follows (hexadecimal):

82 12 00 00 00 12 00 00 03 F2 00 00 00 00 01 6E AB 57 79 6B

I have confirmed that this data is my business data, which is very correct

I don't know what the problem is, what should I do, thank you!

[honfika]

Is there a Transfer-Encoding header, which is for example deflate, gzip or br?

The data should be decrypted, but not decompressed. So if it is gzip compressed, you should uncomress it. (Or use the BeforeRequest/Response events, the *DataSent/Received events are basically for counting the data size)

[aizazadi1]

Thank you very much for your reply. I tried to decompress it in gzip, deflate and brotli, but it failed. My data comes from websocket. Is there any particularity in this way? Thank you! The Datas： 82 8A EF 2D ED 45 EF 2D ED 4F EF 2D EE B4 EF 2C 82 8A 30 BA D9 0D 30 BA D9 07 30 BA DA FC 30 BB 82 8A 89 C2 A5 58 89 C2 A5 52 89 C2 A6 A9 89 C3 82 8A 10 25 E4 25 10 25 E4 2F 10 25 E7 D4 10 24 82 8A B3 94 FB B7 B3 94 FB BD B3 94 F8 46 B3 95 82 8A 87 E4 96 A2 87 E4 96 A8 87 E4 95 53 87 E5 82 8A E0 06 32 2D E0 06 32 27 E0 06 31 DC E0 07 82 8A 24 47 51 A0 24 47 51 AA 24 47 52 51 24 46 82 8A E1 43 D8 4C E1 43 D8 46 E1 43 DB BD E1 42 82 8A 67 A6 E8 13 67 A6 E8 19 67 A6 EB E2 67 A7

[aizazadi1]

In addition, I try to capture data in BeforeRequest and BeforeResponse, and the results are the same The code is as follows: private async Task onRequest(object sender, SessionEventArgs e) { ProcessSession(e); }

private void ProcessSession(SessionEventArgsBase session) { session.DataSent += (sender, args) => { var obj = (SessionEventArgsBase)sender; string url = obj.HttpClient.ConnectRequest?.Url; Console.WriteLine("onRequestDataSentUrl=" + url); byte[] bytes = new byte[args.Count]; Array.Copy(args.Buffer, args.Offset, bytes, 0, args.Count); LogUtil.WriteSendLog(HexMath.Bytes2Hex(bytes)); }; } Thank you very much for your reply！

[honfika]

You can get the requestbody with the following code in beforerequesthandler:

var data = await e.GetRequestBody();

Also please check the value of the following expression: e.HttpClient.ConnectRequest.TunnelType It can be a websocket.

e.HttpClient.ConnectRequest may be null (bot only when it is not a websocket and nor HTTPS)

[aizazadi1]

As you said, I can check e.HttpClient.ConnectRequest.TunnelType, which is WebSocket, but when I execute var data = await e.getrequestbody(); the program Throw exception

Request don't have a body. Please verify that this request is a Http POST/PUT/PATCH and request content length is greater than zero before accessing the body.

what should I do, Thank you very much for your reply！

[honfika]

Yes, since it is a websocket, it has no body, because it is a stream.

Then you can use the DataReceived events, but maybe websocket has a special protocol which should be transformed to get the real data. TWP is not able to do that.

[aizazadi1]

Thank you for your reply！ I don't think you understand me

If this is the case now, I can get the data by using the DataReceived events, and I can confirm that it is the decrypted data, but it is the data sent from the server to the client.

My requirement is to capture the plaintext data sent by the client to the server

[honfika]

Yes, that is the decrypted data. (Not SSL....)

You can read the WebSocket Framing Prpotocol here: https://tools.ietf.org/html/rfc6455#section-5.2

Lets check this message: 82 8A EF 2D ED 45 EF 2D ED 4F EF 2D EE B4 EF 2C

The first byte of your mesasges is 82 => 4 last bit is 2, which means binary frame: https://tools.ietf.org/html/rfc6455#page-65

2nd byte is 8A => mask = 1 length = 0xa = 10

Your messages length is 16, because 6 byte is the "header", 10 bytes real data

No extended payload length, because length is < 126 bytes

4 bytes masking key: EF 2D ED 45

remainnig bytes (maybe i'm wrong, this should be checked): EF xor EF => 0? 2D xor 2D => 0? ED xor ED => 0? 4F xor 45 => 0xa EF xor EF 2D xor 2D EE xor ED B4 xor 45 EF xor EF 2C xor 2D

[honfika]

Sorrry, I wrote DataReceived, but of course you need DataSent. But they are similar.

DataReceived => server to client DataSent => client to server

So the bytes that you wrote are not encypted, but it has a special format...

[honfika]

Maybe I'll add special events for websockets later, which does this transformation.

[aizazadi1]

Great, the problem should be solved, thank you very much! I'll try it now！！！Thanks again.

[honfika]

Ok. Note that the data in the DataSent/Received event is not guaranteed to be a complete websocket packet, since this event is called immediately after any data is received on the socket. So it can happen that the size in the websocket frame (2nd byte) is larger than the actual size in the event arguments.

[aizazadi1]

Thank you very much for your reminding. I tried to analyze the sent data package, but also found the problem you said. It has been solved at present, and the system is running normally. Thank you again!

[honfika]

When somebody else needs it: This feature was added to TWP, check example in the following file: https://github.com/justcoding121/Titanium-Web-Proxy/blob/4f43bd16c747de54f76c0651659b28002ddf5ffc/examples/Titanium.Web.Proxy.Examples.Basic/ProxyTestController.cs

[aizazadi1]

👍 Cow B，d=====(￣▽￣*)b

[jpsilv]

I'm facing the same issue, not sure if anyone is around here after 3 years but worth the try. @honfika could you give me a brief explanation of what to do when the message is compressed?

Looking at the messages in chrome, I got the following:

Accept-Encoding: gzip, deflate, br
Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits
Sec-WebSocket-Key: xgmcEPo+OTAPXh+9aDzKBA==

So I'm guessing my messages are compressed.

I have checked these values and none of them match the messages that chrome shows:

string test = frame.GetText()
string test2 = System.Text.Encoding.UTF8.GetString(frame.Data.ToArray());
string test3 = System.Text.Encoding.UTF8.GetString(e.Buffer);
string test4 = Convert.ToBase64String(frame.Data.ToArray());
string test5 = Convert.ToBase64String(e.Buffer);
string test6 = BitConverter.ToString(frame.Data.ToArray()).Replace("-", string.Empty);
string test7 = BitConverter.ToString(e.Buffer).Replace("-", string.Empty);
string test8 = BitConverter.ToString(frame.Data.ToArray());
string test9 = BitConverter.ToString(e.Buffer);
string test14 = DecodeMessage(frame.Data.ToArray());
string test15 = DecodeMessage(e.Buffer);

DecodeMessage code:

private String DecodeMessage(Byte[] bytes)
{
    String incomingData = String.Empty;
    Byte secondByte = bytes[1];
    Int32 dataLength = secondByte & 127;
    Int32 indexFirstMask = 2;
    if (dataLength == 126)
        indexFirstMask = 4;
    else if (dataLength == 127)
        indexFirstMask = 10;

    IEnumerable<Byte> keys = bytes.Skip(indexFirstMask).Take(4);
    Int32 indexFirstDataByte = indexFirstMask + 4;

    Byte[] decoded = new Byte[bytes.Length - indexFirstDataByte];
    for (Int32 i = indexFirstDataByte, j = 0; i < bytes.Length; i++, j++)
    {
        decoded[j] = (Byte)(bytes[i] ^ keys.ElementAt(j % 4));
    }

    return incomingData = Encoding.UTF8.GetString(decoded, 0, decoded.Length);
}

Also, I'm using the method WebSocketDataSentReceived from ProxyTestController.

Edit: I tried to decompress with this code but it just gave an exception 'System.IO.InvalidDataException' in System.IO.Compression.dll:

public static string Unzip(byte[] bytes)
{
    using (var msi = new MemoryStream(bytes))
    using (var mso = new MemoryStream())
    {
        using (var gs = new GZipStream(msi, CompressionMode.Decompress))
        {
            //gs.CopyTo(mso);
            CopyTo(gs, mso);
        }

        return Encoding.UTF8.GetString(mso.ToArray());
    }
}