drwetter / testssl.sh

Testing TLS/SSL encryption anywhere on any port
https://testssl.sh
GNU General Public License v2.0
7.91k stars 1.02k forks source link

Need help in adding custom code to send Application(HTTP) Encrypted Traffic inside tls_sockets() API #2178

Closed srikr closed 2 years ago

srikr commented 2 years ago

Which version are you referring to 3.0.x or 3.1dev? (please check also how old your version is compare to the ones here) Hi, I am using 3.1dev testssl.sh version where in which I am trying to Test TLS Fingerprint for BOT. I am impressed with the SSL Testing options this tool provides only thing is on top of all the SSL Negative/Positive Testing I wanted HTTP Application Encrypted Traffic to be sent using the same Sockets. Below are something I was trying to do to achieve the same but failed: a) Initially inside tls_socket() function I tried to call socksend with HTTP App Data, it did send but the problem is it is sending in Plain Text resulting in Failure b) Then I tried to traverse the code and see to calculate session keys based on the Server Hello Packet what my Server Exchanges it is not able to do following due to no handshake_secret and msg_transcript getting derived: master_secret="$(derive-master-secret "$cipher" "$handshake_secret")" master_traffic_keys="$(derive-application-traffic-keys "$cipher" "$master_secret" "$msg_transcript" server)" APP_TRAF_KEY_INFO="$tls_version $cipher $master_traffic_keys 0 " master_traffic_keys="$(derive-application-traffic-keys "$cipher" "$master_secret" "$msg_transcript" client)" APP_TRAF_KEY_INFO+="$master_traffic_keys 0" c) I also tried to use send_app_data() which said(# Send application data over a TLS 1.3 channel that has already been created.) it can be only use with TLS1.3 and not TLS1.2 so I dropped. Even it is only for TLS1.3 I tried to use that but it needs following parameter as input read -r tls_version cipher server_key server_iv server_seq client_key client_iv client_seq <<< "$APP_TRAF_KEY_INFO" which I am not able to derive result in failure. I am not sure how to proceed after this now, It would be great if I can get some help in achieving the same. I am also attaching TCPDump [vs_virtualservice-abd75ca8-f7cb-4c24-bd19-a937d920b8d5.1651833023.zip] of the grease ciphersuite tests(TLS Stream 18 in wireshark or client hello packet number 348) for your reference (Note: There are few places, tool sends HTTP Encrypted Traffic but not using same TLS sockets though rather uses separate openssl commands which uses different TLS Params but I wanted Encrypted Application Traffic to be sent in all the places through the same sockets and same TLS Params during running the test like grease cipher, grease extension and other SSL Vulnerabilities etc)

dcooper16 commented 2 years ago

Hello @srikr,

It's not clear to me what it is you are trying to accomplish, but it doesn't seem to be in line with the way that testssl.sh works. In most cases, testssl.sh sends a CientHello and then just wants a small amount of information about the server's response (e.g., what protocol version or cipher suite did the server choose). For this reason, testssl.sh never completes the TLS handshake and does not derive session keys. So, sending application traffic would not be a possibility. As you noted, in the few places where testssl.sh needs to send application traffic, it uses OpenSSL.

In the case of TLS 1.3, a lot of the server's response is encrypted. With TLS 1.2 and earlier, tls_sockets() can easily extract the extensions and the certificates that the server sends, but this is all encrypted in TLS 1.3. So, I wrote a lot of code to derive the traffic keys for a TLS 1.3 handshake so that tls_sockets() could decrypt the extensions and certificates. (Extending this to work with TLS 1.2 or earlier would be too much work, since there are so many different options involved).

Once I had written the code for TLS 1.3 to derive the keys and decrypt the extensions and certificates, I went one step further and wrote code to complete the handshake and then send application data, but this code isn't yet being used. Also, since tls_sockets() is relatively slow, it does the least work possible. The application traffic keys are only derived if data needs to be decrypted, and the handshake is only completed if the call to tls_sockets() specifically requests that. At the moment, with one exception, no call to tls_sockets() asks for the handshake to be completed, and my guess would be that most do not call for the session keys to be derived. So, sending application data with each call to tls_sockets() would be a major change to the way that the code currently works.

I also don't understand the reason you want to send this data every time. If the purpose of a call to tls_sockets() is to find out if a particular cipher suite is supported or to obtain the server's certificate, the information needed is obtained before the handshake is complete, so what benefit is there in completing the handshake and then sending application data?

In any case, the best place to look is the only place at the moment in testssl.sh that attempts to send application data over TLS, service_detection(). This is a place where application does need to be send. Ideally OpenSSL is used, but if the server only supports TLS 1.3 and OpenSSL does not support TLS 1.3, then it tries using tls_sockets(). It might not work (the code does not work reliably at the moment), but it's better than not even trying. A short snippet of the code is as follows:

               tls_sockets "04" "$TLS13_CIPHER" "all+" "" "" false
               if [[ $? -eq 0 ]]; then
                    plaintext="$(tm_out "$GET_REQ11" | hexdump -v -e '16/1 "%02X"')"
                    plaintext="${plaintext%%[!0-9A-F]*}"
                    send_app_data "$plaintext"
                    if [[ $? -eq 0 ]]; then
                         receive_app_data true
                         [[ $? -eq 0 ]] || > "$TMPFILE"
                    else
                         > "$TMPFILE"
                    fi
                    send_close_notify "$DETECTED_TLS_VERSION"
               fi

If receive_app_data() is successful, the received data will be in $TMPFILE.

Note that in order for this to work, the server must support TLS 1.3, and in the call to tls_sockets() the third parameter must be all+ and the sixth parameter must be false. This call to tls_sockets() in service_detection() is the only one in testssl.sh that specifies these parameters in this way.

srikr commented 2 years ago

Thanks @dcooper16 for the detailed response. Below is the test I am trying to do:

The above suggestion what you are giving I looked at that as well.. the problem there is, it can be used only for TLS1.3 and not for TLS1.2. As you said I can give it a try atleast with TLS1.3 and see.. The sad part is Many of the Customer Client support only TLS1.2 and I have to test for both TLS1.1 and TLS1.2. Basically in the case of TLS1.2 I want to solve the below two TLS sessions to 1 TLS Sessions: 1) tls_socket API does 1 TLS handshake and close the connection 2) openssl s_client opens other TLS connection with different TLS parameter(Cipher, Extensions, Supported Group, Curve Properties and so on) It would be great if you have any suggestion to implement this or how to derive master keys and encrypt the application data for me to kick start for TLS1.2 as well.

Note: I got the intent of testssl.sh but I wan't to leverage all those tests for these kind of feature testing just by adding a patch to it.

srikr commented 2 years ago

Hi @dcooper16 I tried to mimic the same code you copied above by commenting existing code for greased cipher test:

if "$normal_hello_ok"; then
          list=""
          for ciph in "${grease_cipher_suites[@]}"; do
               list+=", $ciph"
          done
          #tls_sockets "$proto" "${list:2}, $selected_cipher_hex, 00,ff" "all"
          #success=$?
          #if [[ $success -ne 0 ]] && [[ $success -ne 2 ]]; then
          #     prln_svrty_medium " Server fails if ClientHello contains unrecognized cipher suite values."
          #     fileout "$jsonID" "MEDIUM" "Server fails if ClientHello contains unrecognized cipher suite values."
          #     bug_found=true
          #fi
          tls_sockets "04" "${list:2}, $TLS13_CIPHER, 00,ff" "all+" "" "" false
          if [[ $? -eq 0 ]]; then
               plaintext="$(tm_out "$GET_REQ11" | hexdump -v -e '16/1 "%02X"')"
               plaintext="${plaintext%%[!0-9A-F]*}"
               send_app_data "$plaintext"
               if [[ $? -eq 0 ]]; then
                  receive_app_data true
                  [[ $? -eq 0 ]] || > "$TMPFILE"
               else
                  > "$TMPFILE"
               fi
               send_close_notify "$DETECTED_TLS_VERSION"
          else
               > "$TMPFILE"
          fi
     fi

The problem I see here is the Application Data is not getting encrypted inside send_app_data rather it is going as Plain Text PUSH,ACK Packet and the reason I think is the variable $APP_TRAF_KEY_INFO is empty(no info about any of these Parameters(tls_version cipher server_key server_iv server_seq client_key client_iv client_seq)) read -r tls_version cipher server_key server_iv server_seq client_key client_iv client_seq <<< "$APP_TRAF_KEY_INFO"

Any Thoughts on how to populate this variable $APP_TRAF_KEY_INFO to make this work. FYI Below is the Command I am running to Trigger Grease Tests: ./testssl.sh -g --debug 6 172.16.82.4:443 | tee logs/consolelogs.txt

Plain Text: 474554202F70736D2F4745542F676574312E7068703F6E616D653D61626320485454502F312E310D0A486F73743A203137322E31362E38322E340D0A557365722D4167656E743A206162630D0A4163636570742D456E636F64696E673A206964656E746974790D0A4163636570743A20746578742F2A0D0A436F6E6E656374696F6E3A20436C6F73650D0A0D0A

Note: Above is the Plain Text that I am sending to send_app_data and after all the below logic:

aad="170303$(printf "%04X" "$(( ${#plaintext}/2 + tag_len + 1 ))")"
     if "$include_headers"; then
          res="$(sym-encrypt "$cipher" "$client_key" "$(get-nonce "$client_iv" $client_seq)" "${plaintext}17" "$aad")"
     else
          res="$(sym-encrypt "$cipher" "$client_key" "$(get-nonce "$client_iv" $client_seq)" "${plaintext}17" "")"
     fi
     debugme echo "TLS Version = $tls_version, Cipher = $cipher, Client Key = $client_key"
     debugme echo "Encrypted Value = $?"
     debugme echo "Response Value = $res"
     #[[ $? -eq 0 ]] || return 1
     client_seq+=1
     APP_TRAF_KEY_INFO="$tls_version $cipher $server_key $server_iv $server_seq $client_key $client_iv $client_seq"

     res="$aad$res"
     len=${#res}
     data=""
     for (( i=0; i < len; i+=2 )); do
          data+=",x${res:i:2}"
     done

the actual data that is getting sent is below which is completely different: ,x17,x03,x03,x00,x9E

dcooper16 commented 2 years ago

Hello @srikr,

There really isn't enough information here for me to understand why things are not working. I would suggest running testssl.sh in debug mode and then working your way though the /tmp/testssl-....log file to try to figure out why it isn't working.

srikr commented 2 years ago

Sorry @dcooper16 My Bad, I thought I will just put my observation as you might fully know the code. Please find attached all the Logs under /tmp/testssl.j1YYVN and screen console logs. alllogs.tar.gz consolelogs.txt

Let me know if you need any more info. Below is the code change I did for testing purpose

# Check that server ignores unrecognized cipher suite values
     # see https://datatracker.ietf.org/doc/draft-ietf-tls-grease
     if "$normal_hello_ok"; then
          list=""
          for ciph in "${grease_cipher_suites[@]}"; do
               list+=", $ciph"
          done
          #tls_sockets "$proto" "${list:2}, $selected_cipher_hex, 00,ff" "all"
          #success=$?
          #if [[ $success -ne 0 ]] && [[ $success -ne 2 ]]; then
          #     prln_svrty_medium " Server fails if ClientHello contains unrecognized cipher suite values."
          #     fileout "$jsonID" "MEDIUM" "Server fails if ClientHello contains unrecognized cipher suite values."
          #     bug_found=true
          #fi
          tls_sockets "04" "${list:2}, $TLS13_CIPHER, 00,ff" "all+" "" "" false
          if [[ $? -eq 0 ]]; then
               data="GET /psm/GET/get1.php?name=abc HTTP/1.1\r\nHost: $NODE\r\nUser-Agent: "abc"\r\nAccept-Encoding: identity\r\nAccept: text/*\r\nConnection: Close\r\n\r\n"
               plaintext="$(tm_out "$data" | hexdump -v -e '16/1 "%02X"')"
               plaintext="${plaintext%%[!0-9A-F]*}"
               debugme echo -n "Srikanth Plain Text = $plaintext"
               send_app_data "$plaintext"
               debugme echo -n "response value = $?"
               return 1
               if [[ $? -eq 0 ]]; then
                  receive_app_data true
                  [[ $? -eq 0 ]] || > "$TMPFILE"
               else
                  > "$TMPFILE"
               fi
               send_close_notify "$DETECTED_TLS_VERSION"
          else
               > "$TMPFILE"
          fi
     fi
dcooper16 commented 2 years ago

Hello @srikr,

As I said previously, I would suggest you run the program in debug mode (e.g., bash -x ./testssl.sh -g 172.16.82.4:443, see https://github.com/drwetter/testssl.sh/wiki/Findings-and-HowTo-Fix-them ) and then look at the resulting log file (a file of the form /tmp/testssl-....log) to try to find out what is happening.

Looking at the information you sent, it is clear that the traffic keys are not being generated. The server's response (e.g., the encrypted extensions and certificate) is not even being decrypted. However, there is not enough information available to figure out why it is not being decrypted.

srikr commented 2 years ago

Sorry @dcooper16 I ran the test again in debug mode.. which gave more info but couldn't really understand about the decryption problem. Attached is the logs what you have requested for: testssl-23029.log My Function call starts from line number 17517

dcooper16 commented 2 years ago

Hello @srikr,

I took a look at the log, and the problem seems to here:

|14594>     tls_sockets(): hello_done=1
|14594>     tls_sockets(): skip=false

This confused me, since I couldn't find this anywhere in the code. This led me to look again at the console output you provided, where I noticed that you are using a very old version of the 3.1dev branch:

    testssl       3.1dev from https://testssl.sh/dev/
    (67780b1 2020-04-28 10:08:19 -- )

I thought perhaps that the above was a bug in testssl.sh that was fixed sometime in the past two years, but this code does not appear in the version of testssl.sh that you are using either.

srikr commented 2 years ago

thanks @dcooper16 a lot for your time and looking into the logs and pin pointing exactly the issue. So inorder for me to download the latest testssl.sh I have to download from git repo: https://github.com/drwetter/testssl.sh here correct or let me know which code should I need to download to avoid such costly and more time consuming issues.

srikr commented 2 years ago

I quickly downloaded latest dev build from https://github.com/drwetter/testssl.sh which had following build version which is working fine

testssl       3.1dev from https://testssl.sh/dev/
    (ff23a2b 2022-05-06 08:17:49)

This works at its best for TLS1.3.. Is it possible to write similar logic for TLS1.2/TLS1.1 version as well or any other suggestion or recommendations how can the same be achieved for TLS Versions 1.2 and 1.1.

dcooper16 commented 2 years ago

Is it possible to write similar logic for TLS1.2/TLS1.1 version

Theoretically, anything is possible. However, the code for TLS 1.3 took a lot a work, and my guess would be that trying to implement the full handshake for TLS 1.2 and TLS 1.1 would be much more difficult. It is not something that I was willing to try.

as well or any other suggestion or recommendations how can the same be achieved for TLS Versions 1.2 and 1.1.

I don't have any other suggestions. However, it seems that there must be an easier way to accomplish what you are trying to do than to try to modify testssl.sh in this way.

srikr commented 2 years ago

sure @dcooper16 thanks for the clear and detailed responses, the reason I was willing to try using testssl.sh is as follows: a) It is doing lot of attacks testing with SSL which I can leverage for the TLS Fingerprinting Testing. b) It does lot of tests in a sequence using one api call so adding a small code in that API can suffice lot of my testing requirements c) It supports all versions of TLS d) Using a For Loop in different tests I can run all the tests for different HTTP User Agents. e) Also it is a shell script, Easier to change and test

Anyways I am happy that I can run all the tests for TLS1.3 which many of the tools are yet to Support.

drwetter commented 2 years ago

Please let me know if I should reopen this