search

drwetter / testssl.sh

Testing TLS/SSL encryption anywhere on any port
https://testssl.sh
GNU General Public License v2.0
7.89k stars 1.02k forks source link

macOS 10.14.5: testssl.sh 3.0rc5: warning: command substitution: ignored null byte in input #1292

Closed bknowles closed 5 years ago

bknowles commented 5 years ago

Summary

On first glance, this issue appears to be related to https://github.com/drwetter/testssl.sh/issues/352 where a lot of background information can be found. However, it appears that this is actually a different problem.

Hardware Architecture (uname -a):

Darwin CECEN-SX2EVGTFL 18.6.0 Darwin Kernel Version 18.6.0: Thu Apr 25 23:16:27 PDT 2019; root:xnu-4903.261.4~2/RELEASE_X86_64 x86_64

Testssl version (from the banner: testssl.sh -b 2>/dev/null | head -4 | tail -2)

testssl.sh       3.0rc5 from https://testssl.sh/dev/
(eef63b1 2019-07-03 11:54:56 -- )

Git commit (git log | head -1)

commit eef63b1726b9f44ff63a6dc19c6e2ea16e78b644

OpenSSL Version (used by testssl.sh: testssl.sh -b 2>/dev/null | awk -F':' '/openssl/ { print $2}')

./bin/openssl.Darwin.x86_64

Steps to Reproduce (testssl.sh or docker command line, if possible incl. host)

$ ./testssl.sh https://host.domain.private:7233

This is a private host, using private DNS, on a private RFC-1918 10.* IP address. Trust me, you are not going to be able to hit this machine yourselves.

What's Wrong (output is needed)

I get the following warnings output to stderr throughout various test sections:

 Testing robust (perfect) forward secrecy, (P)FS -- omitting Null Authentication/Encryption, 3DES, RC4 

 PFS is offered (OK)          ./testssl.sh: line 6941: warning: command substitution: ignored null byte in input
./testssl.sh: line 1265: warning: command substitution: ignored null byte in input
./testssl.sh: line 6941: warning: command substitution: ignored null byte in input
./testssl.sh: line 1265: warning: command substitution: ignored null byte in input
./testssl.sh: line 6941: warning: command substitution: ignored null byte in input
./testssl.sh: line 1265: warning: command substitution: ignored null byte in input
./testssl.sh: line 6941: warning: command substitution: ignored null byte in input
./testssl.sh: line 1265: warning: command substitution: ignored null byte in input
./testssl.sh: line 6941: warning: command substitution: ignored null byte in input
./testssl.sh: line 1265: warning: command substitution: ignored null byte in input
./testssl.sh: line 6941: warning: command substitution: ignored null byte in input
./testssl.sh: line 1265: warning: command substitution: ignored null byte in input
./testssl.sh: line 6941: warning: command substitution: ignored null byte in input
./testssl.sh: line 1265: warning: command substitution: ignored null byte in input
./testssl.sh: line 6941: warning: command substitution: ignored null byte in input
./testssl.sh: line 1265: warning: command substitution: ignored null byte in input
./testssl.sh: line 6941: warning: command substitution: ignored null byte in input
./testssl.sh: line 1265: warning: command substitution: ignored null byte in input
./testssl.sh: line 6941: warning: command substitution: ignored null byte in input
./testssl.sh: line 1265: warning: command substitution: ignored null byte in input
./testssl.sh: line 6941: warning: command substitution: ignored null byte in input
./testssl.sh: line 1265: warning: command substitution: ignored null byte in input
./testssl.sh: line 6941: warning: command substitution: ignored null byte in input
./testssl.sh: line 1265: warning: command substitution: ignored null byte in input
ECDHE-RSA-AES256-GCM-SHA384 ECDHE-RSA-AES256-SHA384 ECDHE-RSA-AES256-SHA DHE-RSA-AES256-GCM-SHA384
                              DHE-RSA-AES256-SHA256 DHE-RSA-AES256-SHA DHE-RSA-CAMELLIA256-SHA ECDHE-RSA-AES128-GCM-SHA256 ECDHE-RSA-AES128-SHA256
                              ECDHE-RSA-AES128-SHA DHE-RSA-AES128-GCM-SHA256 DHE-RSA-AES128-SHA256 DHE-RSA-AES128-SHA DHE-RSA-SEED-SHA
                              DHE-RSA-CAMELLIA128-SHA 
./testssl.sh: line 6941: warning: command substitution: ignored null byte in input
./testssl.sh: line 6941: warning: command substitution: ignored null byte in input
./testssl.sh: line 6941: warning: command substitution: ignored null byte in input
./testssl.sh: line 6941: warning: command substitution: ignored null byte in input
./testssl.sh: line 6941: warning: command substitution: ignored null byte in input
./testssl.sh: line 6941: warning: command substitution: ignored null byte in input
./testssl.sh: line 6941: warning: command substitution: ignored null byte in input
./testssl.sh: line 6941: warning: command substitution: ignored null byte in input
./testssl.sh: line 6941: warning: command substitution: ignored null byte in input
./testssl.sh: line 6941: warning: command substitution: ignored null byte in input
./testssl.sh: line 6941: warning: command substitution: ignored null byte in input
./testssl.sh: line 6941: warning: command substitution: ignored null byte in input
 Elliptic curves offered:     sect283k1 sect283r1 sect409k1 sect409r1 sect571k1 sect571r1 secp256k1 prime256v1 secp384r1 secp521r1 brainpoolP256r1 
                              brainpoolP384r1 brainpoolP512r1 
 DH group offered:            Unknown DH group (1024 bits)

 Testing server preferences 

 Has server cipher order?     ./testssl.sh: line 6941: warning: command substitution: ignored null byte in input
./testssl.sh: line 1265: warning: command substitution: ignored null byte in input
./testssl.sh: line 1278: warning: command substitution: ignored null byte in input
./testssl.sh: line 6941: warning: command substitution: ignored null byte in input
./testssl.sh: line 1265: warning: command substitution: ignored null byte in input
./testssl.sh: line 1265: warning: command substitution: ignored null byte in input
nope (NOT ok)
 Negotiated protocol          TLSv1.2
 Negotiated cipher            ./testssl.sh: line 1265: warning: command substitution: ignored null byte in input
ECDHE-RSA-RC4-SHA, 570 bit ECDH (B-571) -- inconclusive test, matching cipher in list missing, better see below
 Negotiated cipher per proto  (matching cipher in list missing)
./testssl.sh: line 6941: warning: command substitution: ignored null byte in input
./testssl.sh: line 1278: warning: command substitution: ignored null byte in input
./testssl.sh: line 1265: warning: command substitution: ignored null byte in input
     ECDHE-RSA-AES256-SHA:          TLSv1, TLSv1.1
     ECDHE-RSA-AES256-GCM-SHA384:   TLSv1.2
 No further cipher order check has been done as order is determined by the client

What did you expect instead?

None of the warnings above should be shown. More importantly, these warnings do not show up for version 2.9.5 or 2.9dev of testssl.

bknowles commented 5 years ago

Folks,

I'm happy to continue to debug the issue of how these non-printable characters are generated, but since you're not going to be able to hit the server(s) yourself, you're going to have to tell me what debugging methods and processes you want to try.

In the meanwhile, I think you may also want to consider whether or not you want to go ahead and apply the fix that we have identified using cat -v which prevents these unprintable characters from resulting in the errors I've encountered on macOS.

Thoughts?

drwetter commented 5 years ago

Am 17. Juli 2019 20:01:08 MESZ schrieb Brad Knowles notifications@github.com:

Folks,

I'm happy to continue to debug the issue of how these non-printable characters are generated, but since you're not going to be able to hit the server(s) yourself, you're going to have to tell me what debugging methods and processes you want to try.

In the meanwhile, I think you may also want to consider whether or not you want to go ahead and apply the fix that we have identified using cat -v which prevents these unprintable characters from resulting in the errors I've encountered on macOS.

Thoughts?

Hi Brad,

if we can't understand the cause of problem well enough and if I have the feeling that it is just "you" --that'll be difficult.

Maybe you can help in any direction? That can be access but doesn't have to.

Or maybe anybody else has seen this?

In ~2 weeks I also have more time to look at it again.

Cheers, Dirk

-- Sent from my mobile. Excuse my brevity&typos+the phone's autocorrection

bknowles commented 5 years ago

if we can't understand the cause of problem well enough and if I have the feeling that it is just "you" --that'll be difficult.

There are multiple moving parts here:

Alternatively, we have a simple known solution to the problem that involves piping all output from the openssl command through cat -v, so that the bizarre unprintable characters that come through all tested OpenSSL/LibreSSL implementations don't end up causing problems for the bizarre whacked-out version of bash that is provided by macOS.

Maybe you can help in any direction? That can be access but doesn't have to.

Your ability to access this server directly would require that you be physically located in our HQ building in Austin, Texas, or connected over our corporate VPN to that network. That's not going to happen, unless you want to become a W-2 employee of my employer, or go through the process to become a contractor or consultant from one of the very few external companies that we contract with to gain access to this very sensitive system.

The server I'm testing against is running TIBCO software. This is at the core of sales processing for my employer, and one of the most sensitive systems in the entire company. If you want to turn this into a consulting opportunity, I'm happy to take that to my management and see what their response is.

On the other hand, we know that we have a simple solution to this problem that can be easily applied today.

In the meanwhile, I'm happy to continue to work on debugging this problem with you, but you'll have to tell me what debugging you want done and how to do it.

Or maybe anybody else has seen this?

Dunno. Anyone using testssl.sh on a Mac would be vulnerable to this issue, if they ever get unprintable characters output by openssl.

In ~2 weeks I also have more time to look at it again.

Okay, I'll fork the repo and maintain my own private version, for now.

drwetter commented 5 years ago

Am 18. Juli 2019 17:42:18 MESZ schrieb Brad Knowles notifications@github.com:

if we can't understand the cause of problem well enough and if I have the feeling that it is just "you" --that'll be difficult.

There are multiple moving parts here:

  • I'm on a Mac, and anyone using testssl.sh on a Mac is subject to a whole host of bizarre problems that result from the macOS-provided weird version of bash.
  • I'm testing against an internal machine that is completely and totally inaccessible from the outside world, under multiple levels of filters and firewalls to prevent any kind of connection between this machine and the outside world.
  • My employer is using a Microsoft certificate authority system that I am not at all familiar with, and which is likely to have a whole host of its own weird bugs.
  • We're using self-signed CA certs from this Microsoft certificate authority system.
  • We're using the certificate chain created by this Microsoft certificate authority system on a TIBCO server, and TIBCO is well-known for extremely poor support of SSL in general.

Alternatively, we have a simple known solution to the problem that involves piping all output from the openssl command through cat -v, so that the bizarre unprintable characters that come through all tested OpenSSL/LibreSSL implementations don't end up causing problems for the bizarre whacked-out version of bash that is provided by macOS.

Maybe you can help in any direction? That can be access but doesn't have to.

Your ability to access this server directly would require that you be physically located in our HQ building in Austin, Texas, or connected over our corporate VPN to that network. That's not going to happen, unless you want to become a W-2 employee of my employer, or go through the process to become a contractor or consultant from one of the very few external companies that we contract with to gain access to this very sensitive system.

The server I'm testing against is running TIBCO software. This is at the core of sales processing for my employer, and one of the most sensitive systems in the entire company. If you want to turn this into a consulting opportunity, I'm happy to take that to my management and see what their response is.

On the other hand, we know that we have a simple solution to this problem that can be easily applied today.

In the meanwhile, I'm happy to continue to work on debugging this problem with you, but you'll have to tell me what debugging you want done and how to do it.

Or maybe anybody else has seen this?

Dunno. Anyone using testssl.sh on a Mac would be vulnerable to this issue, if they ever get unprintable characters output by openssl.

In ~2 weeks I also have more time to look at it again.

Okay, I'll fork the repo and maintain my own private version, for now.

-- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/drwetter/testssl.sh/issues/1292#issuecomment-512871534

Brad,

from what you showed me it's not a Mac (= specific client side) problem. It also appeared testing with Linux. My conclusion was it's a server side thing.

Is there any hint in the server's header e.g. which TIBCO shows and I can use for search engines (not necessarily Google but shodan and friends)?

-- Sent from my mobile. Excuse my brevity&typos+the phone's autocorrection

bknowles commented 5 years ago

Brad, from what you showed me it's not a Mac (= specific client side) problem. It also appeared testing with Linux. My conclusion was it's a server side thing. Is there any hint in the server's header e.g. which TIBCO shows and I can use for search engines (not necessarily Google but shodan and friends)?

Linux hosts also see the unprintable characters, but they aren't running the whacked-out version of bash that we have on macOS, so the unprintables don't result in the kinds of error messages on Linux that we see on macOS.

It's the combination of the unprintables plus the whacked-out version of bash that causes the real problems.

I'm not sure what TIBCO shows on the headers for a successful connection. I'll have to look at that tomorrow and see what I can find.

bknowles commented 5 years ago

As far as I can tell, TIBCO doesn't speak HTTP, and so does not produce any headers when you try to speak HTTP to it.

When I run this command:

echo "GET / HTTP/1.1\nHOST:HOSTNAME.domain.private\n\n" | openssl s_client -CAfile 2.pem -connect HOSTNAME.domain.private:7233 | cat -v

I get the expected output regarding the SSL certificate, certificate chain, session, etc.... But other than the variable potential unprintable characters at the end, I don't see any other output.

bknowles commented 5 years ago

The TIBCO EMS protocol is apparently an implementation of JMS, see https://techieswiki.com/what-is-tibco-ems-enterprise-message-service-tutorial.html

Towards that end, can we expect that the protocol would be inherently binary in nature, and therefore likely to result in unprintable characters when you connect to it, even if you connect via SSL?

bknowles commented 5 years ago

@drwetter -- Thoughts?

drwetter commented 5 years ago

Ah, ok. A small step for mankind :-)

Seriously: I was looking at shodan for a similar server. Is there any term I can use for searching?

https://www.shodan.io/search?query=TIBCO is probably not helping here.

Is port 7233 common for that protocol?

bknowles commented 5 years ago

I should think that TIBCO EMS is pretty well known. The pages I've found when duck-duck-go'ing for "TIBCO EMS protocol" have indicated a number of different common ports, including 7222, 7243, etc....

The page at https://community.tibco.com/questions/ssl-protocol-tibco-ems shows 7243. But we have a very wide variety of TIBCO EMS ports that we use internally, depending on which dev/dev2/integration/int2/qa/qa2/test/test2/performance/perf2 environment you're talking about.

I've mentioned two such ports in this thread, namely 7233 and 17293, but they are just two among many.

If you want to search shodan for ports to test against publicly, I would try every port you find mentioned on any page anywhere, because you never know who might have been stupid enough to leave their TIBCO EMS server publicly available. But that's a lot of work.

bknowles commented 5 years ago

The page at https://thirumalreddi.blogspot.com/2008/12/tibco-ems.html shows 7222 and 7344 as the example ports.

The page at https://docs.tibco.com/pub/policy-director/1.0.1/doc/html/GUID-834CCC0D-3384-4506-A143-C468B6BB8178.html shows 7222 and 7243 as the default ports that TIBCO EMS usually uses.

drwetter commented 5 years ago

Ok, thanks for the pointers.

Later, Dirk

bknowles commented 5 years ago

I created a shodan account, and searched for everything it had listening to port 7243. So far as I can tell, it only found nodes that are part of a DHT, and so therefore presumably doing Bittorrent, or some such.

Searching for port 7222 on shodan, it's mostly the same -- with the exception of one host that was found to be running nginx on this port, where it did a permanent 302 redirect to a different location.

I'm not finding any obvious hits for hosts running TIBCO EMS on these ports.

bknowles commented 5 years ago

Doing some stumbling around in Shodan, I found that I can do "product" queries. Since EMS is the TIBCO version of JMS, I was able to find a number of hosts running this product, with the query https://www.shodan.io/search?query=product%3A%22Java+Message+Service%22 and most of them appear to be listening on port 7676.

Maybe you can try probing some of these machines to see what JMS returns when you try to analyze the SSL connection against that server?

drwetter commented 5 years ago

thanks. Will do, as said in >= 1 week.

bknowles commented 5 years ago

thanks. Will do, as said in >= 1 week.

No problem. I have a solution that works for me, so I'm good.

At this point I'm just trying to figure out what else I can do to help you.

drwetter commented 5 years ago

I am back.

. Since EMS is the TIBCO version of JMS, I was able to find a number of hosts running this product, with the query https://www.shodan.io/search?query=product%3A%22Java+Message+Service%22 and most of them appear to be listening on port 7676.

unfortunately I found none with TLS on that port yet. And: not sure whether that'll help as Java servlet 1 with TLS isn't the same as Java servlet 2.

What text is returned when using echo | openssl s_client -connect $IP:$PORT --servername $HOST?

bknowles commented 5 years ago

What text is returned when using echo | openssl s_client -connect $IP:$PORT --servername $HOST?

It varies with each connection attempt, but amounts to unprintable binary characters for the most part.

See the above examples that have been piped through hexdump -C to try to help make some sense of the output we have gotten.

drwetter commented 5 years ago

I only see the result of a testssl.sh output with a cut *run_prototest_openssl-tls1.txt output. The *txt outputs weren't useful at all, unfortunately.

I need the part after the return code pls.

So far (in *run_prototest_openssl-tls1.txt) I haven't seen any similarity to JMS.

bknowles commented 5 years ago

This issue has gotten so long that it now has at least fifteen hidden items. Inside that section, if you click the "load more" button, you will find a number of times where I ran a command like echo | ~/src/git/testssl.sh/bin/openssl.Darwin.x86_64 s_client -showcerts -CAfile 2.pem -connect hostname.domain.private:17293 which was then piped through hexdump -C.

Or some variant thereof, perhaps with the OS-standard version of s_client, etc....

Those various dumps show clearly that the bytes appearing after the characters "in certificate chain)" are binary in nature, and not printable. And they are different every time you run the same command.

So, this is clearly a binary non-HTTP protocol that JMS is expecting to speak, yet we're trying to speak to it in an ASCII text HTTP-like protocol.

bknowles commented 5 years ago

Note that TIBCO was acquired for 4.3 Billion Dollars in 2014 (according to the page at https://fortune.com/2014/09/29/tibco-software-goes-private-with-4-3-billion-vista-equity-purchase/), and their software is used throughout the Fortune 500, especially in companies that handle B2C transactions -- i.e., virtually any company that sells products or services directly to customers, whether through a direct retail presence or via an electronic store, or some combination thereof.

So, I would sincerely hope that you won't be able to find any TIBCO EMS servers that are publicly exposed via Shodan. If you do find them through Shodan, then in all likelihood that means a multi-billion dollar Fortune 500 company has seriously dropped the ball and directly exposed their most sensitive internal customer financial transactions and PII data to the public internet.

I really do think that the closest thing you're likely to find that is publicly accessible are JMS servers, and you might even be able to stand one or more of those up yourself, if you really want to get into the nitty-gritty of testing that software to see how you can speak an ASCII HTTP-like protocol to a server that is expecting to receive a binary and very non-HTTP like protocol.

drwetter commented 5 years ago

Hi Brad,

could you please provide me for what I was asking for, i.e. an uncut version of

echo | openssl s_client -connect $IP:$PORT --servername $HOST | hexdump -C

I am interested in the binary chars what comes after e.g.

    Start Time: 1565767489
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: yes
---
DONE

If the bytes which come after this are different, it would be great to have 2-3 samples until the end.

Please spare me any further details which are not relevant for this issue. They clog the thread and just cost time filtering it for reading.

Thx & cheers, Dirk

drwetter commented 5 years ago

see referenced PR

bknowles commented 5 years ago

Sorry, I've been heads-down on preparing to move an entire Fortune 50 production eStore system to a completely new AWS environment with new account structure, etc....

I'll have to pop the stack a few levels to get back to exactly what was going on here.

bknowles commented 5 years ago

Running this command:

echo | openssl s_client -connect $IP:$PORT --servername $HOST | hexdump -C | tail

Resulted in the following output:

00002ba0  2e 55 2e 0a 0a 20 20 20  20 53 74 61 72 74 20 54  |.U...    Start T|
00002bb0  69 6d 65 3a 20 31 35 36  36 32 35 33 34 30 39 0a  |ime: 1566253409.|
00002bc0  20 20 20 20 54 69 6d 65  6f 75 74 20 20 20 3a 20  |    Timeout   : |
00002bd0  37 32 30 30 20 28 73 65  63 29 0a 20 20 20 20 56  |7200 (sec).    V|
00002be0  65 72 69 66 79 20 72 65  74 75 72 6e 20 63 6f 64  |erify return cod|
00002bf0  65 3a 20 30 20 28 6f 6b  29 0a 20 20 20 20 45 78  |e: 0 (ok).    Ex|
00002c00  74 65 6e 64 65 64 20 6d  61 73 74 65 72 20 73 65  |tended master se|
00002c10  63 72 65 74 3a 20 6e 6f  0a 2d 2d 2d 0a 00 00 00  |cret: no.---....|
00002c20  01 00 00 00 03 2d c3 9b  da                       |.....-...|
00002c29

Running it again, gave us the following:

00001880  0a 0a 20 20 20 20 53 74  61 72 74 20 54 69 6d 65  |..    Start Time|
00001890  3a 20 31 35 36 36 32 35  33 35 30 38 0a 20 20 20  |: 1566253508.   |
000018a0  20 54 69 6d 65 6f 75 74  20 20 20 3a 20 37 32 30  | Timeout   : 720|
000018b0  30 20 28 73 65 63 29 0a  20 20 20 20 56 65 72 69  |0 (sec).    Veri|
000018c0  66 79 20 72 65 74 75 72  6e 20 63 6f 64 65 3a 20  |fy return code: |
000018d0  30 20 28 6f 6b 29 0a 20  20 20 20 45 78 74 65 6e  |0 (ok).    Exten|
000018e0  64 65 64 20 6d 61 73 74  65 72 20 73 65 63 72 65  |ded master secre|
000018f0  74 3a 20 6e 6f 0a 2d 2d  2d 0a 00 00 00 01 00 00  |t: no.---.......|
00001900  00 03 32 9e a7 f8                                 |..2...|
00001906

This output is consistent with the patterns we have seen in the previous times I ran this same command string.

drwetter commented 5 years ago

Ok, thanks, good to know.

If the fix committed doesn't work, speak up