Hangs before testing begins

jmkgreen commented 3 years ago

Please make sure that you provide enough information so that we understand what your issue is about.

Did you check the documentation in ~/doc/ or, if it is a different problem: Did you google for it?

Yep - similar to #1489 but now much worse.

uname -a

Linux DESKTOP-G6CKCF5 4.4.0-19041-Microsoft #488-Microsoft Mon Sep 01 13:43:00 PST 2020 x86_64 x86_64 x86_64 GNU/Linux

testssl version from the banner: testssl.sh -b 2>/dev/null | head -4 | tail -2

The above command hangs.

git log | head -1 (if running from git repo)

I'm running the 3.0.2 zip downloaded into an Ubuntu 20.04 instance running under WSL2 on Win10. The same behaviour occurs from the latest git which I tried initially.

openssl version used by testssl.sh: testssl.sh -b 2>/dev/null | awk -F':' '/openssl/ { print $2}'

Hangs.

steps to reproduce: testssl.sh or docker command line, if possible incl. host

Via docker it seems to work fine.

what exactly was happening, output is needed

➜  testssl.sh-3.0.2 ./testssl.sh https://www.google.com

No engine or GOST support via engine with your /usr/bin/openssl
^C

Waited several minutes before hitting ctrl+c.

what did you expect instead?

A test...

drwetter commented 3 years ago

-v pls. At least the hanging process needs to be identified. You may want to have a look @ 'https://github.com/drwetter/testssl.sh/wiki/Findings-and-HowTo-Fix-them (Debug yourself).

Cheers, Dirk

jmkgreen commented 3 years ago

As requested.

➜  testssl.sh-3.0.2 time ./testssl.sh -v https://www.bbc.co.uk

No engine or GOST support via engine with your /usr/bin/openssl

###########################################################
    testssl.sh       3.0.2 from https://testssl.sh/

      This program is free software. Distribution and
             modification under GPLv2 permitted.
      USAGE w/o ANY WARRANTY. USE IT AT YOUR OWN RISK!

       Please file bugs @ https://testssl.sh/bugs/

###########################################################

 Using "OpenSSL 1.1.1f  31 Mar 2020" [~79 ciphers]
 on DESKTOP-G6CKCF5:/usr/bin/openssl
 (built: "Apr 20 11:53:50 2020", platform: "debian-amd64")

./testssl.sh -v https://www.bbc.co.uk  0.23s user 2.36s system 0% cpu 12:18.91 total

I'll take a look at the document but this thing always used to "just work" under an older environment. Not sure if it's that I'm now using WSL v2 or an updated Ubuntu that is causing the issue.

drwetter commented 3 years ago

by -v I was like asking for more verbose input from you, James, not from testssl.sh ;-) (sorry)

jmkgreen commented 3 years ago

I'm not sure how I can be of much help. Your document says to run the whole script. I'm guessing that you expect people to be including your tool in their own script? Can't see any other references to a script.

I ran with --debug=2 --log and I have to say it all looks pretty normal once it becomes unstuck...

## Scan started as: "testssl.sh --debug=2 --log https://www.bbc.co.uk"
## at DESKTOP-G6CKCF5:/usr/bin/openssl
## version testssl: 3.0.2  from
## version openssl: "1.1.1f" from "Apr 20 11:53:50 2020")

Testing all IPv4 addresses (port 443): 212.58.237.252 212.58.233.252
------------------------------------------------------------------------------------------
 Start 2020-10-01 10:45:09        -->> 212.58.237.252:443 (www.bbc.co.uk) <<--

 Further IP addresses:   212.58.233.252
 rDNS (212.58.237.252):  --

sending client hello... sending client hello... reading server hello...
sending close_notify...
  (286 lines returned)

sending client hello... sending client hello... reading server hello...
sending close_notify...
  (276 lines returned)
one proto determined: tls1_3
OPTIMAL_PROTO:
HTTP/1.1 200 OK
Date: Thu, 01 Oct 2020 09:45:12 GMT
...

I've just re-run with --debug=6. It immediately spits out k=v options, warns about GOST support, then hangs. Output appears identical to above.

The following records exactly what happens. Make yourself a coffee in the middle! https://asciinema.org/a/362904

drwetter commented 3 years ago

Hi James,

can't still tell where it hangs. You need me pls either to provide the command in the process list (ps fawux) or, better: SETX=true bash -x testssl.sh <CMDLINE>". When you run the latter you'll spot the culprit

asciicinema is great. I always wanted to amend the description, see #1242 . Maybe with a little bit more of action than yours ;-)

Cheers, Dirk

jmkgreen commented 3 years ago

Hope this helps then:

|16952>         find_openssl_binary(): HAS_CHACHA20=false
|16953>         find_openssl_binary(): HAS_AES128_GCM=false
|16954>         find_openssl_binary(): HAS_AES256_GCM=false
|16955>         find_openssl_binary(): HAS_ZLIB=false
|16957>         find_openssl_binary(): /usr/bin/openssl ciphers -s
|16957>         find_openssl_binary(): grep -aiq 'unknown option'
|16958>         find_openssl_binary(): OSSL_CIPHERS_S=-s
|16962>         find_openssl_binary(): /usr/bin/openssl s_client -ssl2 -connect invalid.
|16962>         find_openssl_binary(): grep -aiq 'unknown option'
|16965>         find_openssl_binary(): /usr/bin/openssl s_client -ssl3 -connect invalid.
|16965>         find_openssl_binary(): grep -aiq 'unknown option'
|16968>         find_openssl_binary(): /usr/bin/openssl s_client -tls1_3 -connect invalid.
|16968>         find_openssl_binary(): grep -aiq 'unknown option'
^

ctrl+c obviously applied.

drwetter commented 3 years ago

Thanks. Strange though

What does

/usr/bin/openssl version -a return
/usr/bin/openssl s_client -tls1_3 -connect invalid. return (mind the trailing dot here)
/usr/bin/openssl genpkey -algorithm X448 return

jmkgreen commented 3 years ago

➜  testssl.sh-3.0.2 /usr/bin/openssl version -a
OpenSSL 1.1.1f  31 Mar 2020
built on: Mon Apr 20 11:53:50 2020 UTC
platform: debian-amd64
options:  bn(64,64) rc4(16x,int) des(int) blowfish(ptr)
compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -Wa,--noexecstack -g -O2 -fdebug-prefix-map=/build/openssl-P_ODHM/openssl-1.1.1f=. -fstack-protector-strong -Wformat -Werror=format-security -DOPENSSL_TLS_SECURITY_LEVEL=2 -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DRC4_ASM -DMD5_ASM -DAESNI_ASM -DVPAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DX25519_ASM -DPOLY1305_ASM -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2
OPENSSLDIR: "/usr/lib/ssl"
ENGINESDIR: "/usr/lib/x86_64-linux-gnu/engines-1.1"
Seeding source: os-specific

➜  testssl.sh-3.0.2 /usr/bin/openssl s_client -tls1_3 -connect invalid.
connect:errno=11

(the above hung for a while)

➜  testssl.sh-3.0.2 /usr/bin/openssl genpkey -algorithm X448
-----BEGIN PRIVATE KEY-----
MEYCAQAwBQYDK2VvBDoEOAyB0AK7epn2ReazViZck+R4b9yFjsKB/WQ87ABoXWqb
kYcs2JBD5Rg/ZaVMqalPXCq6AxMZvJbN
-----END PRIVATE KEY-----

drwetter commented 3 years ago

Not that it is of any help for you but your /usr/bin/openssl seems not to work in your context (WSL).

Does /usr/bin/openssl s_client -tls1_3 -connect testssl.sh:443 -servername testssl.sh </dev/null work?

And: Does is still hang when you swap invalid. with test. example. localhost. or x (no trailing dot for x) ?

The only workarounds which I can imagine right now are not nice ones.

jmkgreen commented 3 years ago

/usr/bin/openssl s_client -tls1_3 -connect testssl.sh:443 -servername testssl.sh </dev/null comes back just fine.

With test. - no, with example. - no, with localhost. - immediate connection refused messages, with x - no.

drwetter commented 3 years ago

sigh.

I guess it's a DNS issue when /usr/bin/openssl s_client -tls1_2 -connect invalid. doesn't work either?

jmkgreen commented 3 years ago

➜  testssl.sh-3.0.2 /usr/bin/openssl s_client -tls1_2 -connect invalid.
connect:errno=11

The rest of the environment does seem to work. It's reason for being is git, and other linux based tooling like aws cli.

drwetter commented 3 years ago

invalid., test. and example. are legitimate names (https://tools.ietf.org/html/rfc6761) and your DNS resolver should return NXDOMAIN.

There's something broken with your DNS config or with WSL on your side I can't help you with. Sorry

jmkgreen commented 3 years ago

To be clear, are you suggesting the tool requires those to operate?

drwetter commented 3 years ago

To be clear, are you suggesting the tool requires those to operate?

Sarcastically: Yes, the tool requires to have a proper DNS resolver. This is propaby not what you want to hear but what do you expect me to do without breaking other setups?

We need to make a check whether TLS 1.3 is natively supported. We spent a lot of time getting this check to work -- for probably everybody except your setup. Or maybe for Microsoft's broken implementation.

In your setup the first I would recommend to do is to understand why this fails. Maybe it's a config problem, maybe it's an intrinsic issue of WSL2. If you don't want that which I can understand, you need to either to change the platform or patch privately the line to HAS_TLS13=true or maybe a local DNS entry for invalid. or invalid works (Windows' /etc/hosts or WSL's?)

drwetter commented 3 years ago

Hi @jmkgreen ,

could you please try

prompt> for t in  invalid. test. example.  test.;  do
      time /usr/bin/openssl s_client  -tls1_2  -connect $t
done
prompt>

drwetter commented 3 years ago

Idea is to make the Special-Use Domain Names configurable. It seems a more common problem as I assumed a while back.

ghost commented 3 years ago

I'm experiencing similar slowness in a WSL2 environment. I've run your latest command above, and get the following:

for t in  invalid. test. example.  test.;  do
>       time /usr/bin/openssl s_client  -tls1_2  -connect $t
> done
140264340821312:error:2008F002:BIO routines:BIO_lookup_ex:system lib:../crypto/bio/b_addr.c:726:Name or service not known
connect:errno=22

real    0m7.222s
user    0m0.006s
sys     0m0.000s
140247974712640:error:2008F002:BIO routines:BIO_lookup_ex:system lib:../crypto/bio/b_addr.c:726:Temporary failure in name resolution
connect:errno=11

real    0m20.023s
user    0m0.010s
sys     0m0.001s
139774726587712:error:2008F002:BIO routines:BIO_lookup_ex:system lib:../crypto/bio/b_addr.c:726:Name or service not known
connect:errno=22

real    0m15.017s
user    0m0.008s
sys     0m0.001s
140431012365632:error:2008F002:BIO routines:BIO_lookup_ex:system lib:../crypto/bio/b_addr.c:726:Name or service not known
connect:errno=22

real    0m2.049s
user    0m0.006s
sys     0m0.001s

drwetter commented 3 years ago

Ok, least negative entries are cached - somehow.

Is there a non-existing windows domain name which resolves instantly?

ghost commented 3 years ago

Not sure! But here's something interesting. The same command runs much more quickly in Git Bash (stand-alone non-WSL linux environment which comes with git for Windows):

for t in  invalid. test. example.  test.;  do
>       time /usr/bin/openssl s_client  -tls1_2  -connect $t
> done
34359738384:error:2008F002:BIO routines:BIO_lookup_ex:system lib:crypto/bio/b_addr.c:730:Name or service not known
connect:errno=88

real    0m5.086s
user    0m0.062s
sys     0m0.093s
34359738384:error:2008F002:BIO routines:BIO_lookup_ex:system lib:crypto/bio/b_addr.c:730:Name or service not known
connect:errno=88

real    0m2.428s
user    0m0.062s
sys     0m0.062s
34359738384:error:2008F002:BIO routines:BIO_lookup_ex:system lib:crypto/bio/b_addr.c:730:Name or service not known
connect:errno=88

real    0m3.495s
user    0m0.062s
sys     0m0.062s
34359738384:error:2008F002:BIO routines:BIO_lookup_ex:system lib:crypto/bio/b_addr.c:730:Name or service not known
connect:errno=88

real    0m2.474s
user    0m0.061s
sys     0m0.046s

ghost commented 3 years ago

Here is nslookup for the same domains in WSL:

for t in  invalid. test. example.  test.; do
    time nslookup $t
done
Server:         172.17.232.241
Address:        172.17.232.241#53

** server can't find invalid: NXDOMAIN

real    0m2.511s
user    0m0.183s
sys     0m0.202s
Server:         172.17.232.241
Address:        172.17.232.241#53

** server can't find test: NXDOMAIN

real    0m1.075s
user    0m0.000s
sys     0m0.049s
Server:         172.17.232.241
Address:        172.17.232.241#53

** server can't find example: NXDOMAIN

real    0m10.076s
user    0m0.000s
sys     0m0.055s
Server:         172.17.232.241
Address:        172.17.232.241#53

** server can't find test: NXDOMAIN

real    0m12.244s
user    0m0.010s
sys     0m0.031s

ghost commented 3 years ago

Here's the same thing in PowerShell:

foreach ($t in @('invalid.', 'test.', 'example.', 'test.')) {
>>     $time = Measure-Command { nslookup $t | Out-Default }
>>     Write-Host $t completed in $time.TotalSeconds seconds`n
>> }
*** UnKnown can't find invalid.: Non-existent domain
Server:  UnKnown
Address:  fe80::1213:31ff:fe1b:952a

DNS request timed out.
    timeout was 2 seconds.
invalid. completed in 2.1815786 seconds

*** UnKnown can't find test.: Non-existent domain
Server:  UnKnown
Address:  fe80::1213:31ff:fe1b:952a

test. completed in 0.3177416 seconds

*** UnKnown can't find example.: Non-existent domain
Server:  UnKnown
Address:  fe80::1213:31ff:fe1b:952a

example. completed in 0.1592713 seconds

*** UnKnown can't find test.: Non-existent domain
Server:  UnKnown
Address:  fe80::1213:31ff:fe1b:952a

test. completed in 0.1291524 seconds

abkil commented 3 years ago

Hi, After so many months with this issue, I finally found out this solution ! I don't know if it's the best solution but testssl starts now after 3 seconds instead of hanging forever.

drwetter commented 3 years ago

Can you try the branch _windows_dnsfix using NXCONNECT=localhost:0 ./testssl.sh $YOURTARGET and report back pls?

ghost commented 3 years ago

TL;DR

Yes! NXCONNECT=localhost:0 makes a dramatic difference in performance. Thanks!

Below are the outputs of the test domains above, both without and with NXCONNECT defined. I have also tested to make sure that the speedup with NXCONNECT defined was not due to caching by rerunning the non-NXCONNECT script again, and I can confirm that the speedup is definitely due to defining NXCONNECT, not due to caching.

WITHOUT `NXCONNECT`

$ for t in  invalid. test. example.  test.;  do
> time ./testssl.sh $t
> done

###########################################################
    testssl.sh       3.1dev from https://testssl.sh/dev/
    (fe6c22f 2021-09-15 09:53:20 -- )

      This program is free software. Distribution and
             modification under GPLv2 permitted.
      USAGE w/o ANY WARRANTY. USE IT AT YOUR OWN RISK!

       Please file bugs @ https://testssl.sh/bugs/

###########################################################

 Using "OpenSSL 1.1.1f  31 Mar 2020" [~98 ciphers]
 on CCM310:/usr/bin/openssl
 (built: "Aug 23 17:02:39 2021", platform: "debian-amd64")

Fatal error: No IPv4/IPv6 address(es) for "invalid" available

real    7m16.018s
user    0m2.873s
sys     0m1.822s

###########################################################
    testssl.sh       3.1dev from https://testssl.sh/dev/
    (fe6c22f 2021-09-15 09:53:20 -- )

      This program is free software. Distribution and
             modification under GPLv2 permitted.
      USAGE w/o ANY WARRANTY. USE IT AT YOUR OWN RISK!

       Please file bugs @ https://testssl.sh/bugs/

###########################################################

 Using "OpenSSL 1.1.1f  31 Mar 2020" [~98 ciphers]
 on CCM310:/usr/bin/openssl
 (built: "Aug 23 17:02:39 2021", platform: "debian-amd64")

Fatal error: No IPv4/IPv6 address(es) for "test" available

real    7m17.799s
user    0m3.434s
sys     0m2.094s

###########################################################
    testssl.sh       3.1dev from https://testssl.sh/dev/
    (fe6c22f 2021-09-15 09:53:20 -- )

      This program is free software. Distribution and
             modification under GPLv2 permitted.
      USAGE w/o ANY WARRANTY. USE IT AT YOUR OWN RISK!

       Please file bugs @ https://testssl.sh/bugs/

###########################################################

 Using "OpenSSL 1.1.1f  31 Mar 2020" [~98 ciphers]
 on CCM310:/usr/bin/openssl
 (built: "Aug 23 17:02:39 2021", platform: "debian-amd64")

Fatal error: No IPv4/IPv6 address(es) for "example" available

real    7m16.845s
user    0m3.301s
sys     0m1.829s

###########################################################
    testssl.sh       3.1dev from https://testssl.sh/dev/
    (fe6c22f 2021-09-15 09:53:20 -- )

      This program is free software. Distribution and
             modification under GPLv2 permitted.
      USAGE w/o ANY WARRANTY. USE IT AT YOUR OWN RISK!

       Please file bugs @ https://testssl.sh/bugs/

###########################################################

 Using "OpenSSL 1.1.1f  31 Mar 2020" [~98 ciphers]
 on CCM310:/usr/bin/openssl
 (built: "Aug 23 17:02:39 2021", platform: "debian-amd64")

Fatal error: No IPv4/IPv6 address(es) for "test" available

real    7m21.978s
user    0m3.721s
sys     0m2.875s

WITH `NXCONNECT`

$ for t in  invalid. test. example.  test.;  do time NXCONNECT=localhost:0 ./testssl.sh $t; done

###########################################################
    testssl.sh       3.1dev from https://testssl.sh/dev/
    (fe6c22f 2021-09-15 09:53:20 -- )

      This program is free software. Distribution and
             modification under GPLv2 permitted.
      USAGE w/o ANY WARRANTY. USE IT AT YOUR OWN RISK!

       Please file bugs @ https://testssl.sh/bugs/

###########################################################

 Using "OpenSSL 1.1.1f  31 Mar 2020" [~98 ciphers]
 on CCM310:/usr/bin/openssl
 (built: "Aug 23 17:02:39 2021", platform: "debian-amd64")

Fatal error: No IPv4/IPv6 address(es) for "invalid" available

real    1m18.920s
user    0m3.297s
sys     0m2.357s

###########################################################
    testssl.sh       3.1dev from https://testssl.sh/dev/
    (fe6c22f 2021-09-15 09:53:20 -- )

      This program is free software. Distribution and
             modification under GPLv2 permitted.
      USAGE w/o ANY WARRANTY. USE IT AT YOUR OWN RISK!

       Please file bugs @ https://testssl.sh/bugs/

###########################################################

 Using "OpenSSL 1.1.1f  31 Mar 2020" [~98 ciphers]
 on CCM310:/usr/bin/openssl
 (built: "Aug 23 17:02:39 2021", platform: "debian-amd64")

Fatal error: No IPv4/IPv6 address(es) for "test" available

real    1m17.626s
user    0m3.566s
sys     0m2.038s

###########################################################
    testssl.sh       3.1dev from https://testssl.sh/dev/
    (fe6c22f 2021-09-15 09:53:20 -- )

      This program is free software. Distribution and
             modification under GPLv2 permitted.
      USAGE w/o ANY WARRANTY. USE IT AT YOUR OWN RISK!

       Please file bugs @ https://testssl.sh/bugs/

###########################################################

 Using "OpenSSL 1.1.1f  31 Mar 2020" [~98 ciphers]
 on CCM310:/usr/bin/openssl
 (built: "Aug 23 17:02:39 2021", platform: "debian-amd64")

Fatal error: No IPv4/IPv6 address(es) for "example" available

real    1m18.689s
user    0m3.772s
sys     0m2.421s

###########################################################
    testssl.sh       3.1dev from https://testssl.sh/dev/
    (fe6c22f 2021-09-15 09:53:20 -- )

      This program is free software. Distribution and
             modification under GPLv2 permitted.
      USAGE w/o ANY WARRANTY. USE IT AT YOUR OWN RISK!

       Please file bugs @ https://testssl.sh/bugs/

###########################################################

 Using "OpenSSL 1.1.1f  31 Mar 2020" [~98 ciphers]
 on CCM310:/usr/bin/openssl
 (built: "Aug 23 17:02:39 2021", platform: "debian-amd64")

Fatal error: No IPv4/IPv6 address(es) for "test" available

real    1m17.160s
user    0m3.473s
sys     0m2.152s

drwetter commented 3 years ago

Hi @davidwales ,

slight misunderstanding. Just a NXCONNECT=localhost:0 -p ./testssl.sh $anyrealtarget maybe against testssl.sh -p $anyrealtarget would suffice (amended with -p).

Cheers, Dirk

ghost commented 3 years ago

So... This was without NXCONNECT:

$ time ./testssl.sh -p duckduckgo.com

###########################################################
    testssl.sh       3.1dev from https://testssl.sh/dev/
    (fe6c22f 2021-09-15 09:53:20 -- )

      This program is free software. Distribution and
             modification under GPLv2 permitted.
      USAGE w/o ANY WARRANTY. USE IT AT YOUR OWN RISK!

       Please file bugs @ https://testssl.sh/bugs/

###########################################################

 Using "OpenSSL 1.1.1f  31 Mar 2020" [~98 ciphers]
 on CCM310:/usr/bin/openssl
 (built: "Aug 23 17:02:39 2021", platform: "debian-amd64")

 Start 2021-09-17 09:36:55        -->> 20.43.111.112:443 (duckduckgo.com) <<--

 rDNS (20.43.111.112):   --
 Service detected:       HTTP

 Testing protocols via sockets except NPN+ALPN

 SSLv2      not offered (OK)
 SSLv3      not offered (OK)
 TLS 1      not offered
 TLS 1.1    not offered
 TLS 1.2    offered (OK)
 TLS 1.3    offered (OK): final
 NPN/SPDY   not offered
 ALPN/HTTP2 h2, http/1.1 (offered)

 Done 2021-09-17 09:37:04 [  94s] -->> 20.43.111.112:443 (duckduckgo.com) <<--

real    1m34.440s
user    0m5.460s
sys     0m2.712s

And this was with NXCONNECT:

$ time NXCONNECT=localhost:0 ./testssl.sh -p duckduckgo.com

###########################################################
    testssl.sh       3.1dev from https://testssl.sh/dev/
    (fe6c22f 2021-09-15 09:53:20 -- )

      This program is free software. Distribution and
             modification under GPLv2 permitted.
      USAGE w/o ANY WARRANTY. USE IT AT YOUR OWN RISK!

       Please file bugs @ https://testssl.sh/bugs/

###########################################################

 Using "OpenSSL 1.1.1f  31 Mar 2020" [~98 ciphers]
 on CCM310:/usr/bin/openssl
 (built: "Aug 23 17:02:39 2021", platform: "debian-amd64")

 Start 2021-09-17 10:24:18        -->> 20.43.111.112:443 (duckduckgo.com) <<--

 rDNS (20.43.111.112):   --

It's been running for 5 hours now, with no sign of halting!

ghost commented 3 years ago

I just tried the NXCONNECT version again, and it was quicker this time:

$ time NXCONNECT=localhost:0 ./testssl.sh -p duckduckgo.com

###########################################################
    testssl.sh       3.1dev from https://testssl.sh/dev/
    (fe6c22f 2021-09-15 09:53:20 -- )

      This program is free software. Distribution and
             modification under GPLv2 permitted.
      USAGE w/o ANY WARRANTY. USE IT AT YOUR OWN RISK!

       Please file bugs @ https://testssl.sh/bugs/

###########################################################

 Using "OpenSSL 1.1.1f  31 Mar 2020" [~98 ciphers]
 on CCM310:/usr/bin/openssl
 (built: "Aug 23 17:02:39 2021", platform: "debian-amd64")

 Start 2021-09-17 15:29:16        -->> 20.43.111.112:443 (duckduckgo.com) <<--

 rDNS (20.43.111.112):   --
 Service detected:       HTTP

 Testing protocols via sockets except NPN+ALPN

 SSLv2      not offered (OK)
 SSLv3      not offered (OK)
 TLS 1      not offered
 TLS 1.1    not offered
 TLS 1.2    offered (OK)
 TLS 1.3    offered (OK): final
 NPN/SPDY   not offered
 ALPN/HTTP2 h2, http/1.1 (offered)

 Done 2021-09-17 15:29:23 [  17s] -->> 20.43.111.112:443 (duckduckgo.com) <<--

real    0m18.875s
user    0m5.781s
sys     0m4.850s

ghost commented 3 years ago

It looks like it's quicker with NXCONNECT, except for one time when it hung for 5 hours. Perhaps duckduckgo dropped the connection, and testssl.sh didn't notice?

drwetter commented 3 years ago

The occasion where it hung is not something we can use for production unless it was a one time thing.

If you like you can exchange localhost by 127.0.0.1 and later the port by something else and try but I am afraid in general we need good data as a basis.

ghost commented 3 years ago

The one where it hung was only once. I can run some more tests if you like?

drwetter commented 3 years ago

It would help tremendously to get a picture whether this is the right direction. And I would be able to make improvements step by step.

At the moment I can't tell whether the name localhost is good under windows or whether 127.0.0.1 would fit better, I believe the second is better. Then I can't tell whether port 0 is a good choice - I was assuming that there's in 99.999% no service listening and either it hangs for a while or it's sending a ICMP . The latter is good, the former is not.

drwetter commented 3 years ago

Also other folks affected by this (WSL users) are cordially invited to help.

Does e.g. NXCONNECT=127.0.0.1:0 ./testssl.sh -p $anyrealtarget from the https://github.com/drwetter/testssl.sh/tree/windows_dns_fix branch save time as opposed to ./testssl.sh -p $anyrealtarget ??

ghost commented 3 years ago

Retesting, with a few different domains. TL:DR: Setting the NXCONNECT variable to either 127.0.0.1:0 or localhost:0 leads to a 6x speedup. I didn't get any hangs this time, so it may have been a one-off issue due to my network.

Without `NXCONNECT`

$ for target in duckduckgo.com google.com github.com; do time ./testssl.sh -p $target; done

###########################################################
    testssl.sh       3.1dev from https://testssl.sh/dev/
    (fe6c22f 2021-09-15 09:53:20 -- )

      This program is free software. Distribution and
             modification under GPLv2 permitted.
      USAGE w/o ANY WARRANTY. USE IT AT YOUR OWN RISK!

       Please file bugs @ https://testssl.sh/bugs/

###########################################################

 Using "OpenSSL 1.1.1f  31 Mar 2020" [~98 ciphers]
 on CCM310:/usr/bin/openssl
 (built: "Aug 23 17:02:39 2021", platform: "debian-amd64")

 Start 2021-09-22 10:12:44        -->> 20.43.111.112:443 (duckduckgo.com) <<--

 rDNS (20.43.111.112):   --
 Service detected:       HTTP

 Testing protocols via sockets except NPN+ALPN

 SSLv2      not offered (OK)
 SSLv3      not offered (OK)
 TLS 1      not offered
 TLS 1.1    not offered
 TLS 1.2    offered (OK)
 TLS 1.3    offered (OK): final
 NPN/SPDY   not offered
 ALPN/HTTP2 h2, http/1.1 (offered)

 Done 2021-09-22 10:12:49 [  91s] -->> 20.43.111.112:443 (duckduckgo.com) <<--

real    1m32.044s
user    0m5.187s
sys     0m3.538s

###########################################################
    testssl.sh       3.1dev from https://testssl.sh/dev/
    (fe6c22f 2021-09-15 09:53:20 -- )

      This program is free software. Distribution and
             modification under GPLv2 permitted.
      USAGE w/o ANY WARRANTY. USE IT AT YOUR OWN RISK!

       Please file bugs @ https://testssl.sh/bugs/

###########################################################

 Using "OpenSSL 1.1.1f  31 Mar 2020" [~98 ciphers]
 on CCM310:/usr/bin/openssl
 (built: "Aug 23 17:02:39 2021", platform: "debian-amd64")

 Start 2021-09-22 10:14:13        -->> 172.217.167.110:443 (google.com) <<--

 Further IP addresses:   2404:6800:4006:811::200e
 rDNS (172.217.167.110): syd09s17-in-f14.1e100.net.
 Service detected:       HTTP

 Testing protocols via sockets except NPN+ALPN

 SSLv2      not offered (OK)
 SSLv3      not offered (OK)
 TLS 1      offered (deprecated)
 TLS 1.1    offered (deprecated)
 TLS 1.2    offered (OK)
 TLS 1.3    offered (OK): final
 NPN/SPDY   not offered
 ALPN/HTTP2 h2, http/1.1, grpc-exp (offered)

 Done 2021-09-22 10:14:20 [  89s] -->> 172.217.167.110:443 (google.com) <<--

real    1m30.398s
user    0m5.279s
sys     0m2.682s

###########################################################
    testssl.sh       3.1dev from https://testssl.sh/dev/
    (fe6c22f 2021-09-15 09:53:20 -- )

      This program is free software. Distribution and
             modification under GPLv2 permitted.
      USAGE w/o ANY WARRANTY. USE IT AT YOUR OWN RISK!

       Please file bugs @ https://testssl.sh/bugs/

###########################################################

 Using "OpenSSL 1.1.1f  31 Mar 2020" [~98 ciphers]
 on CCM310:/usr/bin/openssl
 (built: "Aug 23 17:02:39 2021", platform: "debian-amd64")

 Start 2021-09-22 10:15:43        -->> 52.64.108.95:443 (github.com) <<--

 rDNS (52.64.108.95):    --
 Service detected:       HTTP

 Testing protocols via sockets except NPN+ALPN

 SSLv2      not offered (OK)
 SSLv3      not offered (OK)
 TLS 1      not offered
 TLS 1.1    not offered
 TLS 1.2    offered (OK)
 TLS 1.3    offered (OK): final
 NPN/SPDY   not offered
 ALPN/HTTP2 h2, http/1.1 (offered)

 Done 2021-09-22 10:15:49 [  88s] -->> 52.64.108.95:443 (github.com) <<--

real    1m29.195s
user    0m4.754s
sys     0m1.830s

With `NXCONNECT=127.0.0.1:0`:

$ for target in duckduckgo.com google.com github.com; do time NXCONNECT=127.0.0.1:0 ./testssl.sh -p $targe
t; done

###########################################################
    testssl.sh       3.1dev from https://testssl.sh/dev/
    (fe6c22f 2021-09-15 09:53:20 -- )

      This program is free software. Distribution and
             modification under GPLv2 permitted.
      USAGE w/o ANY WARRANTY. USE IT AT YOUR OWN RISK!

       Please file bugs @ https://testssl.sh/bugs/

###########################################################

 Using "OpenSSL 1.1.1f  31 Mar 2020" [~98 ciphers]
 on CCM310:/usr/bin/openssl
 (built: "Aug 23 17:02:39 2021", platform: "debian-amd64")

 Start 2021-09-22 10:17:51        -->> 20.43.111.112:443 (duckduckgo.com) <<--

 rDNS (20.43.111.112):   --
 Service detected:       HTTP

 Testing protocols via sockets except NPN+ALPN

 SSLv2      not offered (OK)
 SSLv3      not offered (OK)
 TLS 1      not offered
 TLS 1.1    not offered
 TLS 1.2    offered (OK)
 TLS 1.3    offered (OK): final
 NPN/SPDY   not offered
 ALPN/HTTP2 h2, http/1.1 (offered)

 Done 2021-09-22 10:17:56 [  14s] -->> 20.43.111.112:443 (duckduckgo.com) <<--

real    0m14.586s
user    0m4.479s
sys     0m1.876s

###########################################################
    testssl.sh       3.1dev from https://testssl.sh/dev/
    (fe6c22f 2021-09-15 09:53:20 -- )

      This program is free software. Distribution and
             modification under GPLv2 permitted.
      USAGE w/o ANY WARRANTY. USE IT AT YOUR OWN RISK!

       Please file bugs @ https://testssl.sh/bugs/

###########################################################

 Using "OpenSSL 1.1.1f  31 Mar 2020" [~98 ciphers]
 on CCM310:/usr/bin/openssl
 (built: "Aug 23 17:02:39 2021", platform: "debian-amd64")

 Start 2021-09-22 10:18:04        -->> 172.217.167.110:443 (google.com) <<--

 Further IP addresses:   2404:6800:4006:812::200e
 rDNS (172.217.167.110): syd09s17-in-f14.1e100.net.
 Service detected:       HTTP

 Testing protocols via sockets except NPN+ALPN

 SSLv2      not offered (OK)
 SSLv3      not offered (OK)
 TLS 1      offered (deprecated)
 TLS 1.1    offered (deprecated)
 TLS 1.2    offered (OK)
 TLS 1.3    offered (OK): final
 NPN/SPDY   not offered
 ALPN/HTTP2 h2, http/1.1, grpc-exp (offered)

 Done 2021-09-22 10:18:11 [  14s] -->> 172.217.167.110:443 (google.com) <<--

real    0m15.277s
user    0m4.566s
sys     0m1.953s

###########################################################
    testssl.sh       3.1dev from https://testssl.sh/dev/
    (fe6c22f 2021-09-15 09:53:20 -- )

      This program is free software. Distribution and
             modification under GPLv2 permitted.
      USAGE w/o ANY WARRANTY. USE IT AT YOUR OWN RISK!

       Please file bugs @ https://testssl.sh/bugs/

###########################################################

 Using "OpenSSL 1.1.1f  31 Mar 2020" [~98 ciphers]
 on CCM310:/usr/bin/openssl
 (built: "Aug 23 17:02:39 2021", platform: "debian-amd64")

 Start 2021-09-22 10:18:20        -->> 13.237.44.5:443 (github.com) <<--

 rDNS (13.237.44.5):     ec2-13-237-44-5.ap-southeast-2.compute.amazonaws.com.
 Service detected:       HTTP

 Testing protocols via sockets except NPN+ALPN

 SSLv2      not offered (OK)
 SSLv3      not offered (OK)
 TLS 1      not offered
 TLS 1.1    not offered
 TLS 1.2    offered (OK)
 TLS 1.3    offered (OK): final
 NPN/SPDY   not offered
 ALPN/HTTP2 h2, http/1.1 (offered)

 Done 2021-09-22 10:18:25 [  13s] -->> 13.237.44.5:443 (github.com) <<--

real    0m13.401s
user    0m4.629s
sys     0m2.041s

With `NXCONNECT=localhost:0`:

$ for target in duckduckgo.com google.com github.com; do time NXCONNECT=localhost:0 ./testssl.sh -p $targe
t; done

###########################################################
    testssl.sh       3.1dev from https://testssl.sh/dev/
    (fe6c22f 2021-09-15 09:53:20 -- )

      This program is free software. Distribution and
             modification under GPLv2 permitted.
      USAGE w/o ANY WARRANTY. USE IT AT YOUR OWN RISK!

       Please file bugs @ https://testssl.sh/bugs/

###########################################################

 Using "OpenSSL 1.1.1f  31 Mar 2020" [~98 ciphers]
 on CCM310:/usr/bin/openssl
 (built: "Aug 23 17:02:39 2021", platform: "debian-amd64")

 Start 2021-09-22 10:20:09        -->> 20.43.111.112:443 (duckduckgo.com) <<--

 rDNS (20.43.111.112):   --
 Service detected:       HTTP

 Testing protocols via sockets except NPN+ALPN

 SSLv2      not offered (OK)
 SSLv3      not offered (OK)
 TLS 1      not offered
 TLS 1.1    not offered
 TLS 1.2    offered (OK)
 TLS 1.3    offered (OK): final
 NPN/SPDY   not offered
 ALPN/HTTP2 h2, http/1.1 (offered)

 Done 2021-09-22 10:20:14 [  14s] -->> 20.43.111.112:443 (duckduckgo.com) <<--

real    0m15.100s
user    0m4.536s
sys     0m2.330s

###########################################################
    testssl.sh       3.1dev from https://testssl.sh/dev/
    (fe6c22f 2021-09-15 09:53:20 -- )

      This program is free software. Distribution and
             modification under GPLv2 permitted.
      USAGE w/o ANY WARRANTY. USE IT AT YOUR OWN RISK!

       Please file bugs @ https://testssl.sh/bugs/

###########################################################

 Using "OpenSSL 1.1.1f  31 Mar 2020" [~98 ciphers]
 on CCM310:/usr/bin/openssl
 (built: "Aug 23 17:02:39 2021", platform: "debian-amd64")

 Start 2021-09-22 10:20:22        -->> 142.250.71.78:443 (google.com) <<--

 Further IP addresses:   2404:6800:4006:80a::200e
 rDNS (142.250.71.78):   syd15s17-in-f14.1e100.net.
 Service detected:       HTTP

 Testing protocols via sockets except NPN+ALPN

 SSLv2      not offered (OK)
 SSLv3      not offered (OK)
 TLS 1      offered (deprecated)
 TLS 1.1    offered (deprecated)
 TLS 1.2    offered (OK)
 TLS 1.3    offered (OK): final
 NPN/SPDY   not offered
 ALPN/HTTP2 h2, http/1.1, grpc-exp (offered)

 Done 2021-09-22 10:20:28 [  13s] -->> 142.250.71.78:443 (google.com) <<--

real    0m14.410s
user    0m4.618s
sys     0m1.912s

###########################################################
    testssl.sh       3.1dev from https://testssl.sh/dev/
    (fe6c22f 2021-09-15 09:53:20 -- )

      This program is free software. Distribution and
             modification under GPLv2 permitted.
      USAGE w/o ANY WARRANTY. USE IT AT YOUR OWN RISK!

       Please file bugs @ https://testssl.sh/bugs/

###########################################################

 Using "OpenSSL 1.1.1f  31 Mar 2020" [~98 ciphers]
 on CCM310:/usr/bin/openssl
 (built: "Aug 23 17:02:39 2021", platform: "debian-amd64")

 Start 2021-09-22 10:20:37        -->> 52.64.108.95:443 (github.com) <<--

 rDNS (52.64.108.95):    ec2-52-64-108-95.ap-southeast-2.compute.amazonaws.com.
 Service detected:       HTTP

 Testing protocols via sockets except NPN+ALPN

 SSLv2      not offered (OK)
 SSLv3      not offered (OK)
 TLS 1      not offered
 TLS 1.1    not offered
 TLS 1.2    offered (OK)
 TLS 1.3    offered (OK): final
 NPN/SPDY   not offered
 ALPN/HTTP2 h2, http/1.1 (offered)

 Done 2021-09-22 10:20:42 [  13s] -->> 52.64.108.95:443 (github.com) <<--

real    0m13.606s
user    0m4.639s
sys     0m2.004s

drwetter commented 2 years ago

I updated the https://github.com/drwetter/testssl.sh/tree/windows_dns_fix branch, see commit log. .

My main concern is that other users encounter longer delays, depending on the windows config, so if it can be found out during testing that this is not happening would be appreciated!

ghost commented 2 years ago

I re-ran the test above with the latest commit (b6b5a67b92c1118b6c197d0f669d8ab3714bdb63). It seems to be working well!

$ for target in duckduckgo.com google.com github.com; do time ./testssl.sh -p $target; done

###########################################################
    testssl.sh       3.1dev from https://testssl.sh/dev/
    (b6b5a67 2021-10-02 15:25:42 -- )

      This program is free software. Distribution and
             modification under GPLv2 permitted.
      USAGE w/o ANY WARRANTY. USE IT AT YOUR OWN RISK!

       Please file bugs @ https://testssl.sh/bugs/

###########################################################

 Using "OpenSSL 1.1.1f  31 Mar 2020" [~98 ciphers]
 on CCM310:/usr/bin/openssl
 (built: "Aug 23 17:02:39 2021", platform: "debian-amd64")

 Start 2021-10-05 09:35:12        -->> 20.43.111.112:443 (duckduckgo.com) <<--

 rDNS (20.43.111.112):   --
 Service detected:       HTTP

 Testing protocols via sockets except NPN+ALPN

 SSLv2      not offered (OK)
 SSLv3      not offered (OK)
 TLS 1      not offered
 TLS 1.1    not offered
 TLS 1.2    offered (OK)
 TLS 1.3    offered (OK): final
 NPN/SPDY   not offered
 ALPN/HTTP2 h2, http/1.1 (offered)

 Done 2021-10-05 09:35:18 [  15s] -->> 20.43.111.112:443 (duckduckgo.com) <<--

real    0m16.087s
user    0m4.422s
sys     0m2.096s

###########################################################
    testssl.sh       3.1dev from https://testssl.sh/dev/
    (b6b5a67 2021-10-02 15:25:42 -- )

      This program is free software. Distribution and
             modification under GPLv2 permitted.
      USAGE w/o ANY WARRANTY. USE IT AT YOUR OWN RISK!

       Please file bugs @ https://testssl.sh/bugs/

###########################################################

 Using "OpenSSL 1.1.1f  31 Mar 2020" [~98 ciphers]
 on CCM310:/usr/bin/openssl
 (built: "Aug 23 17:02:39 2021", platform: "debian-amd64")

 Start 2021-10-05 09:35:26        -->> 142.250.70.238:443 (google.com) <<--

 Further IP addresses:   2404:6800:4015:801::200e
 rDNS (142.250.70.238):  mel05s02-in-f14.1e100.net.
 Service detected:       HTTP

 Testing protocols via sockets except NPN+ALPN

 SSLv2      not offered (OK)
 SSLv3      not offered (OK)
 TLS 1      offered (deprecated)
 TLS 1.1    offered (deprecated)
 TLS 1.2    offered (OK)
 TLS 1.3    offered (OK): final
 NPN/SPDY   not offered
 ALPN/HTTP2 h2, http/1.1, grpc-exp (offered)

 Done 2021-10-05 09:35:35 [  16s] -->> 142.250.70.238:443 (google.com) <<--

real    0m17.340s
user    0m4.500s
sys     0m2.127s

###########################################################
    testssl.sh       3.1dev from https://testssl.sh/dev/
    (b6b5a67 2021-10-02 15:25:42 -- )

      This program is free software. Distribution and
             modification under GPLv2 permitted.
      USAGE w/o ANY WARRANTY. USE IT AT YOUR OWN RISK!

       Please file bugs @ https://testssl.sh/bugs/

###########################################################

 Using "OpenSSL 1.1.1f  31 Mar 2020" [~98 ciphers]
 on CCM310:/usr/bin/openssl
 (built: "Aug 23 17:02:39 2021", platform: "debian-amd64")

 Start 2021-10-05 09:35:46        -->> 13.236.229.21:443 (github.com) <<--

 rDNS (13.236.229.21):   ec2-13-236-229-21.ap-southeast-2.compute.amazonaws.com.
 Service detected:       HTTP

 Testing protocols via sockets except NPN+ALPN

 SSLv2      not offered (OK)
 SSLv3      not offered (OK)
 TLS 1      not offered
 TLS 1.1    not offered
 TLS 1.2    offered (OK)
 TLS 1.3    offered (OK): final
 NPN/SPDY   not offered
 ALPN/HTTP2 h2, http/1.1 (offered)

 Done 2021-10-05 09:35:52 [  15s] -->> 13.236.229.21:443 (github.com) <<--

real    0m16.813s
user    0m5.675s
sys     0m3.393s

drwetter commented 2 years ago

Thanks!

Any more reports? Your help is appreciated

drwetter commented 2 years ago

Ping @ all WSL users (minus @davidwales ) . Could you help (using the https://github.com/drwetter/testssl.sh/tree/windows_dns_fix branch):

for target in duckduckgo.com google.com github.com; do time ./testssl.sh -p $target; done

Please report back whether you still see a delay

teward commented 2 years ago

Ping @ all WSL users (minus @davidwales ) . Could you help (using the https://github.com/drwetter/testssl.sh/tree/windows_dns_fix branch):

for target in duckduckgo.com google.com github.com; do time ./testssl.sh -p $target; done

Please report back whether you still see a delay

No delay here on a WSL2 box. It had the hang previously but not anymore

TobiX commented 2 years ago

Please report back whether you still see a delay

Can confirm: No delay with that branch (9s vs. 1:22m for one run)

drwetter commented 2 years ago

Thanks, I just merged this and look into it whether I can backport the PR to 3.0.

drwetter / testssl.sh