Unable to use testssl.sh in a proxy-required / no public DNS environment due to DNS failures

Trying to use testssl.sh (the latest dev branch on commit e51301d9ee2956f2c70b68a0f511925c1f33be28) it seems proxying does work in my configuration because of the DNS checks that take place before any of the actual SSL protocol stuff happens. This seems to be a somewhat known issue- I read https://github.com/drwetter/testssl.sh/issues/748 and what I got out of that was that this was resolved. But it doesn't seem to be the case- or more likely I misunderstood what was "fixed"- maybe having public DNS is an absolute requirement as it's currently written?

My environment is a typical strict/secure "corporate" environment:

No Internet DNS available, only internal DNS
No direct outbound TCP/UDP to WAN
HTTP proxy available, supporting CONNECT

The error I encounter is:

17:13:37 › ./testssl.sh -n none https://test.com 
For now I am providing the config file to have GOST support

###########################################################
    testssl.sh       3.1dev from https://testssl.sh/dev/
    (e51301d 2020-10-17 17:04:49 -- )

      This program is free software. Distribution and
             modification under GPLv2 permitted.
      USAGE w/o ANY WARRANTY. USE IT AT YOUR OWN RISK!

       Please file bugs @ https://testssl.sh/bugs/

###########################################################

 Using "OpenSSL 1.0.2-chacha (1.0.2k-dev)" [~179 ciphers]
 on oseu501665:./bin/openssl.Linux.ppc64le
 (built: "Sep 15 18:42:12 2020", platform: "linux-ppc64le")

Fatal error: No IPv4/IPv6 address(es) for "test.com" available

I thought that by specifying -n none there would not be any DNS requests made, but when I read through testssl.sh I see a function that does DNS resolution, so maybe -n only disables some of the more detailed DNS checks that testssl.sh does?

Is my use-case supported, or do you intend to support it? There are quite a few ways to hack around this but none of them really suit my needs, which is supporting an internal web interface to testssl.sh. I could hook the gethostbyname libc functions with LD_PRELOAD and return a dummy address but I'm not even sure that would be a quick fix as I don't know what will be done with the IP addresses that are returned

Thanks, appreciate your work on this and any info you can provide

EDIT: I should clarify- the proxy is set in the environment via http_proxy=http://127.0.0.1:3128 and https_proxy=http://127.0.0.1:3128. I should also add that I tried explicitly specifying --proxy http://127.0.0.1:3128 as well as --proxy 127.0.0.1:3128 did not make any difference. Sorry for leaving these details out initially!

How is testssl.sh supposed to connect to your proxy if you don't tell it how to find it?

How is testssl.sh supposed to connect to your proxy if you don't tell it how to find it?

Rereading my issue, I realized I never explicitly stated that the proxy was actually set :)

$ env | grep '^http_'
http_proxy=http://127.0.0.1:3128
https_proxy=http://127.0.0.1:3128

The proxy is set in the environment, via http_proxy and https_proxy, by IP address. Sorry, I should have clarified that. The proxy works fine (as expected) with other tools and APIs that honor those. So it's not pilot error- at least not at that idiotic of a level ;)

I should also clarify- I have this problem when using -n none and -n min AND when not explicitly specifying any -n setting. In this environment, any DNS queries that are made (with a big bundle of "one-off" exceptions) will return NXDOMAIN

At the time I tested this (before entering the issue) I just assumed setting the proxy in the environment in the way that most HTTP clients and client APIs tend to support was the correct way (actually, I think I just grepped for http_proxy in the main .sh file, saw it, and called it good) but to be sure I just went back and checked to make sure I wasn't making poor assumptions- following the flow of that much shell script is painful, so I figured there's a chance it was only being used in certain ways (e.g. I saw it used with curl and wget, for downloading cert revocation data and such ...)

When I grepped through the usage function, I did see:

     --proxy <host:port|auto>      (experimental) proxy connects via <host:port>, auto: values from \$env (\$http(s)_proxy)

So it appears it should be picking up the proxy from the environment. Maybe it's a bug there? Nope. I tried explicitly specifying the proxy using --proxy and got the same behavior:

$ ./testssl.sh --proxy 127.0.0.1:3128  www.test.com
...
Fatal error: No IPv4/IPv6 address(es) for "www.test.com" available

Immediately after, I confirmed that curl and wget were picking it up fine, in case I had misspelled something, but things were all functioning as expected.

I then considered it was possible I messed up when building openssl (I had to build my own from the patched fork, it's a ppc64le machine) but I was able to confirm that openssl s_client -proxy ... -connect ... works fine when invoked directly:

$ bin/openssl.Linux.ppc64le  s_client -proxy 127.0.0.1:3128 -connect www.test.com:443
CONNECTED(00000003)
depth=1 C = US, ST = VA, L = Herndon, O = Network Solutions L.L.C., CN = Network Solutions DV Server CA 2
verify error:num=20:unable to get local issuer certificate
---
Certificate chain
 0 s:/CN=www.test.com

I realize it's possible you just never intended for this to operate in an environment without direct access to Internet nameservers- if that's the case, I can think of all sorts of causes for artificial and material failures where a very simple sanity check (as opposed to a true dns interrogation/analysis) could cause the script to bail

Anyway- I think you understand the issue, though I don't have the time to track down exactly what line in the script is attempting to perform a DNS query

Do you agree this is a bug? And if so, do you intend to fix it, or is this enough of an edge case that it's not worth the time? If not, will you take a PR? I don't have a ton of time on my hands right now, but it would be nice to have this working so I'll give it a look when I do have some time

Thanks

I never tried w/o any local DNS yet but I suspect it can't work as there are other DNS lookups which need to be done before using a connect via proxy.

We need to clarify but probably there's not much hope -- as far as this is concerned.

So it appears it should be picking up the proxy from the environment

as indicated that's were --proxy auto is needed. The docs are there ro read ;-)

PS: https://testssl.sh/openssl-1.0.2i-chacha.pm.ipv6.contributed/

It's Friday and I'm tired, so I didn't proofread this- and it's a bit long. So keep that in mind. The summary is I've found the issue and found a few very non-invasive ways to patch it and/or workaround it. I'd like to ensure you understand what the issue it, and then propose how it can be fixed for everyone

as indicated that's were --proxy auto is needed. The docs are there ro read ;-)

They're excellent and comprehensive docs, that's for sure, but irrelevant in the context of this issue. You're correct that I misread the --proxy flag usage though and my head parsed it incorrectly when it say auto. I think env would be a better value there, but it's not important. Any sane person would just explicitly specify the proxy, as I did in the detailed walk-through I added after you closed the issue

The problem here is just a case that wasn't considered at implementation time, pure and simple- which I think you said above, but then for some reason followed up on with a comment about this being an issue of not reading the docs. Modifying/removing a completely superficial and non-essential check rectifies the issue, it doesn't really have anything to do with the docs

Here are the facts:

testssl.sh needs to make TCP connections
testssl.sh needs to do normal bidirectional data transfer over those connections- typically HTTP protocol, but can also be POP3S, IMAPS, etc, etc
testssl.sh supports use of an HTTP proxy for establishing the connection
testssl.sh supports an option (-n none) that (per the documentation) " performs no DNS lookups at all"

To accomplish a comprehensive analysis, there is no DNS required for any of this. HTTP proxies handle the DNS and is one of several reasons modern environments prohibit direct DNS egress from endpoints. Only the proxy servers need unfettered network access to upstream recursive resolvers, authoritative or root nameservers.

If not using -n none then a bunch of additional data can be brought into scope for an assessment. Looking at TXT record content, reverse DNS, MX records, and other DNS RRs can be useful in forming a more colorful picture. But none are required to accomplish the stated goals of testssl.sh. This is why -n none doesn't impose any substantive or even noticeable problems on the results.

The conclusion I made is this is something silly and artificial and it can either be fixed, removed, or made optional.

...

I spent 30 minutes looking at and determined that this seems to just be superficial check that isn't at all needed for the functionality of the app. At first pass, it seems to be two issues, both easily fixed. One seems to be hard-coded logic requiring an IP with -n none (as documented) even though it won't be used. The other issue isn't actually completely separate from that but is actually tightly related. It's for the pre-flight check that provides the user with a choice to bail out if a test connection fails or is janky/appears to not be SSL/TLS. This is with good intentions, to save a user from potentially attempting to interrogate a port that is closed, or not speaking SSL/TLS. If it decides it looks bad, it lets the user override it by typing "yes"

So, for now, a quick workaround seems to be to just set --ip to a dummy/sentinel value that is a valid dotted quad. You can use this in most cases:

./testssl.sh --proxy=http://127.0.0.1:3128 -n min --ip=0.0.0.0 https://www.homedepot.com

It doesn't really matter what it is, as long as it's set so it can get past the "$NODEIP is set" check and it can pass determine_ip_addresses()->is_ipv4addr()

After it starts, you'll get the warning that the endpoint isn't likely to be valid SSL/TLS, with the sub-reason potentially depending on the dummy value you gave it- e.g. if you pointed it at a random LAN address with an open port, you'll get different results than if you gave it an IP that isn't routable, or is filtered. In the end you just need to type "yes"

Since you taught me about documentation, I went ahead and checked to see if there was some sort of "batch" mode, to choose yes so that it can run unattended, e.g. from a web service or as part of a CI/CD pipeline. I found --warnings=batch, but it did not do what I had hoped. It seems to provide a negative response to the prompt, causing a swift exit. For this reason, you have to put up with either:

Manually typing yes (kills potential use for a web service or CI/CD pipeline)
Invoking as echo yes | testssl.sh ...
Using yes | testssl.sh ... (though yes spews infinitely and a bit overkill)
Using write(child.stdin, "yes\n", 4) if invoking from a native code or bytecode-based app, using whatever objects/functions are available to you
Make a small patch (the obvious long-term solution)

@drwetter If you don't want to accept the few lines that will fix this that's fine, but if I can save someone else the hours I've wasted with this (mostly caused by my poorly described initial issue description, which I have since updated) with a PR that will probably amount to 5-10 lines, I'll be happy to send it in- that way I won't have to keep a fork and a branch alive should I continue to use it

IMO, It would also be prudent to update the documentation regardless of whether you accept code changes- as part of the same PR, a separate one, or your own commit. I'm not working in a very rare environment here- it's typical of many universities, corporations, and other entities that choke all of their traffic through an HTTP proxy. I'm surprised I'm the first/only one to have this issue.

I'll leave it up to you, if you still think this is pilot error having to do with incorrectly specifying how to use a proxy and that the solution is in the documentation, or if you think that this workaround is intuitive after reading the documentation then I give up and I'll keep my fork/branch separate. I think the more reasonable and community friendly option is to leave this issue open and accept a PR so there's a proper flag and accompanying logic to handle this properly

If you have a preference about how you'd like the user experience to be- I see a few options:

Try to auto-detect this sort of environment using a basic heuristic (this could end up being problematic in some environments)
Provide an explicit flag for this case
Change the fatal error into one similar to the warning where the user can choose "yes" to continue at their own peril

Choice #2 seems the most appropriate at the moment.

My priority in making the choice would be:

The one with the most robustness/reliability across all environments (specifically, not breaking existing users)
The one most straightforward for the user experience (e.g. a single intuitively named flag, maybe even a third option for -n that is honored only when --proxy is enabled
The one that is clearest and most compact in the code. The changes are so trivial that regardless of the path taken, this shouldn't be much of an issue

Thanks, once again, I appreciate the work that you've put into this tool. The scope is incredible

I think if you know how the tool works (which seems likely!) you won't be skeptical of my claims that things work fine with the change to the superficial issue- but just so you can see, here is a full run- I understand there will be some negative impact if changes apply to runs that aren't using -n none, so any changes will need to be aware of that mode and only change behavior in the -n none case

› echo yes | ./testssl.sh --proxy=http://127.0.0.1:3128 -n none --ip=0.0.0.0 https://www.homedepot.com 
For now I am providing the config file to have GOST support

###########################################################
    testssl.sh       3.1dev from https://testssl.sh/dev/
    (e51301d 2020-10-17 17:04:49 -- )

      This program is free software. Distribution and
             modification under GPLv2 permitted.
      USAGE w/o ANY WARRANTY. USE IT AT YOUR OWN RISK!

       Please file bugs @ https://testssl.sh/bugs/

###########################################################

 Using "OpenSSL 1.0.2-chacha (1.0.2k-dev)" [~179 ciphers]
 on XXXXX:./bin/openssl.Linux.ppc64le
 (built: "Sep 25 19:03:44 2020", platform: "linux-ppc64le")

 Start 2020-12-19 00:01:39        -->> 0.0.0.0:443 (www.homedepot.com) <<--

 Via Proxy:              127.0.0.1:3128 
 A record via:           supplied IP "0.0.0.0"
 rDNS (0.0.0.0):         (instructed to minimize DNS queries)
 Your OpenSSL cannot connect to 0.0.0.0:443
 The results might look ok but they could be nonsense. Really proceed ? ("yes" to continue) -->  Service detected:       Couldn't determine what's running on port 443, assuming no HTTP service => skipping all HTTP checks

 Testing protocols via sockets except NPN+ALPN 

 SSLv2      not offered (OK)
 SSLv3      not offered (OK)
 TLS 1      not offered
 TLS 1.1    offered (deprecated)
 TLS 1.2    offered (OK)
 TLS 1.3    offered (OK): final
 NPN/SPDY   not tested as proxies do not support proxying it
 ALPN/HTTP2 not tested as proxies do not support proxying it

 Testing cipher categories 

 NULL ciphers (no encryption)                      not offered (OK)
 Anonymous NULL Ciphers (no authentication)        not offered (OK)
 Export ciphers (w/o ADH+NULL)                     not offered (OK)
 LOW: 64 Bit + DES, RC[2,4], MD5 (w/o export)      not offered (OK)
 Triple DES Ciphers / IDEA                         not offered
 Obsoleted CBC ciphers (AES, ARIA etc.)            offered
 Strong encryption (AEAD ciphers) with no FS       offered (OK)
 Forward Secrecy strong encryption (AEAD ciphers)  offered (OK)

 Testing server's cipher preferences 

 Has server cipher order?     no matching cipher in this list found (pls report this): DHE-RSA-SEED-SHA:SEED-SHA:DES-CBC3-SHA:RC4-MD5:DES-CBC-SHA:RC4-SHA:AES128-SHA:AES128-SHA256:AES256-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA:ECDH-RSA-DES-CBC3-SHA:ECDH-RSA-AES128-SHA:ECDH-RSA-AES256-SHA:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:DHE-DSS-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:AES256-SHA256:ECDHE-RSA-DES-CBC3-SHA:ECDHE-RSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-SHA256:ADH-AES256-GCM-SHA384:AECDH-AES128-SHA:ECDHE-RSA-RC4-SHA:ECDHE-ECDSA-AES128-SHA  . 

 Testing robust forward secrecy (FS) -- omitting Null Authentication/Encryption, 3DES, RC4 

 FS is offered (OK)           TLS_AES_256_GCM_SHA384
                              TLS_CHACHA20_POLY1305_SHA256
                              ECDHE-RSA-AES256-GCM-SHA384
                              ECDHE-RSA-AES256-SHA384 ECDHE-RSA-AES256-SHA
                              ECDHE-RSA-CHACHA20-POLY1305
                              TLS_AES_128_GCM_SHA256 TLS_AES_128_CCM_SHA256
                              TLS_AES_128_CCM_8_SHA256
                              ECDHE-RSA-AES128-GCM-SHA256
                              ECDHE-RSA-AES128-SHA256 ECDHE-RSA-AES128-SHA 
 Elliptic curves offered:     prime256v1 X25519 

 Testing server defaults (Server Hello) 

 TLS extensions (standard)    "renegotiation info/#65281" "server name/#0"
                              "EC point formats/#11" "session ticket/#35"
                              "status request/#5" "next protocol/#13172"
                              "supported versions/#43" "key share/#51"
                              "max fragment length/#1"
                              "application layer protocol negotiation/#16"
 Session Ticket RFC 5077 hint no -- no lifetime advertised
 SSL Session ID support       yes
 Session Resumption           Tickets no, ID resumption test failed
 TLS clock skew               Random values, no fingerprinting possible 
 Signature Algorithm          SHA256 with RSA
 Server key size              RSA 2048 bits (exponent is 65537)
 Server key usage             Digital Signature, Key Encipherment
 Server extended key usage    TLS Web Server Authentication, TLS Web Client Authentication
 Serial / Fingerprints        0ADC18B793FF16566DFD15C9932843E0 / SHA1 F686667795D38A2044815F0CDCC505949D917D04
                              SHA256 7CA892069AED05DFFE0CC0F8720ED2262D7622A2F47402CD8F6337ACF58AE7B7
 Common Name (CN)             www.homedepot.com 
 subjectAltName (SAN)         www.homedepot.com homedepot.com
                              secure2.homedepot.com 
 Trust (hostname)             Ok via SAN and CN (same w/o SNI)
 Chain of trust               Ok   
 EV cert (experimental)       yes 
 Certificate Validity (UTC)   480 >= 60 days (2020-01-13 00:00 --> 2022-04-13 12:00)
                              >= 398 days certificate life time but issued before 2020/09/01
 ETS/"eTLS", visibility info  not present
 Certificate Revocation List  http://crl3.digicert.com/sha2-ev-server-g2.crl
                              http://crl4.digicert.com/sha2-ev-server-g2.crl
 OCSP URI                     http://ocsp.digicert.com
 OCSP stapling                offered, revocation not tested as "openssl ocsp" doesn't support a proxy
 OCSP must staple extension   --
 DNS CAA RR (experimental)    (instructed to minimize DNS queries)
 Certificate Transparency     yes (certificate extension)
 Certificates provided        2
 Issuer                       DigiCert SHA2 Extended Validation Server CA (DigiCert Inc from US)
 Intermediate cert validity   #1: ok > 40 days (2028-10-22 12:00). DigiCert SHA2 Extended Validation Server CA <-- DigiCert High Assurance EV Root CA
 Intermediate Bad OCSP (exp.) Ok

 Testing vulnerabilities 

 Heartbleed (CVE-2014-0160)                not vulnerable (OK), no heartbeat extension
 CCS (CVE-2014-0224)                       not vulnerable (OK)
 Ticketbleed (CVE-2016-9244), experiment.  --   (applicable only for HTTPS)
 ROBOT                                     not vulnerable (OK)
 Secure Renegotiation (RFC 5746)           OpenSSL handshake didn't succeed
 Secure Client-Initiated Renegotiation     not vulnerable (OK)
 CRIME, TLS (CVE-2012-4929)                test failed (couldn't connect)
 POODLE, SSL (CVE-2014-3566)               not vulnerable (OK), no SSLv3 support
 TLS_FALLBACK_SCSV (RFC 7507)              Check failed, unexpected result , run bak -Z --debug=1 and look at /tmp/testssl.m5MQ4M/*tls_fallback_scsv.txt
 SWEET32 (CVE-2016-2183, CVE-2016-6329)    not vulnerable (OK)
 FREAK (CVE-2015-0204)                     not vulnerable (OK)
 DROWN (CVE-2016-0800, CVE-2016-0703)      not vulnerable on this host and port (OK)
                                           make sure you don't use this certificate elsewhere with SSLv2 enabled services
                                           https://censys.io/ipv4?q=7CA892069AED05DFFE0CC0F8720ED2262D7622A2F47402CD8F6337ACF58AE7B7 could help you to find out
 LOGJAM (CVE-2015-4000), experimental      not vulnerable (OK): no DH EXPORT ciphers, no DH key detected with <= TLS 1.2
 BEAST (CVE-2011-3389)                     not vulnerable (OK), no SSL3 or TLS1
 LUCKY13 (CVE-2013-0169), experimental     potentially VULNERABLE, uses cipher block chaining (CBC) ciphers with TLS. Check patches
 Winshock (CVE-2014-6321), experimental    not vulnerable (OK)
 RC4 (CVE-2013-2566, CVE-2015-2808)        no RC4 ciphers detected (OK)

Could not determine the protocol, only simulating generic clients.

 Running client simulations via sockets 

 Android 4.4.2                TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
 Android 5.0.0                TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
 Android 6.0                  TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
 Android 7.0 (native)         TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
 Android 8.1 (native)         TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
 Android 9.0 (native)         TLSv1.3 TLS_AES_256_GCM_SHA384, 253 bit ECDH (X25519)
 Android 10.0 (native)        TLSv1.3 TLS_AES_256_GCM_SHA384, 253 bit ECDH (X25519)
 Chrome 74 (Win 10)           TLSv1.3 TLS_AES_256_GCM_SHA384, 253 bit ECDH (X25519)
 Chrome 79 (Win 10)           TLSv1.3 TLS_AES_256_GCM_SHA384, 253 bit ECDH (X25519)
 Firefox 66 (Win 8.1/10)      TLSv1.3 TLS_AES_256_GCM_SHA384, 253 bit ECDH (X25519)
 Firefox 71 (Win 10)          TLSv1.3 TLS_AES_256_GCM_SHA384, 253 bit ECDH (X25519)
 IE 6 XP                      No connection
 IE 8 Win 7                   No connectionwal
 IE 8 XP                      No connection
 IE 11 Win 7                  TLSv1.2 ECDHE-RSA-AES256-SHA384, 256 bit ECDH (P-256)
 IE 11 Win 8.1                TLSv1.2 ECDHE-RSA-AES256-SHA384, 256 bit ECDH (P-256)
 IE 11 Win Phone 8.1          TLSv1.2 ECDHE-RSA-AES128-SHA256, 256 bit ECDH (P-256)
 IE 11 Win 10                 TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
 Edge 15 Win 10               TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
 Edge 17 (Win 10)             TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
 Opera 66 (Win 10)            TLSv1.3 TLS_AES_256_GCM_SHA384, 253 bit ECDH (X25519)
 Safari 9 iOS 9               TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
 Safari 9 OS X 10.11          TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
 Safari 10 OS X 10.12         TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
 Safari 12.1 (iOS 12.2)       TLSv1.3 TLS_CHACHA20_POLY1305_SHA256, 253 bit ECDH (X25519)
 Safari 13.0 (macOS 10.14.6)  TLSv1.3 TLS_CHACHA20_POLY1305_SHA256, 253 bit ECDH (X25519)
 Apple ATS 9 iOS 9            TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
 Java 6u45                    No connection
 Java 7u25                    No connection
 Java 8u161                   TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
 Java 11.0.2 (OpenJDK)        TLSv1.3 TLS_AES_256_GCM_SHA384, 256 bit ECDH (P-256)
 Java 12.0.1 (OpenJDK)        TLSv1.3 TLS_AES_256_GCM_SHA384, 256 bit ECDH (P-256)
 OpenSSL 1.0.2e               TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
 OpenSSL 1.1.0l (Debian)      TLSv1.2 ECDHE-RSA-AES256-GCM-SHA384, 256 bit ECDH (P-256)
 OpenSSL 1.1.1d (Debian)      TLSv1.3 TLS_AES_256_GCM_SHA384, 253 bit ECDH (X25519)
 Thunderbird (68.3)           TLSv1.3 TLS_AES_256_GCM_SHA384, 253 bit ECDH (X25519)

 Rating (experimental) 

 Rating specs (not complete)  SSL Labs's 'SSL Server Rating Guide' (version 2009q from 2020-01-30)
 Specification documentation  https://github.com/ssllabs/research/wiki/SSL-Server-Rating-Guide
 Protocol Support (weighted)  97 (29)
 Key Exchange     (weighted)  90 (27)
 Cipher Strength  (weighted)  60 (24)
 Final Score                  80
 Overall Grade                B
 Grade cap reasons            Grade capped to B. TLS 1.1 offered

 Done 2020-12-19 00:03:09 [  91s] -->> 0.0.0.0:443 (www.homedepot.com) <<--

EDIT: To be clear, this system has no direct route to the Internet over layer-3 and does not permit DNS queries to external/Internet/public zones- either by permitting layer-3 traffic to go to an arbitrary resolver or by acting as a recursive resolver for any public zones.

Should be possible using http_proxy, works for me in regular linux distribution (maybe not wsl or cygwin or whatever thats called). A recent merge provides custom ability to add http request headers if the proxy has an auth on it too

Should be possible using http_proxy, works for me in regular linux distribution (maybe not wsl or cygwin or whatever thats called).

Only with --proxy=auto as said

          if [[ "$PROXY" == auto ]]; then
               # Get $ENV https_proxy is the one we care about for connects
               PROXY="${https_proxy#*\/\/}"
               # Fallback:
               [[ -z "$PROXY" ]] && PROXY="${http_proxy#*\/\/}"
               [[ -z "$PROXY" ]] && fatal "you specified \"--proxy=auto\" but \"\$http(s)_proxy\" is empty" $ERR_CMDLINE
          fi

TL;DR:

./testssl.sh --proxy=http://<your_proxyip> -n none --ip=0.0.0.0 https://www.homedepot.com works but requires a confirmation:

 Your OpenSSL cannot connect to 0.0.0.0:443
 The results might look ok but they could be nonsense. Really proceed ? ("yes" to continue) --> yes

whereas ./testssl.sh --proxy=http://<your_proxyip> -n none https://www.homedepot.com doesn't work.

./testssl.sh --proxy=auto -n none --ip=0.0.0.0 https://www.homedepot.com works when http_proxy is set to an IP.

It should work also when in /etc/hosts DNS resolution is supplied.

@chrisdlangton @drwetter I think we all agree how it works and what the issue is. The only question is do you want to explicitly support this or do you want to leave the workarounds as the solutions.

For my case I just patched in an extra flag so that none of those are necessary- not for the purpose of being difficult but because the workarounds aren't ideal for my case. I'm invoking testssl.sh from a web application and there's no way to look up the DNS from the environment it's running in dynamically in-band- and DNS over TCP/HTTPS is far beyond the effort that should be spent solving this. Writing "yes" to the child process' stdin is a technical possibility but it's just not a great solution in my opinion. Tangent- this is especially true with Python if you're using Popen(..., shell=False) and are already reading stdout via a pipe- because of the documented deadlock issue with Popen() when both stdout and stdin are pipes. It's silly, but solving that requires you to spawn a dedicated thread to function as the stdin writer or the stdout reader. Not difficult, just ugly. Optionally, you could use Popen() with shell=True but I don't like to use Popen() that way- by now I'm sure you can tell I'm a bit picky and have at least a handful of standards that I think are reasonable, but can probably be called arbitrary :)

You can close this if you disagree that this is behavior/functionality that should be changed/enhanced

If you'd like a patch I'm happy to send what I'm using though I'm sure you can figure out your own very easily- and with your preferred choice of flag name, and which one of the few places you'd like to patch. Alternately, if you'd like to tell me what you'd like the flag name to be, I'll do it that way and send it in

Either way, thanks for the help and again thanks for all the work that went into the project. I don't want to waste any more of anyone's time

The only question is do you want to explicitly support this or do you want to leave the workarounds as the solutions.

There will be a proper solution. Didn't seem too difficult.

If you have a suggestion, go ahead!

Well, I made a lot of noise about this and then didn't follow through on sending a PR along. I'll be sending something in a few moments that seems to work without regressions, but it's going to need some more testing, especially for the "standard" cases

Rather than implement a whole new set of conditionals, I piggy-backed on the existing logic, trying to use as few code changes as possible. I added a value for the --nodns flag called strict

As a note to myself, I didn't check into how the changes will impact the case where --nodns strict is used, but the user wants /etc/hosts to be used as a reference. If there's a regression here, I think the adjustment is simple, I just need to remember to test it...

@drwetter I don't see any test-suites in the repository, do you happen to have anything you use for regression testing?

I believe your complain can be put much shorter: IP address resolution via proxy doesn't work, right?

In general if we want to make use of the proxy's DNS lookups it would change our approach. Currently we deliberately do DNS lookups client side as we want to scan every IP of the supplied name. So there are e.g. preflight requests in e.g.determine_optimal_proto() like

./bin/openssl.Linux.x86_64 s_client -tls1_2 -connect <IPSET>:443 -msg -proxy <MPROXYIP:PORT> -servername www.homedepot.com -no_comp where IPSET is the IP / are the IPs we determined upfront.

We could change that to ./bin/openssl.Linux.x86_64 s_client -tls1_2 -connect www.homedepot.com:443 -msg -proxy -servername www.homedepot.com -no_comp`` which would work but e.g. for cloudflare.com which resolves in Europe to

cloudflare.com has address 104.16.132.229
cloudflare.com has address 104.16.133.229
cloudflare.com has IPv6 address 2606:4700::6810:84e5
cloudflare.com has IPv6 address 2606:4700::6810:85e5

we would only scan one IPv4 address (or if the proxy decides to use IPv6 one IPv6 address) or any combination. Your PR actually did that. And doing traffic analysis it initially picked 104.16.133.229, then 104.16.132.229.

That maybe would work for you when you're in an environment where no client side DNS lookups are allowed and there's no other option. Others will complain sooner or later.

For the socket part: I haven't gotten that far yet but somehow your PR did that.

Bottom line: IF we pursue this further maybe taken the idea from your PR we need to tell the user that s/he might not scan every IP but a mixture thereof very clearly. Then also the cmd line option should be like --nodns=proxy or --nodns=strict-proxy and should only be possible to choose when there was a proxy specified.

I believe your complain can be put much shorter: IP address resolution via proxy doesn't work, right?

Not quite, no. It would be "testssl.sh can't operate at all without a recursive resolver, even when in nodns mode and using a proxy" - at least not without workarounds like hardcoding IP addresses into /etc/hosts, the 0.0.0.0 hack, etc

The first attempt at describing this caused you to close the issue with something along the lines of "of course your proxy can not be used if it can't be resolved"

After trying to describe the issue more clearly, you pointed me to the documentation, which I understood as "behaves as designed, won't fix"

From there I thought we finally got on the same page with the "complaint" and solution(s) but I'm not so sure anymore. Poor communication on my part, most likely

Either way, as I mentioned in one of the last few posts above, it may be a poor investment of both of our time to pursue this any further. After all, I seem to be the only one who has a need for this as clearly nobody else has "complained"

You are of course welcome to take the PR but I understand it's not really acceptable as-is, and there's practically zero upside in supporting the type of environment I'm in. I have only a very specific use-case that I need to support so it's probably most appropriate to keep this commit in my own branch. Keeps me happy and doesn't break any of your users or muck your code up

I'll close out the issue and the PR, but add a thank you again for the work you've done on the project. It's really quite astonishing and far and away the most thorough and complete OSS solution for assessing SSL/TLS endpoints

drwetter / testssl.sh

Unable to use testssl.sh in a proxy-required / no public DNS environment due to DNS failures #1750