TLS/SSL test type(s) - Githubissues

This issue is for planning a new test type that exercises TLS (SSL), which was suggested as test type 10 on issue #133. I recommend reviewing the comment thread for #133 since it contains several prior thoughts concerning a TLS/SSL test type that remain applicable. Though I am not manually copying those comments over here, please feel free to reiterate points raised there.

Add any comments and questions that you believe can help us reach a consensus on reasonable requirements for this new test type.

Not all participants will agree that a TLS test is warranted in this project. Historically, we have not focused on the performance of web servers, intending our project to be about web application frameworks. However, our project is about measuring the high-water mark of performance using real-world web application technology stacks. And from that perspective, including a TLS test type is reasonable as long as there are some stacks for which TLS would be included "in-stack." And I know that is satisfied in the real-world because many stacks include nginx or another mainstream web server that has robust TLS capability.

Perhaps it is important to remember that, to whatever degree it is possible, we want the test implementations in this project to represent canonical usage of the platforms and frameworks we exercise. Many platforms and frameworks are of the opinion that TLS is an external concern to be handled by a front-end web server, load balancer, or other external system or service. That is perfectly fine and acceptable. Such platforms and frameworks may elect to bypass implementing this TLS test type. All test types in this project are optional, and omission should not be considered a failing or even evidence of lack of capability for a framework or platform. (Aside: We intend to eventually provide contributors a way to document the architectural approach that they have taken, explain the details of each test implementation they have provided, and in the case of omitting a test, explain why that test type is not applicable.)

I propose the following:

This may be split into multiple test types since real-world TLS usage is applied to any and all request types. However, I am reluctant to duplicate many existing test types for a few reasons (e.g., additional test types increase the total runtime of the full suite). I tentatively propose TLS versions of two existing test types: Plaintext and Fortunes. Tentatively, TLS-Plaintext and TLS-Fortunes.
If possible, I'd like to agree to a singular TLS cipher-suite configuration that is broadly supported and considered acceptable to a wide audience. If that is infeasible, I'd like for us to set some "minimum" acceptable level and illustrate the options that satisfy the minimum.
If possible, I'd like to find and use a tool that validates TLS configuration that we can use in our suite's validation of implementations. Perhaps curl or similar can do this; I am not sure. But by way of comparison, I am imagining something that is approximately a command-line version of the SSL Labs tests.

Thoughts, questions, corrections, comments appreciated!

I think the openssl client is probably a good start for testing for the configuration of a server.

I agree that it is pointless replicating all tests for TLS as one or two like you have proposed will give enough data to provide a Delta between TLS and non TLS tests to allow for an overhead calculation.

I would say that AES is the most widely supported algo for the bulk ciper. Sha256 or higher for the hash shouldn't be controversial either. The main issue will be deciding on the certificate and initial key exchange.

The two tests should be enough to extrapolate from when comparing to non-TLS.

TLS should be 1.2 (1.3 isn't widely supported yet, < 1.1 is bad)

Cipher Suite should be encouraged to be a common one between frameworks; as there are dramatic differences in performances and varying them would be comparing the algorithm rather than the differing effects of using TLS.

Not sure what the cipher suite should be.

SSL Labs Cipher Suite (in order of preference) reports for various browsers

Chrome

TLS_GREASE_1A (0x1a1a)                                                    -
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02b)       Forward Secrecy | 128
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f)         Forward Secrecy | 128
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 (0xc02c)       Forward Secrecy | 256
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (0xc030)         Forward Secrecy | 256
TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 (0xcca9) Forward Secrecy | 256
TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (0xcca8)   Forward Secrecy | 256
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013)            Forward Secrecy | 128
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (0xc014)            Forward Secrecy | 256
TLS_RSA_WITH_AES_128_GCM_SHA256 (0x9c)                 WEAK | 128
TLS_RSA_WITH_AES_256_GCM_SHA384 (0x9d)                 WEAK | 256
TLS_RSA_WITH_AES_128_CBC_SHA (0x2f)                    WEAK | 128
TLS_RSA_WITH_AES_256_CBC_SHA (0x35)                    WEAK | 256
TLS_RSA_WITH_3DES_EDE_CBC_SHA (0xa)                    WEAK | 112

Edge

TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 (0xc02c)       Forward Secrecy | 256
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02b)       Forward Secrecy | 128
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (0xc030)         Forward Secrecy | 256
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f)         Forward Secrecy | 128
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 (0xc024)       Forward Secrecy | 256
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 (0xc023)       Forward Secrecy | 128
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (0xc028)         Forward Secrecy | 256
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (0xc027)         Forward Secrecy | 128
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA (0xc00a)          Forward Secrecy | 256
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA (0xc009)          Forward Secrecy | 128
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (0xc014)            Forward Secrecy | 256
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013)            Forward Secrecy | 128
TLS_RSA_WITH_AES_256_GCM_SHA384 (0x9d)                 WEAK | 256
TLS_RSA_WITH_AES_128_GCM_SHA256 (0x9c)                 WEAK | 128
TLS_RSA_WITH_AES_256_CBC_SHA256 (0x3d)                 WEAK | 256
TLS_RSA_WITH_AES_128_CBC_SHA256 (0x3c)                 WEAK | 128
TLS_RSA_WITH_AES_256_CBC_SHA (0x35)                    WEAK | 256
TLS_RSA_WITH_AES_128_CBC_SHA (0x2f)                    WEAK | 128
TLS_RSA_WITH_3DES_EDE_CBC_SHA (0xa)                    WEAK | 112

Firefox

TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02b)       Forward Secrecy | 128
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f)         Forward Secrecy | 128
TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 (0xcca9) Forward Secrecy | 256
TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 (0xcca8)   Forward Secrecy | 256
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 (0xc02c)       Forward Secrecy | 256
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (0xc030)         Forward Secrecy | 256
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA (0xc00a)          Forward Secrecy | 256
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA (0xc009)          Forward Secrecy | 128
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013)            Forward Secrecy | 128
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (0xc014)            Forward Secrecy | 256
TLS_DHE_RSA_WITH_AES_128_CBC_SHA (0x33)                Forward Secrecy | 128
TLS_DHE_RSA_WITH_AES_256_CBC_SHA (0x39)                Forward Secrecy | 256
TLS_RSA_WITH_AES_128_CBC_SHA (0x2f)                    WEAK | 128
TLS_RSA_WITH_AES_256_CBC_SHA (0x35)                    WEAK | 256
TLS_RSA_WITH_3DES_EDE_CBC_SHA (0xa)                    WEAK | 112

Top common is TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02b) ?

However, our project is about measuring the high-water mark of performance using real-world web application technology stacks. And from that perspective, including a TLS test type is reasonable as long as there are some stacks for which TLS would be included "in-stack." And I know that is satisfied in the real-world because many stacks include nginx or another mainstream web server that has robust TLS capability.

Would an "Is edge configuration" flag be sensible? TLS is valid in non-internet facing stacks; and there are already stacks in the tests that use for example ngnix as their front-end server; so they would presumably be edge/internet facing ready.

However, that can come with a performance disadvantage, so comparing non-internet facing stacks against internet facing stacks may be a little apples and oranges. While both fruit, one is more resistant to bruising?

The disadvantage I can see to that is it make be taken as approval that a particular stack is internet facing ready; while it may not be. So would at least need caveats; and may not be good for that reason?

@bhauer

If possible, I'd like to find and use a tool that validates TLS configuration that we can use in our suite's validation of implementations. Perhaps curl or similar can do this; I am not sure. But by way of comparison, I am imagining something that is approximately a command-line version of the SSL Labs tests.

I'm using this one: https://testssl.sh/, https://github.com/drwetter/testssl.sh I think it will tick every box: mainly shell script, non-exotic external dependencies, similar functionality to SSL Labs tests.

Off-topic: there is a similar command-line tool for SSH auditing https://github.com/arthepsy/ssh-audit It is not related to the testssl.sh.

Update: Here is a sample report for https://www.techempower.com/:

xyz@zyx:~/tmp/testssl.sh$ ./testssl.sh https://www.techempower.com/

###########################################################
    testssl.sh       2.9dev from https://testssl.sh/dev/
    (7e62dc3 2017-12-07 09:59:58 -- )

      This program is free software. Distribution and
             modification under GPLv2 permitted.
      USAGE w/o ANY WARRANTY. USE IT AT YOUR OWN RISK!

       Please file bugs @ https://testssl.sh/bugs/

###########################################################

 Using "OpenSSL 1.0.2-chacha (1.0.2i-dev)" [~183 ciphers]
 on ######:./bin/openssl.Linux.x86_64
 (built: "Jun 22 19:32:29 2016", platform: "linux-x86_64")

 Start 2018-02-15 21:53:39        -->> 69.89.75.196:443 (www.techempower.com) <<--

 rDNS (69.89.75.196):    --
 Service detected:       HTTP

 Testing protocols via sockets except SPDY+HTTP2 

 SSLv2      not offered (OK)
 SSLv3      not offered (OK)
 TLS 1      offered
 TLS 1.1    offered
 TLS 1.2    offered (OK)
 TLS 1.3    not offered
 SPDY/NPN   http/1.1 (advertised)
 HTTP2/ALPN http/1.1 (offered)

 Testing ~standard cipher categories 

 NULL ciphers (no encryption)                  not offered (OK)
 Anonymous NULL Ciphers (no authentication)    not offered (OK)
 Export ciphers (w/o ADH+NULL)                 not offered (OK)
 LOW: 64 Bit + DES encryption (w/o export)     not offered (OK)
 Weak 128 Bit ciphers (SEED, IDEA, RC[2,4])    not offered (OK)
 Triple DES Ciphers (Medium)                   offered
 High encryption (AES+Camellia, no AEAD)       offered (OK)
 Strong encryption (AEAD ciphers)              offered (OK)

 Testing robust (perfect) forward secrecy, (P)FS -- omitting Null Authentication/Encryption, 3DES, RC4 

 PFS is offered (OK)          ECDHE-RSA-AES256-GCM-SHA384 ECDHE-RSA-AES256-SHA384 ECDHE-RSA-AES256-SHA DHE-RSA-AES256-GCM-SHA384
                              DHE-RSA-AES256-SHA256 DHE-RSA-AES256-SHA DHE-RSA-CAMELLIA256-SHA ECDHE-RSA-AES128-GCM-SHA256
                              ECDHE-RSA-AES128-SHA256 ECDHE-RSA-AES128-SHA DHE-RSA-AES128-GCM-SHA256 DHE-RSA-AES128-SHA256
                              DHE-RSA-AES128-SHA DHE-RSA-CAMELLIA128-SHA 
 Elliptic curves offered:     prime256v1 

 Testing server preferences 

 Has server cipher order?     yes (OK)
 Negotiated protocol          TLSv1.2
 Negotiated cipher            ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
 Cipher order
    TLSv1:     ECDHE-RSA-AES128-SHA ECDHE-RSA-AES256-SHA DHE-RSA-AES128-SHA DHE-RSA-AES256-SHA AES128-SHA AES256-SHA
               DHE-RSA-CAMELLIA256-SHA CAMELLIA256-SHA DHE-RSA-CAMELLIA128-SHA CAMELLIA128-SHA DES-CBC3-SHA 
    TLSv1.1:   ECDHE-RSA-AES128-SHA ECDHE-RSA-AES256-SHA DHE-RSA-AES128-SHA DHE-RSA-AES256-SHA AES128-SHA AES256-SHA
               DHE-RSA-CAMELLIA256-SHA CAMELLIA256-SHA DHE-RSA-CAMELLIA128-SHA CAMELLIA128-SHA DES-CBC3-SHA 
    TLSv1.2:   ECDHE-RSA-AES128-GCM-SHA256 ECDHE-RSA-AES256-GCM-SHA384 DHE-RSA-AES128-GCM-SHA256 DHE-RSA-AES256-GCM-SHA384
               ECDHE-RSA-AES128-SHA256 ECDHE-RSA-AES128-SHA ECDHE-RSA-AES256-SHA384 ECDHE-RSA-AES256-SHA DHE-RSA-AES128-SHA256
               DHE-RSA-AES128-SHA DHE-RSA-AES256-SHA256 DHE-RSA-AES256-SHA AES128-GCM-SHA256 AES256-GCM-SHA384 AES128-SHA256
               AES256-SHA256 AES128-SHA AES256-SHA DHE-RSA-CAMELLIA256-SHA CAMELLIA256-SHA DHE-RSA-CAMELLIA128-SHA CAMELLIA128-SHA
               DES-CBC3-SHA 

 Testing server defaults (Server Hello) 

 TLS extensions (standard)    "server name/#0" "renegotiation info/#65281" "EC point formats/#11" "session ticket/#35"
                              "status request/#5" "heartbeat/#15" "next protocol/#13172"
                              "application layer protocol negotiation/#16"
 Session Ticket RFC 5077 hint 86400 seconds, session tickets keys seems to be rotated < daily
 SSL Session ID support       yes
 Session Resumption           Tickets: yes, ID: yes
 TLS clock skew               Random values, no fingerprinting possible 
 Signature Algorithm          SHA256 with RSA
 Server key size              RSA 4096 bits
 Fingerprint / Serial         SHA1 D6B3A1EF3FA6574D400E2700FD3601134355F060 / 5EB3A5F4DF31299DD34E5D135D8B4679
                              SHA256 F5E30EA7EA319419848E1F024566135244E223B50147118CE02A635D345672F8
 Common Name (CN)             *.techempower.com
 subjectAltName (SAN)         *.techempower.com techempower.com 
 Issuer                       COMODO RSA Domain Validation Secure Server CA (COMODO CA Limited from GB)
 Trust (hostname)             Ok via SAN wildcard and CN wildcard (same w/o SNI)
 Chain of trust               Ok   
 EV cert (experimental)       no 
 Certificate Expiration       282 >= 60 days (2015-11-13 02:00 --> 2018-11-25 01:59 +0200)
 # of certificates provided   4
 Certificate Revocation List  http://crl.comodoca.com/COMODORSADomainValidationSecureServerCA.crl
 OCSP URI                     http://ocsp.comodoca.com
 OCSP stapling                offered
 OCSP must staple             no
 DNS CAA RR (experimental)    not offered
 Certificate Transparency     no

 Testing HTTP header response @ "/" 

 HTTP Status Code             200 OK
 HTTP clock skew              +7 sec from localtime
 Strict Transport Security    182 days=15768000 s, just this domain
 Public Key Pinning           --
 Server banner                nginx
 Application banner           --
 Cookie(s)                    (none issued at "/")
 Security headers             --
 Reverse Proxy banner         --

 Testing vulnerabilities 

 Heartbleed (CVE-2014-0160)                not vulnerable (OK), timed out
 CCS (CVE-2014-0224)                       not vulnerable (OK)
 Ticketbleed (CVE-2016-9244), experiment.  not vulnerable (OK)
 Secure Renegotiation (CVE-2009-3555)      not vulnerable (OK)
 Secure Client-Initiated Renegotiation     not vulnerable (OK)
 CRIME, TLS (CVE-2012-4929)                not vulnerable (OK)
 BREACH (CVE-2013-3587)                    potentially NOT ok, uses gzip HTTP compression. - only supplied "/" tested
                                           Can be ignored for static pages or if no secrets in the page
 POODLE, SSL (CVE-2014-3566)               not vulnerable (OK)
 TLS_FALLBACK_SCSV (RFC 7507)              Downgrade attack prevention supported (OK)
 SWEET32 (CVE-2016-2183, CVE-2016-6329)    VULNERABLE, uses 64 bit block ciphers
 FREAK (CVE-2015-0204)                     not vulnerable (OK)
 DROWN (CVE-2016-0800, CVE-2016-0703)      not vulnerable on this host and port (OK)
                                           make sure you don't use this certificate elsewhere with SSLv2 enabled services
                                           https://censys.io/ipv4?q=F5E30EA7EA319419848E1F024566135244E223B50147118CE02A635D345672F8 could help you to find out
 LOGJAM (CVE-2015-4000), experimental      not vulnerable (OK): no DH EXPORT ciphers, no common primes detected
 BEAST (CVE-2011-3389)                     TLS1: ECDHE-RSA-AES128-SHA ECDHE-RSA-AES256-SHA DHE-RSA-AES128-SHA DHE-RSA-AES256-SHA
                                                 AES128-SHA AES256-SHA DHE-RSA-CAMELLIA256-SHA CAMELLIA256-SHA
                                                 DHE-RSA-CAMELLIA128-SHA CAMELLIA128-SHA DES-CBC3-SHA 
                                           VULNERABLE -- but also supports higher protocols (possible mitigation): TLSv1.1 TLSv1.2
 LUCKY13 (CVE-2013-0169), experimental     potentially VULNERABLE, uses cipher block chaining (CBC) ciphers with TLS
 RC4 (CVE-2013-2566, CVE-2015-2808)        no RC4 ciphers detected (OK)

 Testing 364 ciphers via OpenSSL plus sockets against the server, ordered by encryption strength 

Hexcode  Cipher Suite Name (OpenSSL)       KeyExch.   Encryption  Bits     Cipher Suite Name (RFC)
-----------------------------------------------------------------------------------------------------------------------------
 xc030   ECDHE-RSA-AES256-GCM-SHA384       ECDH 256   AESGCM      256      TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384              
 xc028   ECDHE-RSA-AES256-SHA384           ECDH 256   AES         256      TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384              
 xc014   ECDHE-RSA-AES256-SHA              ECDH 256   AES         256      TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA                 
 x9f     DHE-RSA-AES256-GCM-SHA384         DH 2048    AESGCM      256      TLS_DHE_RSA_WITH_AES_256_GCM_SHA384                
 x6b     DHE-RSA-AES256-SHA256             DH 2048    AES         256      TLS_DHE_RSA_WITH_AES_256_CBC_SHA256                
 x39     DHE-RSA-AES256-SHA                DH 2048    AES         256      TLS_DHE_RSA_WITH_AES_256_CBC_SHA                   
 x88     DHE-RSA-CAMELLIA256-SHA           DH 2048    Camellia    256      TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA              
 x9d     AES256-GCM-SHA384                 RSA        AESGCM      256      TLS_RSA_WITH_AES_256_GCM_SHA384                    
 x3d     AES256-SHA256                     RSA        AES         256      TLS_RSA_WITH_AES_256_CBC_SHA256                    
 x35     AES256-SHA                        RSA        AES         256      TLS_RSA_WITH_AES_256_CBC_SHA                       
 x84     CAMELLIA256-SHA                   RSA        Camellia    256      TLS_RSA_WITH_CAMELLIA_256_CBC_SHA                  
 xc02f   ECDHE-RSA-AES128-GCM-SHA256       ECDH 256   AESGCM      128      TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256              
 xc027   ECDHE-RSA-AES128-SHA256           ECDH 256   AES         128      TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256              
 xc013   ECDHE-RSA-AES128-SHA              ECDH 256   AES         128      TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA                 
 x9e     DHE-RSA-AES128-GCM-SHA256         DH 2048    AESGCM      128      TLS_DHE_RSA_WITH_AES_128_GCM_SHA256                
 x67     DHE-RSA-AES128-SHA256             DH 2048    AES         128      TLS_DHE_RSA_WITH_AES_128_CBC_SHA256                
 x33     DHE-RSA-AES128-SHA                DH 2048    AES         128      TLS_DHE_RSA_WITH_AES_128_CBC_SHA                   
 x45     DHE-RSA-CAMELLIA128-SHA           DH 2048    Camellia    128      TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA              
 x9c     AES128-GCM-SHA256                 RSA        AESGCM      128      TLS_RSA_WITH_AES_128_GCM_SHA256                    
 x3c     AES128-SHA256                     RSA        AES         128      TLS_RSA_WITH_AES_128_CBC_SHA256                    
 x2f     AES128-SHA                        RSA        AES         128      TLS_RSA_WITH_AES_128_CBC_SHA                       
 x41     CAMELLIA128-SHA                   RSA        Camellia    128      TLS_RSA_WITH_CAMELLIA_128_CBC_SHA                  
 x0a     DES-CBC3-SHA                      RSA        3DES        168      TLS_RSA_WITH_3DES_EDE_CBC_SHA                      

 Running client simulations via sockets 

 Android 2.3.7                TLSv1.0 DHE-RSA-AES128-SHA, 2048 bit DH
 Android 4.1.1                TLSv1.0 ECDHE-RSA-AES128-SHA, 256 bit ECDH (P-256)
 Android 4.3                  TLSv1.0 ECDHE-RSA-AES128-SHA, 256 bit ECDH (P-256)
 Android 4.4.2                TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
 Android 5.0.0                TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
 Android 6.0                  TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
 Android 7.0                  TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
 Chrome 51 Win 7              TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
 Chrome 57 Win 7              TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
 Firefox 49 Win 7             TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
 Firefox 53 Win 7             TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
 IE 6 XP                      No connection
 IE 7 Vista                   TLSv1.0 ECDHE-RSA-AES128-SHA, 256 bit ECDH (P-256)
 IE 8 XP                      TLSv1.0 DES-CBC3-SHA
 IE 8 Win 7                   TLSv1.0 ECDHE-RSA-AES128-SHA, 256 bit ECDH (P-256)
 IE 11 Win 7                  TLSv1.2 DHE-RSA-AES128-GCM-SHA256, 2048 bit DH
 IE 11 Win 8.1                TLSv1.2 DHE-RSA-AES128-GCM-SHA256, 2048 bit DH
 IE 11 Win Phone 8.1 Update   TLSv1.2 DHE-RSA-AES128-GCM-SHA256, 2048 bit DH
 IE 11 Win 10                 TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
 Edge 13 Win 10               TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
 Edge 13 Win Phone 10         TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
 Opera 17 Win 7               TLSv1.2 ECDHE-RSA-AES128-SHA256, 256 bit ECDH (P-256)
 Safari 5.1.9 OS X 10.6.8     TLSv1.0 ECDHE-RSA-AES128-SHA, 256 bit ECDH (P-256)
 Safari 7 iOS 7.1             TLSv1.2 ECDHE-RSA-AES128-SHA256, 256 bit ECDH (P-256)
 Safari 9 OS X 10.11          TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
 Safari 10 OS X 10.12         TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
 Apple ATS 9 iOS 9            TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
 Tor 17.0.9 Win 7             TLSv1.0 ECDHE-RSA-AES128-SHA, 256 bit ECDH (P-256)
 Java 6u45                    No connection
 Java 7u25                    TLSv1.0 ECDHE-RSA-AES128-SHA, 256 bit ECDH (P-256)
 Java 8u31                    TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
 OpenSSL 1.0.1l               TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)
 OpenSSL 1.0.2e               TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256, 256 bit ECDH (P-256)

 Done 2018-02-15 21:58:43 [ 307s] -->> 69.89.75.196:443 (www.techempower.com) <<--

I tentatively propose TLS versions of two existing test types: Plaintext and Fortunes. Tentatively, TLS-Plaintext and TLS-Fortunes.

The TLS-Plaintext will hardly test the handshake and mostly test the encoding costs.

However, the encoding costs might depend more on the SSL/TLS library (OpenSSL / BearSSL etc') rather than the framework (I'm ignoring buffer consolidation at the moment, although this is definitely a consideration where pipelining is concerned).

I'm wondering if the performance testing costs are worth the possible data mining that would be enabled by the TLS-Plaintext test...?

including a TLS test type is reasonable as long as there are some stacks for which TLS would be included "in-stack." And I know that is satisfied in the real-world because many stacks include nginx or another mainstream web server that has robust TLS capability.

Sidenote: Technically speaking, nginx (and friends) has a known issue with forward security and session cache invalidation. I'm not sure if resolving these issues would effect performance, but I'm quite certain that security is more important than any performance performance penalty that patching these issues might incur.

I'm pointing this out because this might be something we want to test (I have no idea how to do that, though).

Keep in mind that this is a framework benchmark. Choosing to use openssl/bearssl is a choice of the framework of language authors. I don't see why this is any less of a framework test because of that choice.

Absolutely, and from my digging in frameworks the way they use openssl etc can make a massive difference, including unneeded memory copies and locks. What I hope is that this benchmark will highlight and put pressure on these things to be changed.

If every framework ends up with the same Delta cost for https overtime then this test would have achieved it's goal in my mind.

Another 2 minor points:

The client and the database machine at ServerCentral did not have AES-NI instructions support. This should be taken into account for the cipher choice(s) or checked not to be limiting factor;
Some data on the client and the server should be gathered with openssl speed - with and without the -evp option and the chosen cyphers.

I'm using this one: https://testssl.sh/, https://github.com/drwetter/testssl.sh I think it will tick every box: mainly shell script, non-exotic external dependencies, similar functionality to SSL Labs tests.

@zloster That looks fantastic. We'll probably also want to ensure that the load generator uses the agreed-to cipher suite (whatever that ends up being) and only that cipher suite.

Sidenote: Technically speaking, nginx (and friends) has a known issue with forward security and session cache invalidation. I'm not sure if resolving these issues would effect performance, but I'm quite certain that security is more important than any performance performance penalty that patching these issues might incur.

@boazsegev I agree that to whatever degree it is possible, I would like to test configurations that are as secure as current best-practices suggest. If/when there is a resolution to the issue you cite, it would be good to apply that to the test implementations.

Keep in mind that this is a framework benchmark. Choosing to use openssl/bearssl is a choice of the framework of language authors. I don't see why this is any less of a framework test because of that choice.

@RX14 is correct on this point. If the framework has specified a preferred (canonical) TLS provider, we'd want to use that in these test implementations. As we have done elsewhere, in cases where a preferred provider is not specified, we'd like to use a mainstream option such as OpenSSL.

What I hope is that this benchmark will highlight and put pressure on these things to be changed.

@Drawaes Agreed! This is my sentiment generally across everything we do in this project. :)

The client and the database machine at ServerCentral did not have AES-NI instructions support. This should be taken into account for the cipher choice(s) or checked not to be limiting factor;

@zloster We are bringing a new environment (named "Citrine" in the results dashboard for the time being), and these servers include AES-NI.

The point raised regarding testing TLS handshake performance (aka for new connections) vs. encoding/encryption costs is an interesting one, and makes me wonder whether we should configure the Fortunes variant to not allow connection re-use. Or perhaps we need a separate test altogether, e.g. TLS-Handshake, that is effectively plaintext without connection re-use (which of course implies no pipelining too). Having a different test might help avoid any conflation of traditional Fortunes results with the TLS variant.

I, for one, certainly think being able to see a framework's performance characteristics for TLS handshakes vs. encryption is valuable.

One point you will potentially be measuring a whole stack of areas that just aren't measured today and might tell you very little about TLS performance per se.

If this is likely to be a test that is looked at first I would recommend there is a test with zero connection reuse without TLS ( unless I am mistaken and this is what Fortunes is today?) As you would want a baseline for establishing the socket etc before moving onto the handshake itself.

Reason being is that if you have an equal field configuration without session resumption then the only difference in reality is the extra round trips of tiny messages. From my measurements the actual cryptographic calculations are a minimal part of the handshake.

Then of course you have to decide do you allow session resumption/ticketing and which keyshares/curves etc which massively expands the problem.

@Drawaes

If this is likely to be a test that is looked at first I would recommend there is a test with zero connection reuse without TLS ( unless I am mistaken and this is what Fortunes is today?)

https://en.wikipedia.org/wiki/HTTP_persistent_connection#HTTP_1.1 Connection reuse AKA "keep-alive" should be the default behaviour for all tests. The plaintext is using additionally "request pipelining" i.e. issuing new GET operations without receiving the response from the previous HTTP operation (https://en.wikipedia.org/wiki/HTTP_pipelining).

Which is what I thought ;) I guess my concern people will look at this test and think... Wow TLS is expensive when in the real world keep alive is the norm and there is no unencrypted baseline to compare to.

I believe the JSON test would be a better choice than the plaintext test when testing TLS.

As I mentioned here, the plaintext pipelining approach exposes servers to DoS attacks through clients that pipeline large quantities of short requests (preferably requesting a large payload).

A 48 byte request can be repeated ~11 times in a single TCP/IP packet when MTU=576 (which is the common limitation on the web).

An attacker could make hundreds of requests, combined with a slow client approach, to starve a server of resources (for example, consider memory consumption for the buffered responses).

There are two defense measures a security aware framework might implement: 1. throttling (slowing down pipelined requests); and 2. limiting (limiting the number of incomplete downloads for slow clients).

This means that security aware frameworks are likely to perform slower on plaintext tests due to pipelining.

This would result in an unfair advantage for frameworks that ignore this possible attack vector.

Hence, I believe that the JSON test would provide a better representation where security is concerned, since it avoids this question altogether.

Just my 2¢... also, I'm somewhat biased (facil.io throttles pipelined requests).

Hi folks, I'm a bit of a noob regarding all this, but I have some feedback.

I just ran a crude same-machine test for aspcore/json manually and put a https endpoint in. I got roughly a third rps less with ssl (47K vs 70K), which is substantial and was not on my radar when I saw the impressive rps values of the benchmarks (or the concerns raised by @boazsegev).

I don't know if connections were kept alive during my test, the keep-alive header had no effect on the result. I assume the expensive part of ssl is the handshake and done only once for each connection. Is it possible to tell wrk to re-open each connection? There's no obvious option.

TechEmpower / FrameworkBenchmarks

TLS/SSL test type(s) #3290