msi-normr commented 7 years ago

Description: ETCD TLS still supports weak 64-bit block ciphers Nessus security scanner indicates ETCD TLS port can still communicate using weak 64-bit block ciphers which is a security vaulnerabilty (SWEET32).

Steps to Reproduce: 1.) Setup ETCD for TLS 2.) Initiate a query to the ETCD https port and specify a weak cipher (ECDHE-RSA-DES-CBC3-SHA, DES-CBC3-SHA) curl -v https://lama1-1-1:4001/v2/members --cert /etc/ssl/local_certs/etcd2/etcd2.crt --key /etc/ssl/local_certs/etcd2/etcd2.key --ciphers ECDHE-RSA-DES-CBC3-SHA

Actual Results: ETCD responded from the query using the weak cipher

cs1-1-1 ~ # curl -v https://lama1-1-1:4001/v2/members --cert /etc/ssl/local_certs/etcd2/etcd2.crt --key /etc/ssl/local_certs/etcd2/etcd2.key --ciphers ECDHE-RSA-DES-CBC3-SHA

Trying 10.10.48.224...
TCP_NODELAY set
Connected to lama1-1-1 (10.10.48.224) port 4001 (#0)
ALPN, offering http/1.1
Cipher selection: ECDHE-RSA-DES-CBC3-SHA
successfully set certificate verify locations:
CAfile: /etc/ssl/certs/ca-certificates.crt CApath: /etc/ssl/certs
TLSv1.2 (OUT), TLS header, Certificate Status (22):
TLSv1.2 (OUT), TLS handshake, Client hello (1):
TLSv1.2 (IN), TLS handshake, Server hello (2):
TLSv1.2 (IN), TLS handshake, Certificate (11):
TLSv1.2 (IN), TLS handshake, Server key exchange (12):
TLSv1.2 (IN), TLS handshake, Server finished (14):
TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
TLSv1.2 (OUT), TLS change cipher, Client hello (1):
TLSv1.2 (OUT), TLS handshake, Finished (20):
TLSv1.2 (IN), TLS change cipher, Client hello (1):
TLSv1.2 (IN), TLS handshake, Finished (20):
SSL connection using TLSv1.2 / ECDHE-RSA-DES-CBC3-SHA
ALPN, server did not agree to a protocol
Server certificate:
subject: C=US; ST=Illinois; L=Chic; O=Mot_S Inc; OU=Test; CN=lama1-1-1
start date: Jul 12 23:28:38 2017 GMT
expire date: Jul 6 23:28:38 2042 GMT
subjectAltName: host "lama1-1-1" matched cert's "lama1-1-1"
issuer: C=US; ST=Illinois; L=Chic; O=Mot_S Inc; OU=Test; CN=Sxxxxxxx_CA_xxxxxxxxxx
SSL certificate verify ok.

GET /v2/members HTTP/1.1 Host: lama1-1-1:4001 User-Agent: curl/7.54.0 Accept: /

< HTTP/1.1 200 OK < Content-Length: 183 < Content-Type: application/json < Date: Thu, 27 Jul 2017 18:56:52 GMT < X-Etcd-Cluster-Id: 7417be5c0ca6813f < {"members":[{"id":"b8c5b89e3b6c0487","name":"f0bc2c5cee094fb7997f02f07cd80a55","peerURLs":["https://cs1-1-1:2380"],"clientURLs":["http://10.10.48.223:2379","https://cs1-1-1:4001"]}]}
Connection #0 to host lama1-1-1 left intact cs1-1-1 ~ # echo $? 0

Expected Results: Expecting ETCD to fail query.

Occurs On: CoreOS 1409.5.0 (Ladybug) ETCD v3.1.6 openssl 1.0.2k

Additional Information:

Openssl indicates this is fixed in v1.1.0 current CoreOS versions still use OpenSSL 1.0.2k
Is there any plans for ETCD to have a http server that will have ciphers configurable.

/usr/lib/systemd/system/etcd-member.service

[Unit] Description=etcd (System Application Container) Documentation=https://github.com/coreos/etcd Wants=network.target Conflicts=etcd.service Conflicts=etcd2.service

[Service] Type=notify Restart=on-failure RestartSec=10s TimeoutStartSec=0 LimitNOFILE=40000

Environment="ETCD_IMAGE_TAG=v3.1.6" Environment="ETCD_NAME=%m" Environment="ETCD_USER=etcd" Environment="ETCD_DATA_DIR=/var/lib/etcd" Environment="RKT_RUN_ARGS=--uuid-file-save=/var/lib/coreos/etcd-member-wrapper.uuid"

ExecStartPre=/usr/bin/mkdir --parents /var/lib/coreos ExecStartPre=-/usr/bin/rkt rm --uuid-file=/var/lib/coreos/etcd-member-wrapper.uuid ExecStart=/usr/lib/coreos/etcd-wrapper $ETCD_OPTS ExecStop=-/usr/bin/rkt stop --uuid-file=/var/lib/coreos/etcd-member-wrapper.uuid

[Install] WantedBy=multi-user.target

/etc/systemd/system/etcd-member.service.d/10-require-certs.conf

[Unit] Requires=link_etcd2_cert.service After=link_etcd2_cert.service

/etc/systemd/system/etcd-member.service.d/50-options.conf

[Service] Environment="ETCD_DISCOVERY=https://discovery.etcd.io/5fe45a47005f6cd5853d4639236b74b9" Environment="ETCD_ADVERTISE_CLIENT_URLS=http://10.10.48.223:2379,https://cs1-1-1:4001" Environment="ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379,https://0.0.0.0:4001" Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=https://cs1-1-1:2380" Environment="ETCD_LISTEN_PEER_URLS=https://cs1-1-1:2380" Environment="ETCD_CERT_FILE=/etc/ssl/certs/etcd2/etcd2.crt" Environment="ETCD_KEY_FILE=/etc/ssl/certs/etcd2/etcd2.key" Environment="ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/certs/etcd2/etcd2-chain.pem" lines 1-43

cs1-1-1 ~ # cat /etc/os-release NAME="Container Linux by CoreOS" ID=coreos VERSION=1409.5.0 VERSION_ID=1409.5.0 BUILD_ID=2017-06-22-2222 PRETTY_NAME="Container Linux by CoreOS 1409.5.0 (Ladybug)" ANSI_COLOR="38;5;75" HOME_URL="https://coreos.com/" BUG_REPORT_URL="https://issues.coreos.com" COREOS_BOARD="amd64-usr"

cs1-1-1 ~ # openssl version OpenSSL 1.0.2k 26 Jan 2017

heyitsanthony commented 7 years ago

If etcd is going to worry about weak ciphers, maybe there should be a FIPS-140 mode? It seems the suggested way is cgo out to a certified library. /cc @xiang90

xiang90 commented 7 years ago

@heyitsanthony

Agreed.

swestcott commented 6 years ago

Weak cipher support in etcd was flagged up in a security test on our Kubernetes cluster too. We can reduce the risk via security group controls, but being able to whitelist ciphers would address the root cause.

swestcott commented 6 years ago

Further to the above, I feel that allowing users to choose ciphers themselves would offer greater flexibility and allow cipher configuration to be tailored for different regulatory environments. For example, PCI DSS refer to NIST SP 800-52 for TLS configuration.

This is the approach Kubernetes have taken via https://github.com/kubernetes/kubernetes/pull/48859

xiang90 commented 6 years ago

@swestcott

It seems k8s approach is simpler, but it requires users to understand exactly what they expect, and it is go specific.

We probably should adopt the similar approach. Would you want to help on this?

Or would @spzala ?

swestcott commented 6 years ago

@xiang90 Understood, but I think it's responsible to expect users working in regulated environments to understand such settings. This also means users can response more quickly to any future cipher compromises.

We'll look to get a PR raised in the new year /cc @ThatsMrTalbot

philips commented 6 years ago

The Kubernetes solution looks reasonable to me.

I feel we should pick some default set of ciphers too. What about this list? https://github.com/ssllabs/research/wiki/SSL-and-TLS-Deployment-Best-Practices#23-use-secure-cipher-suites

Did Kubernetes select a default set? I couldn't tell from glancing at the PR.

xiang90 commented 6 years ago

@philips

I feel we should pick some default set of ciphers too. What about this list?

This might break existing users after a upgrade. Every time we tighten TLS/security or fix security related bugs, we break some users though...

philips commented 6 years ago

True. Is Kubernetes just rolling with all ciphers?

On Fri, Dec 22, 2017 at 1:38 PM Xiang Li notifications@github.com wrote:

@philips https://github.com/philips

I feel we should pick some default set of ciphers too. What about this list?

This might break existing users after a upgrade. Every time we tighten TLS/security or fix security related bugs, we break some users though...

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/coreos/etcd/issues/8320#issuecomment-353678555, or mute the thread https://github.com/notifications/unsubscribe-auth/AACDCNWxY14csJ-MlwqCbQZ0FIbGa3x8ks5tDCFngaJpZM4OlxmM .

-- CTO, CoreOS, Inc Tectonic is enterprise Kubernetes https://coreos.com/tectonic

xiang90 commented 6 years ago

Is Kubernetes just rolling with all ciphers?

The default one is all available ones in Go TLS as far as I can tell.

@swestcott probably have more details to fill in here.

swestcott commented 6 years ago

The default one is all available ones in Go TLS as far as I can tell.

@xiang90 Yes, that's my understanding too. All ciphers by default, with an optional whitelist.

victorgp commented 6 years ago

Yes, in Kubernetes, if the ciphers are not specified, the Go default ones will be used.

xiang90 commented 6 years ago

@swestcott kindly ping. any progress?

ThatsMrTalbot commented 6 years ago

Have the start of something over at https://github.com/ThatsMrTalbot/etcd - it only seems to work when the TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 cipher suite is provided. If it isnt in the list then I get an error in the etcd logs (see below) and openssl s_client -connect 127.0.0.1:2379 -tls1_2 refuses to connect

Error from etcd logs when TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 is not provided:

Failed to dial 0.0.0.0:2379: connection error: desc = "transport: authentication handshake failed: remote error: tls: handshake failure"; please retry.

xiang90 commented 6 years ago

@ThatsMrTalbot

can you send a WIP pr to the etcd upstream repo? then we can discuss there.

knisbet commented 6 years ago

@ThatsMrTalbot are you still working on this?

I'm also looking for this feature, as right now I have some settings hacked into a fork. So I can work on this if needed.

fmazoyer commented 6 years ago

Hi everyone, I have no etcd/security knowledge :-), so I might be asking dumb questions.

Our Security team ran a scan on all our local systems and reported this: Linux,2379,TLS/SSL Server Supports The Use of Static Key Ciphers,"Disable TLS/SSL support for static key

Is this ectd issue linked to 'Sweet32' vulnerability?
Is there a known workaround?

Thanks a lot for your help :-)

hexfusion commented 6 years ago

Our Security team ran a scan on all our local systems and reported this: Linux,2379,TLS/SSL Server Supports The Use of Static Key Ciphers,"Disable TLS/SSL support for static key

Hi @fmazoyer what version of etcd was this scan against?

fmazoyer commented 6 years ago

Hi @hexfusion thanks for the swift reply :-)

It's docker's quay.io/coreos/etcd:v3.2.4

vhosakot commented 5 years ago

I checked all the TLS 1.2 ciphers with etcd 3.3 and found that etcd 3.3 by default (without the --cipher-suites argument) supports the following TLS cipher suites sorted by strength (encryption algorithm key length):

ECDHE-RSA-AES256-GCM-SHA384
ECDHE-RSA-CHACHA20-POLY1305
ECDHE-RSA-AES256-SHA
AES256-GCM-SHA384
AES256-SHA
ECDHE-RSA-AES128-GCM-SHA256
ECDHE-RSA-AES128-SHA
AES128-GCM-SHA256
AES128-SHA

All the TLS 1.2 ciphers were found using the openssl command:

openssl ciphers 'ALL:eNULL:@STRENGTH'

gyuho commented 5 years ago

@vhosakot Right, we default to whatever Go runtime provides (depending on the machine that you run etcd with). You can specify the cipher suites to exclude weaker ones.

vhosakot commented 5 years ago

Thanks @gyuho. Are the ciphers in https://github.com/etcd-io/etcd/blob/dedae6eb7c253635e70403a1fb04842700277b23/pkg/tlsutil/cipher_suites.go#L21-L44 the default ciphers Go runtime provides to etcd?

gyuho commented 5 years ago

@vhosakot I meant to say, default would be empty cipher suite, which then is populated by Go runtime :) Those are only used to check if specified cipher is valid, but we do not provider cipher suite ordering by default (let Go handle it).

vhosakot commented 5 years ago

@gyuho I have etcd up with --cipher-suites TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 and the required keys and certs.

Now, how can I test with a client (like etcdctl or openssl) to verify that these ciphers are supported by etcd? Should the client send one of these ciphers when it connects to etcd?

tremble commented 5 years ago

I've found https://testssl.sh/ ( https://github.com/drwetter/testssl.sh ) helpful for testing ciphers, it doesn't explicitly check that etcd is working behind the ciphers, but it does test the initial SSL/TLS handshakes.

vhosakot commented 5 years ago

@gyuho I started etcd 3.3.10 with --cipher-suites TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384. Etcd is running at 127.0.0.1:12379.

See --cipher-suites at the end in:

/usr/local/bin/etcd --name=contiv-etcd --data-dir=/var/etcd/contiv-data \
  --client-cert-auth --trusted-ca-file=/var/contiv/etcd-secrets/ca.crt \
  --cert-file=/var/contiv/etcd-secrets/etcd-server.crt \
  --key-file=/var/contiv/etcd-secrets/etcd-server.key \
  --peer-client-cert-auth --peer-trusted-ca-file=/var/contiv/etcd-secrets/ca.crt \
  --peer-cert-file=/var/contiv/etcd-secrets/etcd-server.crt \
  --peer-key-file=/var/contiv/etcd-secrets/etcd-server.key \
  --advertise-client-urls=https://0.0.0.0:12379 \
  --listen-client-urls=https://0.0.0.0:12379 --listen-peer-urls=https://0.0.0.0:12380 \
  --cipher-suites TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

But, when I test these three ciphers using openssl, I see just the ciphers TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 and TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 are accepted by etcd and not TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256. See outputs below:

TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 and TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 work:

$ openssl s_client -cipher ECDHE-RSA-AES128-GCM-SHA256 -connect 127.0.0.1:12379 | grep 'supported\|Cipher'
New, TLSv1.2, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Secure Renegotiation IS supported
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256

$ openssl s_client -cipher ECDHE-RSA-AES256-GCM-SHA384 -connect 127.0.0.1:12379 | grep 'supported\|Cipher'
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Secure Renegotiation IS supported
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384

But, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 does not work:

$ openssl s_client -cipher ECDHE-ECDSA-AES128-GCM-SHA256 -connect 127.0.0.1:12379 | grep 'supported\|Cipher'
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
    Cipher    : 0000

Let me know if I need to open an issue against etcd for this. I'm using openssl 1.1.0g.

Full outputs below:

TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 and TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 work:

$ openssl s_client -cipher ECDHE-RSA-AES128-GCM-SHA256 -connect 127.0.0.1:12379
CONNECTED(00000003)
depth=0 CN = contiv-vpp-etcd-server
---
Certificate chain
 0 s:/CN=contiv-vpp-etcd-server
   i:/CN=contiv-vpp
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDmzCCAoOgAwIBAgIJAPL98Lhvl9VwMA0GCSqGSIb3DQEBCwUAMBUxEzARBgNV
BAMMCmNvbnRpdi12cHAwHhcNMTgxMTI3MjAwNTQwWhcNMjExMTI2MjAwNTQwWjAh
MR8wHQYDVQQDDBZjb250aXYtdnBwLWV0Y2Qtc2VydmVyMIIBIjANBgkqhkiG9w0B
AQEFAAOCAQ8AMIIBCgKCAQEA7CFcVcdQprkM5eOkPT1CKTyoIs5q1ghqpRknyMyb
joAsfW5uYXR261546WtjbspMcqj5jRdkxt12jLVW5ftmQUfETgGD+RO7GetK0j+G
ULnBDn9nx5rvT1RaStliBc8m6QCmnSWrRXxr8KfENHwzLqvEfH4G7oKCQ1+iR3C4
8LbTBi/ZiZjnF9hlzcXxEBOCEj9BEZ2yz5m9Wirb9a4k20mgvd/NOdebNFvkFEh9
nghed4DTD0gMTt/8mCda2NbJBJI6TqsvA42fBEjyzt3Fy6/p8Cl1wudiCHh75j1N
qiFfFEbi9V2Dtz1V543LaxQ/5sPbAkeBalXvi79bZHLpeQIDAQABo4HhMIHeMA4G
A1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDAYD
VR0TAQH/BAIwADAdBgNVHQ4EFgQUDpBvd+jPqYm1raWOQCXIqt5x1fAwRQYDVR0j
BD4wPIAUNyNVb8YSqKol+YFlgpfzrUMqMyKhGaQXMBUxEzARBgNVBAMMCmNvbnRp
di12cHCCCQCRTCQQha8XqTA5BgNVHREEMjAwgh12aG9zYWtvdC1kZXYtbWFzdGVy
MTE3MzdlMGJkMIIJbG9jYWxob3N0hwR/AAABMA0GCSqGSIb3DQEBCwUAA4IBAQB9
ypYXz4+iK0h7ncM1GQoeA9c/oHyXDJ0d2S9rtnlefIRaUdIVEJmLd0UzmRmac8ka
JjDuaoKkkR/DFp5vvkujwPUj/uqhM7A97R1SiNHJEo0sau53A6KRJivu3o5tyVhw
pf2S5b+RWaAUUuRS9KXbt7/gCm573cUbknlw7XRO+l78VITsiMyHSt2ahx7K9i/f
qaWNy6IABw+lDJsd3MOdBayEBcoX3/Nsh+JSyUQPAxNd4e2FoMP6eH7Atjm0V4Gi
Xe7KQqmiVG19TNvG9nPELXVw6g2yGoT8qGha3hnzngoyDpVnxS1GTtxf79KVtguv
oZbNG5U+6+dqxddxr3YW
-----END CERTIFICATE-----
subject=/CN=contiv-vpp-etcd-server
issuer=/CN=contiv-vpp
---
Acceptable client certificate CA names
/CN=contiv-vpp
Client Certificate Types: RSA sign, ECDSA sign
Requested Signature Algorithms: RSA+SHA256:ECDSA+SHA256:RSA+SHA384:ECDSA+SHA384:RSA+SHA512:ECDSA+SHA512:RSA+SHA1:ECDSA+SHA1
Shared Requested Signature Algorithms: RSA+SHA256:ECDSA+SHA256:RSA+SHA384:ECDSA+SHA384:RSA+SHA512:ECDSA+SHA512:RSA+SHA1:ECDSA+SHA1
Peer signing digest: SHA512
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1378 bytes and written 229 bytes
Verification error: unable to verify the first certificate
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: 
    Session-ID-ctx: 
    Master-Key: 51383BDD9643331A5D7DF6BABF279FE854581C048B40B2FD63690BD79BD9B27FD73B134CA099AF81CC4569920AEDE35A
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1543860957
    Timeout   : 7200 (sec)
    Verify return code: 21 (unable to verify the first certificate)
    Extended master secret: no
---

$ openssl s_client -cipher ECDHE-RSA-AES256-GCM-SHA384 -connect 127.0.0.1:12379
CONNECTED(00000003)
depth=0 CN = contiv-vpp-etcd-server
---
Certificate chain
 0 s:/CN=contiv-vpp-etcd-server
   i:/CN=contiv-vpp
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDmzCCAoOgAwIBAgIJAPL98Lhvl9VwMA0GCSqGSIb3DQEBCwUAMBUxEzARBgNV
BAMMCmNvbnRpdi12cHAwHhcNMTgxMTI3MjAwNTQwWhcNMjExMTI2MjAwNTQwWjAh
MR8wHQYDVQQDDBZjb250aXYtdnBwLWV0Y2Qtc2VydmVyMIIBIjANBgkqhkiG9w0B
AQEFAAOCAQ8AMIIBCgKCAQEA7CFcVcdQprkM5eOkPT1CKTyoIs5q1ghqpRknyMyb
joAsfW5uYXR261546WtjbspMcqj5jRdkxt12jLVW5ftmQUfETgGD+RO7GetK0j+G
ULnBDn9nx5rvT1RaStliBc8m6QCmnSWrRXxr8KfENHwzLqvEfH4G7oKCQ1+iR3C4
8LbTBi/ZiZjnF9hlzcXxEBOCEj9BEZ2yz5m9Wirb9a4k20mgvd/NOdebNFvkFEh9
nghed4DTD0gMTt/8mCda2NbJBJI6TqsvA42fBEjyzt3Fy6/p8Cl1wudiCHh75j1N
qiFfFEbi9V2Dtz1V543LaxQ/5sPbAkeBalXvi79bZHLpeQIDAQABo4HhMIHeMA4G
A1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDAYD
VR0TAQH/BAIwADAdBgNVHQ4EFgQUDpBvd+jPqYm1raWOQCXIqt5x1fAwRQYDVR0j
BD4wPIAUNyNVb8YSqKol+YFlgpfzrUMqMyKhGaQXMBUxEzARBgNVBAMMCmNvbnRp
di12cHCCCQCRTCQQha8XqTA5BgNVHREEMjAwgh12aG9zYWtvdC1kZXYtbWFzdGVy
MTE3MzdlMGJkMIIJbG9jYWxob3N0hwR/AAABMA0GCSqGSIb3DQEBCwUAA4IBAQB9
ypYXz4+iK0h7ncM1GQoeA9c/oHyXDJ0d2S9rtnlefIRaUdIVEJmLd0UzmRmac8ka
JjDuaoKkkR/DFp5vvkujwPUj/uqhM7A97R1SiNHJEo0sau53A6KRJivu3o5tyVhw
pf2S5b+RWaAUUuRS9KXbt7/gCm573cUbknlw7XRO+l78VITsiMyHSt2ahx7K9i/f
qaWNy6IABw+lDJsd3MOdBayEBcoX3/Nsh+JSyUQPAxNd4e2FoMP6eH7Atjm0V4Gi
Xe7KQqmiVG19TNvG9nPELXVw6g2yGoT8qGha3hnzngoyDpVnxS1GTtxf79KVtguv
oZbNG5U+6+dqxddxr3YW
-----END CERTIFICATE-----
subject=/CN=contiv-vpp-etcd-server
issuer=/CN=contiv-vpp
---
Acceptable client certificate CA names
/CN=contiv-vpp
Client Certificate Types: RSA sign, ECDSA sign
Requested Signature Algorithms: RSA+SHA256:ECDSA+SHA256:RSA+SHA384:ECDSA+SHA384:RSA+SHA512:ECDSA+SHA512:RSA+SHA1:ECDSA+SHA1
Shared Requested Signature Algorithms: RSA+SHA256:ECDSA+SHA256:RSA+SHA384:ECDSA+SHA384:RSA+SHA512:ECDSA+SHA512:RSA+SHA1:ECDSA+SHA1
Peer signing digest: SHA512
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1378 bytes and written 229 bytes
Verification error: unable to verify the first certificate
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 
    Session-ID-ctx: 
    Master-Key: 71D4C68150B5A92ACD20FE9CAB3FA6C81F7D2C775C98AA626B9B63065F33A42D939DE62A504514C6C0F9D84317D67A87
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1543861072
    Timeout   : 7200 (sec)
    Verify return code: 21 (unable to verify the first certificate)
    Extended master secret: no
---

But, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 does not work:

$ openssl s_client -cipher ECDHE-ECDSA-AES128-GCM-SHA256 -connect 127.0.0.1:12379
CONNECTED(00000003)
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 124 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000
    Session-ID: 
    Session-ID-ctx: 
    Master-Key: 
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1543861174
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
---

hexfusion commented 5 years ago

But, when I test these three ciphers using openssl, I see just the ciphers TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 and TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 are accepted by etcd and not TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256. See outputs below:

@vhosakot I believe your openssl needs to support the cipher locally. openssl ciphers will generate a list. Are the ciphers that fail on that list?

vhosakot commented 5 years ago

@hexfusion Yes, the cipher TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 that fails is supported locally by openssl and is set in --cipher-suites for etcd, but still fails. Isn't this an etcd issue?

$ openssl ciphers -v -V | grep ECDHE-ECDSA-AES128-GCM-SHA256
          0xC0,0x2B - ECDHE-ECDSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH     Au=ECDSA Enc=AESGCM(128) Mac=AEAD

$ openssl version
OpenSSL 1.1.0g  2 Nov 2017

hexfusion commented 5 years ago

@vhosakot I guess the server key algorithm is probably rsa instead of ecdsa? I don't believe it can be both. Curious if changing that gives success.

My guess is your server cert will have similar output.

openssl x509 -in server.pem -text -noout 
[..]
Signature Algorithm: sha256WithRSAEncryption 
[..]

example using cfssl CA format for ecdsa algorithm

  "key": {
    "algo": "ecdsa", <-- not rsa
    "size": 256
 },

vhosakot commented 5 years ago

Thanks @hexfusion. So, should the following certs and keys I'm using for etcd server support ecdsa instead of rsa (they are using rsa currently). If so, I'll test with ecdsa certs and keys for etcd server.

  --trusted-ca-file=/var/contiv/etcd-secrets/ca.crt \
  --cert-file=/var/contiv/etcd-secrets/etcd-server.crt \
  --key-file=/var/contiv/etcd-secrets/etcd-server.key \
  --peer-trusted-ca-file=/var/contiv/etcd-secrets/ca.crt \
  --peer-cert-file=/var/contiv/etcd-secrets/etcd-server.crt \
  --peer-key-file=/var/contiv/etcd-secrets/etcd-server.key

hexfusion commented 5 years ago

Thanks @hexfusion. So, should the following certs and keys I'm using for etcd server support ecdsa instead of rsa (they are using rsa currently). If so, I'll test with ecdsa certs and keys for etcd server.

If you want to use ecdsa vs rsa ciphers that makes sense to me.

ECDHE-ECDSA-AES128-GCM-SHA256

vs

ECDHE_RSA_WITH_AES_128_GCM_SHA256

vhosakot commented 5 years ago

Thanks @hexfusion! I'll test with ecdsa certs and keys for etcd server.

vhosakot commented 5 years ago

@gyuho I see the following ciphers do not work and etcd fails to start with the following errors when I configure these ciphers for --cipher-suites. Aren't the following cipher suites supported by etcd?

2018-12-04 23:59:23.323820 C | etcdmain: unexpected TLS cipher suite "TLS_DHE_RSA_WITH_AES_128_GCM_SHA256"
2018-12-05 00:12:23.287448 C | etcdmain: unexpected TLS cipher suite "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256"
2018-12-05 00:39:15.266940 C | etcdmain: unexpected TLS cipher suite "TLS_DHE_RSA_WITH_AES_128_CBC_SHA"
2018-12-05 00:41:38.286827 C | etcdmain: unexpected TLS cipher suite "TLS_DHE_RSA_WITH_AES_128_CBC_SHA256"
2018-12-05 00:45:04.253193 C | etcdmain: unexpected TLS cipher suite "TLS_DHE_RSA_WITH_AES_256_GCM_SHA384"
2018-12-05 00:47:55.280075 C | etcdmain: unexpected TLS cipher suite "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384"
2018-12-05 00:51:46.264873 C | etcdmain: unexpected TLS cipher suite "TLS_RSA_WITH_AES_256_CBC_SHA256"

hexfusion commented 5 years ago

Hi @vhosakot did the other ciphers work with ecdsa cert?

@gyuho I see the following ciphers do not work and etcd fails to start with the following errors when I configure these ciphers for --cipher-suites. Aren't the following cipher suites supported by etcd?

The support or non support of a cipher is most likely going to be openssl vs etcd issue. For example my openssl 1.1.1 does not support any of the ciphers that are failing can you verify both the client/server do in your case also that you are using rsa certs?

vhosakot commented 5 years ago

@hexfusion I'm not using ecdsa certs and keys for etcd currently, just rsa as etcd is using rsa certs and keys currently.

In the following cipher suites that fail, I see just DHE-RSA-AES128-GCM-SHA256 and DHE-RSA-AES256-GCM-SHA384 supported by openssl (in the output of openssl ciphers 'ALL:eNULL:@STRENGTH') and not the others. I am using openssl 1.1.0g:

2018-12-04 23:59:23.323820 C | etcdmain: unexpected TLS cipher suite "TLS_DHE_RSA_WITH_AES_128_GCM_SHA256"
2018-12-05 00:12:23.287448 C | etcdmain: unexpected TLS cipher suite "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256"
2018-12-05 00:39:15.266940 C | etcdmain: unexpected TLS cipher suite "TLS_DHE_RSA_WITH_AES_128_CBC_SHA"
2018-12-05 00:41:38.286827 C | etcdmain: unexpected TLS cipher suite "TLS_DHE_RSA_WITH_AES_128_CBC_SHA256"
2018-12-05 00:45:04.253193 C | etcdmain: unexpected TLS cipher suite "TLS_DHE_RSA_WITH_AES_256_GCM_SHA384"
2018-12-05 00:47:55.280075 C | etcdmain: unexpected TLS cipher suite "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384"
2018-12-05 00:51:46.264873 C | etcdmain: unexpected TLS cipher suite "TLS_RSA_WITH_AES_256_CBC_SHA256"

$ openssl version
OpenSSL 1.1.0g  2 Nov 2017

So, are cipher suites not supported by openssl not supported by etcd too?

hexfusion commented 5 years ago

So, are cipher suites not supported by openssl not supported by etcd too?

@vhosakot yeah sorry that not incorrect I took a deeper look. It seems that currently the only supported list of definable ciphers is listed in your previous comment.

https://github.com/etcd-io/etcd/blob/dedae6eb7c253635e70403a1fb04842700277b23/pkg/tlsutil/cipher_suites.go#L21-L44

https://github.com/etcd-io/etcd/blob/829c9b2129dd02fbf8caf7f1d322fbe7011321c4/embed/config.go#L529-L535

vhosakot commented 5 years ago

Thanks @hexfusion for confirming that those keys aren't supported by etcd. I'll let you know once I test with ecdsa certs and keys for etcd.

etcd-io / etcd

ETCD TLS still supports weak 64-bit block ciphers #8320

/usr/lib/systemd/system/etcd-member.service

/etc/systemd/system/etcd-member.service.d/10-require-certs.conf

/etc/systemd/system/etcd-member.service.d/50-options.conf