Terraform init with etcdv3 backend hangs on failed SSL handshake

[smayre]

Terraform Version

Terraform v0.11.9

Terraform Configuration Files

terraform {
    backend "etcdv3" {
        endpoints = ["https://etcd-server:2379"]
        prefix = "terraform-state/"
        cacert_path = "/path/to/ca.cert"
    }
}

Debug Output

https://gist.github.com/smayre/c63136ca127ce5b5ee4a3a6fede54d2d

Crash Output

None

Expected Behavior

Terraform should initialise successfully.

Actual Behavior

$ terraform init

Initializing the backend...

Successfully configured the backend "etcdv3"! Terraform will automatically
use this backend unless the backend configuration changes.

Terraform hangs and produces no additional output. No keys are created in etcd.

Steps to Reproduce

With the above configuration file:

terraform init

Additional Context

My trouble shooting indicates that Terraform is not successfully completing the SSL handshake.

It appears that even though cert_path and key_path options are not specified, terraform still tries to send a client certificate. This 'certificate' is invalid and causes a 'bad certificate' error and the SSL handshake fails. Terraform does not handle this gracefully and hangs.

If cert_path and key_path are specified then terraform initialises successfully as expected.

Note: in either case my etcd is not configured to perform client cert authentication.

References

[smayre]

When cert_path and key_path are not specified, openssl indicates a bad certificate error:

$ cat main.tf
terraform {
    backend "etcdv3" {
        endpoints = ["https://etcd-server:2379"]
        prefix = "terraform-state/"
        cacert_path = "/path/to/ca.cert"
    }
}
$ terraform init
...

$ sudo openssl s_server -cert etcd-certs/server.cert -key etcd-certs/server.key -accept 2379
Using default temp DH parameters
ACCEPT
ERROR
140706239375248:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:s3_pkt.c:1493:SSL alert number 42
shutting down SSL
CONNECTION CLOSED
ACCEPT

When cert_path and key_path are specified the SSL connection is successful:

$ cat main.tf
terraform {
    backend "etcdv3" {
        endpoints = ["https://etcd-server:2379"]
        prefix = "terraform-state/"
        cacert_path = "/path/to/ca.cert"
        cert_path = "/path/to/client.cert"
        key_path = "/path/to/client.key"
    }
}
$ terraform init
...

$ sudo openssl s_server -cert etcd-certs/server.cert -key etcd-certs/server.key -accept 2379
Using default temp DH parameters
ACCEPT
-----BEGIN SSL SESSION PARAMETERS-----
MIGEAgEBAgIDAwQCwC8EIJoX3aWf9wLBhksQV4qFCD4KbczuA9oDoHq/M1FzEZxz
BDCPDD7LG9EvRUZ3DQwr1UTxynlP/7yUHuTn0DpgA4WSYwAUWDBkRS4VvyBDoOq7
iJ+hBgIEW8/IjKIEAgIBLKQGBAQBAAAApg0EC29jYy1ldGNkLTAx
-----END SSL SESSION PARAMETERS-----
Shared ciphers:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA:AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:DES-CBC3-SHA
Signature Algorithms: RSA+SHA256:ECDSA+SHA256:RSA+SHA384:ECDSA+SHA384:RSA+SHA512:ECDSA+SHA512:RSA+SHA1:ECDSA+SHA1
Shared Signature Algorithms: RSA+SHA256:ECDSA+SHA256:RSA+SHA384:ECDSA+SHA384:RSA+SHA512:ECDSA+SHA512:RSA+SHA1:ECDSA+SHA1
Supported Elliptic Curve Point Formats: uncompressed
Supported Elliptic Curves: 0x001D:P-256:P-384:P-521
Shared Elliptic curves: P-256:P-384:P-521
CIPHER is ECDHE-RSA-AES128-GCM-SHA256
Secure Renegotiation IS supported
PRI * HTTP/2.0

SM

&=LMedz���ȴ�`+�����@te�M�5<ERROR�__�u�b
shutting down SSL
CONNECTION CLOSED

[timroster]

seems to be happening with 0.12.26 still.

[kcloutie]

Still happening with Terraform v0.12.28

[allen-servedio]

In case this helps, here is how our team's internal docs describe what to do in this situation (assuming it is the same problem we see):

Terraform hangs on terraform init

If Terraform hangs during terraform init and etcdv3 backend is configured, please make sure that backend endpoints are configured correctly. List of endpoints should contain ETCD endpoint hostname and port with protocol scheme.

The reason for that is that the Databases for ETCD cluster is hidden behind a load balancer which terminates SSL. As a consequence, instead validating the ETCD certificate for the ETCD cluster using the client library, Terraform validates the SSL connection for the load balancer using the system trusted CA store. This means that backend SSL configuration in Terraform does not have any effect.

Terraform is unable to verify SSL certificate

If Terraform reports certificate validation errors for the backend, make sure you have added the CA certificate provided to system wide trusted certificates store.

To get CA certificate for SSL connection setup if you do not have it handy:

ibmcloud resource service-key <service-key-name> --output=json | jq --raw-output .[0].credentials.connection.grpc.certificate.certificate_base64 | base64 -D

For instance, on macOS it should be added to System Keychain, under Certificates section, and it's trust settings should be switched to Always trust.

[davidalpert]

This tripped me up today also, because terraform init and terraform plan are both hanging indefinitely with an etcd backend.

In my case it appears to be that golang DNS resolution is failing.

I am working locally with terraform source which has an etcd backend configured with internal DNS names which are resolvable on my corporate network. while working from home over VPN however, I must have something wrong with my MacOS DNS configuration because when I added the three internal etcd server hostnames to my local /etc/hosts file, explicitly mapping the internal IP Addresses, terraform init and terraform plan completed successfully nearly right away.

this leads me to believe that when terraform go code cannot successfully resolve the etcd backend hostnames the code hangs in the same way as the above mentioned SSL/certificate misconfiguration issues.

it would be very helpful if there was some diagnostic output to help identify etcd (or other backend) connection timeouts or failures.

[LeoLiuYan]

Still happening with Terraform v0.13.4.

[chuanran]

still happeinging with Terraform v0.14.5. Is anyone actively working on this issue? Thanks.

[PetzJohannes]

I had the exact same issue. Check out my pull request https://github.com/hashicorp/terraform/pull/28937

Maybe you can test the pull and write a comment.

[apparentlymart]

Hello! Thanks for reporting this.

We removed the etcd backends in Terraform v1.3 because they hasn't had a dedicated maintainer for a few versions now and so they were becoming problematic to keep working in newer Terraform versions. Since these backend are no longer present in the Terraform codebase, I'm going to close this issue.

If you're currently using one of these backends with an older version of Terraform, see Removal of Deprecated State Storage Backends in the Terraform v1.3 upgrade guide for some information on the available options for migration.

Thanks again!

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.