search

hashicorp / terraform

Terraform enables you to safely and predictably create, change, and improve infrastructure. It is a source-available tool that codifies APIs into declarative configuration files that can be shared amongst team members, treated as code, edited, reviewed, and versioned.
https://www.terraform.io/
Other
42.53k stars 9.52k forks source link

Terraform provider downloads fail with TLS handshake timeout in gitlab CICD pipeline #30528

Closed cryofracture closed 2 years ago

cryofracture commented 2 years ago

Terraform Version 0.10.0

Expected Behavior We are running terraform as part of a GitLab CI system. Each push to the main environment branches (dev, stage, master/prod) triggers a terraform job, starting with terraform init and on through to terraform apply.

Terraform should download the OpenStack provider

Actual Behavior

Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider terraform-provider-openstack/openstack: could not connect to registry.terraform.io: Failed to request discovery document: Get "https://registry.terraform.io/.well-known/terraform.json": net/http: TLS handshake timeout

Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider hashicorp/local: could not connect to registry.terraform.io: Failed to request discovery document: Get "https://registry.terraform.io/.well-known/terraform.json": net/http: TLS handshake timeout

This occurs on the pipeline whether we have set mtu to < 1400 in /etc/docker/daemon.json and we are unable to disable IPv6 inside the containers themselves, but it has been disabled on the host.

Steps to Reproduce

Run terraform apply inside a container (in our case, docker executor gitlab runner on centos 7) which runs a container executing terraform plan

cryofracture commented 2 years ago

I have read over #15817 and tried all the potential solutions in that post, but cannot get the builds to pass this TLS timeout issue.

cryofracture commented 2 years ago

And here is a debug trace from the pipeline and terraform's actions leading up to the failure:

2022-02-15T23:37:47.030Z [DEBUG] Adding temp file log sink: /tmp/terraform-log807523765
2022-02-15T23:37:47.030Z [INFO]  Terraform version: 1.0.10
2022-02-15T23:37:47.030Z [INFO]  Go runtime version: go1.16.4
2022-02-15T23:37:47.030Z [INFO]  CLI args: []string{"/bin/terraform", "init", "-input=false"}
2022-02-15T23:37:47.030Z [TRACE] Stdout is not a terminal
2022-02-15T23:37:47.030Z [TRACE] Stderr is not a terminal
2022-02-15T23:37:47.030Z [TRACE] Stdin is not a terminal
2022-02-15T23:37:47.030Z [DEBUG] Attempting to open CLI config file: /root/.terraformrc
2022-02-15T23:37:47.030Z [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2022-02-15T23:37:47.030Z [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2022-02-15T23:37:47.030Z [DEBUG] ignoring non-existing provider search directory /root/.terraform.d/plugins
2022-02-15T23:37:47.030Z [DEBUG] ignoring non-existing provider search directory /root/.local/share/terraform/plugins
2022-02-15T23:37:47.030Z [DEBUG] ignoring non-existing provider search directory /usr/local/share/terraform/plugins
2022-02-15T23:37:47.030Z [DEBUG] ignoring non-existing provider search directory /usr/share/terraform/plugins
2022-02-15T23:37:47.031Z [INFO]  CLI command args: []string{"init", "-input=false"}
Initializing the backend...
Initializing provider plugins...
- Finding terraform-provider-openstack/openstack versions matching "~> 1.43.0"...
2022-02-15T23:37:47.044Z [TRACE] Meta.Backend: no config given or present on disk, so returning nil config
2022-02-15T23:37:47.044Z [TRACE] Meta.Backend: backend has not previously been initialized in this working directory
2022-02-15T23:37:47.044Z [DEBUG] New state was assigned lineage "6118fae7-012a-6f79-22a1-7b677eb1a177"
2022-02-15T23:37:47.044Z [TRACE] Meta.Backend: using default local state only (no backend configuration, and no existing initialized backend)
2022-02-15T23:37:47.044Z [TRACE] Meta.Backend: instantiated backend of type <nil>
2022-02-15T23:37:47.044Z [DEBUG] checking for provisioner in "."
2022-02-15T23:37:47.045Z [DEBUG] checking for provisioner in "/bin"
2022-02-15T23:37:47.045Z [INFO]  Failed to read plugin lock file .terraform/plugins/linux_amd64/lock.json: open .terraform/plugins/linux_amd64/lock.json: no such file or directory
2022-02-15T23:37:47.045Z [TRACE] Meta.Backend: backend <nil> does not support operations, so wrapping it in a local backend
2022-02-15T23:37:47.045Z [TRACE] backend/local: state manager for workspace "default" will:
 - read initial snapshot from terraform.tfstate
 - write new snapshots to terraform.tfstate
 - create any backup at terraform.tfstate.backup
2022-02-15T23:37:47.045Z [TRACE] statemgr.Filesystem: reading initial snapshot from terraform.tfstate
2022-02-15T23:37:47.045Z [TRACE] statemgr.Filesystem: snapshot file has nil snapshot, but that's okay
2022-02-15T23:37:47.045Z [TRACE] statemgr.Filesystem: read nil snapshot
2022-02-15T23:37:47.045Z [DEBUG] Service discovery for registry.terraform.io at https://registry.terraform.io/.well-known/terraform.json
2022-02-15T23:37:47.045Z [TRACE] HTTP client GET request to https://registry.terraform.io/.well-known/terraform.json
2022-02-15T23:37:50.047Z [ERROR] Checkpoint error: Get "https://checkpoint-api.hashicorp.com/v1/check/terraform?arch=amd64&os=linux&signature=2[26](https://gitdev.arc.ninjaneers.net/npse/dhcp-kea/-/jobs/230#L26)5d2ba-3dd5-b8aa-95cf-858af18c6037&version=1.0.10": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
- Finding latest version of hashicorp/local...
2022-02-15T23:37:57.1[32](https://<gitlab_url>/-/jobs/230#L32)Z [DEBUG] Service discovery for registry.terraform.io at https://registry.terraform.io/.well-known/terraform.json
2022-02-15T23:[37](https://<gitlab_url>/-/jobs/230#L37):[57](https://<gitlab_url>/-/jobs/230#L57).132Z [TRACE] HTTP client GET request to https://registry.terraform.io/.well-known/terraform.json
crw commented 2 years ago

Hi @cryofracture, thanks for the issue. 0.10.0 is no longer supported by the open source team, moreover this version pre-dates some important changes to golang's networking capabilities (specifically enabling traffic to work on "broken" IPv6 networks). This may also be the result of a misconfigured firewall. Upgrading to a more recent version may help solve this issue. If you upgrade and are still having this issue, please let us know and I will re-open this issue, or feel free to open a new issue.

Also just in case you weren't aware, please check out the Community Forum. This repository is only monitored by a few contributors whereas there is a much larger community in the forum to help troubleshoot. Thanks again for this report!

github-actions[bot] commented 2 years ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.