search

hashicorp / terraform

Terraform enables you to safely and predictably create, change, and improve infrastructure. It is a source-available tool that codifies APIs into declarative configuration files that can be shared amongst team members, treated as code, edited, reviewed, and versioned.
https://www.terraform.io
Other
43.16k stars 9.58k forks source link

Error: Failed to install provider, Error while installing microsoft/azuredevops v0.2.1: local error: tls: bad record MAC #31516

Closed sahinM closed 2 years ago

sahinM commented 2 years ago

Terraform Version

Terraform 1.1.7 on ubuntu-20.04

Terraform Configuration Files

main.tf

provider "tls" {}

provider "azurerm" {
  features {
    key_vault {
      purge_soft_deleted_keys_on_destroy         = false
      purge_soft_deleted_certificates_on_destroy = false
      purge_soft_deleted_secrets_on_destroy      = false
    }
  }
}

provider "azurerm" {
  alias           = "shared"
  subscription_id = "<subsription_id"
  features {}
}

terraform {
  backend "azurerm" {}
}

locals {
  module_version              = "2.2.0"
  enable_ddos_protection_plan = var.environment == "prod" ? 1 : 0
  trafe_priority              = var.region == "we" ? 1 : 2
  tags                        = merge(module.global_variables.common_tags, { environment = var.environment })
}

data "azurerm_network_ddos_protection_plan" "ddos_protection_plan" {
  count = local.enable_ddos_protection_plan

  name                = "ddos-prod-we"
  resource_group_name = "rg-prod-global-ddos-we"
}

data "azurerm_client_config" "current" {}

data "azurerm_resources" "log_analytics_workpace" {
  type                = "Microsoft.OperationalInsights/workspaces"
  resource_group_name = "rg-${var.environment}-global-log-we"
}

data "azurerm_log_analytics_workspace" "log_analytics" {
  name                = data.azurerm_resources.log_analytics_workpace.resources[0].name
  resource_group_name = "rg-${var.environment}-global-log-we"
}

data "azurerm_traffic_manager_profile" "traffic_manager_profile" {
  name                = "traf-${var.environment}"
  resource_group_name = "rg-${var.environment}-global-traf-we"
}

# Get the name of the pipeline Key Vault
data "external" "pipeline_key_vault_name" {
  program = [
    "bash",
    "../pipeline.templates/scripts/bash/get-key-vault-name.sh",
  ]
  query = {
    key_vault_resource_group = "rg-${var.environment}-global-kv-pipeline-we"
  }
}

data "azurerm_key_vault" "pipeline_key_vault" {
  name                = data.external.pipeline_key_vault_name.result.key_vault_name
  resource_group_name = data.external.pipeline_key_vault_name.result.key_vault_resource_group
}

module "global_variables" {
  source = "git::https://dev.azure.com/org/project/_git/infra.tf.global.variables?ref=master"
}

module "resource_group" {
  source = "git::https://dev.azure.com/org/project/_git/infra.tf.module.resource-group?ref=1.0.6"

  name     = "rg-${var.environment}-aks-${var.region}"
  location = var.location

  tags = local.tags
}

module "vnet" {
  source = "git::https://dev.azure.com/org/project/_git/infra.tf.module.virtual-network?ref=2.0.0"

  name                    = "vnet-${var.environment}-${var.region}"
  resource_group_name     = module.resource_group.resource_group_name
  ddos_protection_plan_id = local.enable_ddos_protection_plan == 1 ? data.azurerm_network_ddos_protection_plan.ddos_protection_plan[0].id : null
  address_space           = [var.aks_vnet_address_space]
  location                = var.location

  log_analytics_workspace_id = data.azurerm_log_analytics_workspace.log_analytics.id

  tags = local.tags
}

module "aks_subnet" {
  source = "git::https://dev.azure.com/org/project/_git/infra.tf.module.subnet?ref=1.2.0"

  name                 = "snet-${var.environment}-aks-${var.region}"
  resource_group_name  = module.resource_group.resource_group_name
  virtual_network_name = module.vnet.name
  address_prefixes     = [var.aks_subnet_address_prefixes]
}

module "aks_subnet_nsg" {
  source = "git::https://dev.azure.com/org/project/_git/infra.tf.module.network-security-group?ref=2.0.0"

  nsg_group_name      = "nsg-${var.environment}-aks-${var.region}"
  resource_group_name = module.resource_group.resource_group_name
  location            = var.location

  log_analytics_workspace_id = data.azurerm_log_analytics_workspace.log_analytics.id

  tags = local.tags
}

module "route_table" {
  source = "git::https://dev.azure.com/org/project/_git/infra.tf.module.route-table?ref=2.0.0"

  name                          = "route-${var.environment}-aks-${var.region}"
  resource_group_name           = module.resource_group.resource_group_name
  disable_bgp_route_propagation = false
  location                      = var.location

  tags = local.tags
}

# Add aks subnet to route table
module "subnet_route_table_association" {
  source = "git::https://dev.azure.com/org/project/_git/infra.tf.module.subnet-route-table-association?ref=1.0.0"

  subnet_id      = module.aks_subnet.subnet_id
  route_table_id = module.route_table.id
}

# Add nsg to the aks subnet
module "aks_subnet_nsg_group_association" {
  source = "git::https://dev.azure.com/org/project/_git/infra.tf.module.network-security-group-association?ref=1.0.0"

  subnet_id                 = module.aks_subnet.subnet_id
  network_security_group_id = module.aks_subnet_nsg.nsg_group_id
}

# Private link subnet
## Please note that the subnet used for private links cannot be secured with a network security group because this is not supported
## Ref.: https://docs.microsoft.com/en-us/azure/private-link/disable-private-endpoint-network-policy
## Ref.: https://docs.microsoft.com/en-us/azure/private-link/disable-private-link-service-network-policy
module "private_link_subnet" {
  source = "git::https://dev.azure.com/org/project/_git/infra.tf.module.subnet?ref=1.2.0"

  name                                           = "snet-${var.environment}-pl-${var.region}"
  resource_group_name                            = module.resource_group.resource_group_name
  virtual_network_name                           = module.vnet.name
  address_prefixes                               = [var.pl_subnet_address_prefixes]
  enforce_private_link_endpoint_network_policies = true
}

module "aks_gateway_subnet" {
  source = "git::https://dev.azure.com/org/project/_git/infra.tf.module.subnet?ref=1.2.0"

  name                 = "snet-${var.environment}-waf-${var.region}"
  resource_group_name  = module.resource_group.resource_group_name
  virtual_network_name = module.vnet.name
  address_prefixes     = [var.waf_subnet_address_prefixes]
}

module "aks_gateway_subnet_nsg" {
  source = "git::https://dev.azure.com/org/project/_git/infra.tf.module.network-security-group?ref=2.0.0"

  nsg_group_name      = "nsg-${var.environment}-waf-${var.region}"
  resource_group_name = module.resource_group.resource_group_name
  location            = var.location

  log_analytics_workspace_id = data.azurerm_log_analytics_workspace.log_analytics.id

  tags = local.tags
}

# Add nsg to the aks gateway subnet
module "aks_gateway_subnet_nsg_group_association" {
  source = "git::https://dev.azure.com/org/project/_git/infra.tf.module.network-security-group-association?ref=1.0.0"

  subnet_id                 = module.aks_gateway_subnet.subnet_id
  network_security_group_id = module.aks_gateway_subnet_nsg.nsg_group_id
}

# Add a firewall rule to allow azure infrastructure on aks gateway subnet
module "azure_infrastructure_communication_nsg_rule" {
  source = "git::https://dev.azure.com/org/project/_git/infra.tf.module.network-security-rule?ref=1.0.0"

  rule_name                  = "azure_infrastructure_communication_application_gateway_v2_sku"
  priority                   = "100"
  direction                  = "Inbound"
  access                     = "Allow"
  protocol                   = "*"
  source_port_range          = "*"
  destination_port_range     = "65200-65535"
  source_address_prefix      = "*"
  destination_address_prefix = "*"
  resource_group_name        = module.aks_gateway_subnet_nsg.nsg_group_resource_group_name
  nsg_group_name             = module.aks_gateway_subnet_nsg.nsg_group_name
}

# Add firewall rule to access gateway subnet from traffic manager
module "internet_https_to_aks_gateway_subnet_nsg_rule" {
  source = "git::https://dev.azure.com/org/project/_git/infra.tf.module.network-security-rule?ref=1.0.0"

  rule_name                  = "internet_https_to_gateway_subnet"
  priority                   = "200"
  direction                  = "Inbound"
  access                     = "Allow"
  protocol                   = "Tcp"
  source_port_range          = "*"
  destination_port_range     = "443"
  source_address_prefix      = "Internet"
  destination_address_prefix = "VirtualNetwork"
  resource_group_name        = module.aks_gateway_subnet_nsg.nsg_group_resource_group_name
  nsg_group_name             = module.aks_gateway_subnet_nsg.nsg_group_name
}

module "internet_http_to_aks_gateway_subnet_nsg_rule" {
  source = "git::https://dev.azure.com/org/project/_git/infra.tf.module.network-security-rule?ref=1.0.0"

  rule_name                  = "internet_http_to_gateway_subnet"
  priority                   = "210"
  direction                  = "Inbound"
  access                     = "Allow"
  protocol                   = "Tcp"
  source_port_range          = "*"
  destination_port_range     = "80"
  source_address_prefix      = "Internet"
  destination_address_prefix = "VirtualNetwork"
  resource_group_name        = module.aks_gateway_subnet_nsg.nsg_group_resource_group_name
  nsg_group_name             = module.aks_gateway_subnet_nsg.nsg_group_name
}

# Public IP used by the WAF for incoming (ingress) traffic
module "public_ip_waf" {
  source = "git::https://dev.azure.com/org/project/_git/infra.tf.module.public-ip?ref=2.1.0"

  name                = "pip-${var.environment}-waf-${var.region}"
  allocation_method   = "Static"
  resource_group_name = module.resource_group.resource_group_name
  sku                 = "Standard"
  location            = var.location
  zones               = var.availability_zones

  log_analytics_workspace_id = data.azurerm_log_analytics_workspace.log_analytics.id

  tags = local.tags
}

# Public IP used by the AKS for outgoing (egress) traffic
module "public_ip_aks" {
  source = "git::https://dev.azure.com/org/project/_git/infra.tf.module.public-ip?ref=2.1.0"

  name                = "pip-${var.environment}-aks-${var.region}"
  allocation_method   = "Static"
  resource_group_name = module.resource_group.resource_group_name
  sku                 = "Standard"
  location            = var.location
  zones               = var.availability_zones

  log_analytics_workspace_id = data.azurerm_log_analytics_workspace.log_analytics.id

  tags = local.tags
}

# Traffic Manager Endpoint 
## To be able to create an alias record in the DNS zone to support apex (root) domain names
## with Traffic Manager, the endpoint type has to be set to externalEndpoints

## References:
## https://docs.microsoft.com/en-us/azure/dns/dns-alias-appservice#create-endpoints
## https://azure.microsoft.com/en-us/blog/announcing-alias-records-for-azure-dns/
## https://github.com/MicrosoftDocs/azure-docs/issues/18998
module "traffic_manager_endpoint" {
  source = "git::https://dev.azure.com/org/project/_git/infra.tf.module.traffic-manager-external-endpoint?ref=3.0.0"

  name                      = "trafe-${var.environment}-${var.region}"
  trafficmanager_profile_id = data.azurerm_traffic_manager_profile.traffic_manager_profile.id
  target                    = module.public_ip_waf.public_ip_address
  weight                    = local.trafe_priority
  priority                  = local.trafe_priority
}

resource "tls_private_key" "aks_nodes_ssh" {
  algorithm = "RSA"
  rsa_bits  = 4096
}

module "aks_nodes_ssh_public_key" {
  source = "git::https://dev.azure.com/org/project/_git/infra.tf.module.key-vault-secret?ref=2.3.1"

  key_vault_secret_name  = "aks-nodes-ssh-public-key-${var.region}"
  key_vault_secret_value = trimspace(tls_private_key.aks_nodes_ssh.public_key_openssh)
  key_vault_id           = data.azurerm_key_vault.pipeline_key_vault.id

  tags = local.tags
}

module "aks_nodes_ssh_private_key" {
  source = "git::https://dev.azure.com/org/project/_git/infra.tf.module.key-vault-secret?ref=2.3.1"

  key_vault_secret_name  = "aks-nodes-ssh-private-key-${var.region}"
  key_vault_secret_value = trimspace(tls_private_key.aks_nodes_ssh.private_key_openssh)
  key_vault_id           = data.azurerm_key_vault.pipeline_key_vault.id

  tags = local.tags
}

module "aks" {
  source = "git::https://dev.azure.com/org/project/_git/infra.tf.module.aks?ref=4.3.0"

  name                    = "aks-${var.environment}-${var.region}"
  location                = var.location
  resource_group_name     = module.resource_group.resource_group_name
  aks_cluster_version     = "1.22.6"
  sku_tier                = var.aks_sku_tier
  zones                   = var.availability_zones
  vm_size                 = var.aks_vm_node_size
  max_count               = var.aks_max_count
  vnet_subnet_id          = module.aks_subnet.subnet_id
  outbound_ip_address_ids = [module.public_ip_aks.public_ip_id]
  public_ssh_certificate  = module.aks_nodes_ssh_public_key.key_vault_secret_value

  log_analytics_workspace_id = data.azurerm_log_analytics_workspace.log_analytics.id

  tags = local.tags
}

# Add the AKS managed system identity to the built-in Network Contributor role
# in the scope of the resource group where the AKS is created
# This is needed otherwise the cluster fails to provision network resources such as
# load-balancers for example. Ref.: https://github.com/Azure/AKS/issues/1557
data "azurerm_resource_group" "aks_resource_group" {
  name = module.aks.resource_group_name

  depends_on = [module.aks]
}

module "aks_rg_role_assignment_managed_system_identity_network_contributor" {
  source = "git::https://dev.azure.com/org/project/_git/infra.tf.module.role-assignment?ref=1.0.2"

  principal_id         = module.aks.managed_system_identity_id
  role_definition_name = "Network Contributor"
  scope                = data.azurerm_resource_group.aks_resource_group.id
}

# Update Container insights to enable metrics
## This is done simply by creating a role assignment on the cluster for the OMS agent identity with the built-in role "Monitoring Metrics Publisher"
## Ref.: https://docs.microsoft.com/en-us/azure/azure-monitor/containers/container-insights-update-metrics#update-one-cluster-by-using-the-azure-cli
module "aks_role_assignment_oms_agent_identity_monitoring_metrics_publisher" {
  source = "git::https://dev.azure.com/org/project/_git/infra.tf.module.role-assignment?ref=1.0.2"

  principal_id         = module.aks.oms_agent_identity_id
  role_definition_name = "Monitoring Metrics Publisher"
  scope                = module.aks.id
}

# Allow AKS kubelet managed identity to pull images from the container registries
data "azurerm_container_registry" "this" {
  name                = "cr${var.environment}we"
  resource_group_name = "rg-${var.environment}-cr-we"
}

data "azurerm_container_registry" "shared" {
  name                = "crsharedwe"
  resource_group_name = "rg-shared-cr-we"
  provider            = azurerm.shared
}

module "cr_role_assignment_aks_kubelet_identity_acr_pull" {
  source = "git::https://dev.azure.com/org/project/_git/infra.tf.module.role-assignment?ref=1.0.2"

  principal_id         = module.aks.kubelet_identity_id
  role_definition_name = "AcrPull"
  scope                = data.azurerm_container_registry.this.id
}

module "cr_role_assignment_aks_kubelet_identity_acr_shared_pull" {
  source = "git::https://dev.azure.com/org/project/_git/infra.tf.module.role-assignment?ref=1.0.2"

  principal_id         = module.aks.kubelet_identity_id
  role_definition_name = "AcrPull"
  scope                = data.azurerm_container_registry.shared.id
}

versions.tf

terraform {
  required_version = ">= 1.1.7"
  required_providers {
    # https://github.com/hashicorp/terraform-provider-azurerm/blob/main/CHANGELOG.md
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "3.13.0"
    }
    # https://github.com/hashicorp/terraform-provider-external/blob/main/CHANGELOG.md
    external = {
      source  = "hashicorp/external"
      version = "2.2.2"
    }
    # https://github.com/hashicorp/terraform-provider-null/blob/main/CHANGELOG.md
    null = {
      source  = "hashicorp/null"
      version = "3.1.1"
    }
    # https://github.com/microsoft/terraform-provider-azuredevops/blob/main/CHANGELOG.md
    azuredevops = {
      source  = "microsoft/azuredevops"
      version = "0.2.1"
    }
    # https://github.com/hashicorp/terraform-provider-tls/blob/main/CHANGELOG.md
    tls = {
      source  = "hashicorp/tls"
      version = "3.4.0"
    }
  }
}

Debug Output

2022-07-22T05:52:33.7032974Z ##[section]Starting: Terraform Init
2022-07-22T05:52:33.7040918Z ==============================================================================
2022-07-22T05:52:33.7041221Z Task         : Bash
2022-07-22T05:52:33.7041485Z Description  : Run a Bash script on macOS, Linux, or Windows
2022-07-22T05:52:33.7041745Z Version      : 3.201.1
2022-07-22T05:52:33.7041964Z Author       : Microsoft Corporation
2022-07-22T05:52:33.7042279Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/bash
2022-07-22T05:52:33.7042637Z ==============================================================================
2022-07-22T05:52:33.8498322Z Generating script.
2022-07-22T05:52:33.8525184Z ========================== Starting Command Output ===========================
2022-07-22T05:52:33.8529266Z [command]/usr/bin/bash /home/vsts/work/_temp/4866a833-0d7a-413f-8d60-feb3b5c334ad.sh
2022-07-22T05:52:35.1634796Z Initializing modules...
2022-07-22T05:52:35.1749016Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.module.metric-alert?ref=1.2.2 for agw_host_health_alert...
2022-07-22T05:52:35.4027400Z - agw_host_health_alert in .terraform/modules/agw_host_health_alert
2022-07-22T05:52:35.4029041Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.module.aks?ref=4.3.0 for aks...
2022-07-22T05:52:35.6676001Z - aks in .terraform/modules/aks
2022-07-22T05:52:35.6677833Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.module.monitor_activity_log_alert?ref=1.1.0 for aks_admin_access...
2022-07-22T05:52:35.8679107Z - aks_admin_access in .terraform/modules/aks_admin_access
2022-07-22T05:52:35.8680390Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.module.monitor_activity_log_alert?ref=1.1.0 for aks_cluster_deleted...
2022-07-22T05:52:35.8686945Z - aks_cluster_deleted in .terraform/modules/aks_cluster_deleted
2022-07-22T05:52:35.8687899Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.module.subnet?ref=1.2.0 for aks_gateway_subnet...
2022-07-22T05:52:36.0385510Z - aks_gateway_subnet in .terraform/modules/aks_gateway_subnet
2022-07-22T05:52:36.0387495Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.module.network-security-group?ref=2.0.0 for aks_gateway_subnet_nsg...
2022-07-22T05:52:36.2518511Z - aks_gateway_subnet_nsg in .terraform/modules/aks_gateway_subnet_nsg
2022-07-22T05:52:36.2519651Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.module.network-security-group-association?ref=1.0.0 for aks_gateway_subnet_nsg_group_association...
2022-07-22T05:52:36.4743243Z - aks_gateway_subnet_nsg_group_association in .terraform/modules/aks_gateway_subnet_nsg_group_association
2022-07-22T05:52:36.4745126Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.module.metric-alert?ref=1.2.2 for aks_idp_failed_pod...
2022-07-22T05:52:36.4749141Z - aks_idp_failed_pod in .terraform/modules/aks_idp_failed_pod
2022-07-22T05:52:36.4750519Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.module.metric-alert?ref=1.2.2 for aks_ingress_nginx_failed_pod...
2022-07-22T05:52:36.4754180Z - aks_ingress_nginx_failed_pod in .terraform/modules/aks_ingress_nginx_failed_pod
2022-07-22T05:52:36.4755539Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.module.metric-alert?ref=1.2.2 for aks_node_notready_unknown...
2022-07-22T05:52:36.4760129Z - aks_node_notready_unknown in .terraform/modules/aks_node_notready_unknown
2022-07-22T05:52:36.4761475Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.module.key-vault-secret?ref=2.3.1 for aks_nodes_ssh_private_key...
2022-07-22T05:52:36.6533246Z - aks_nodes_ssh_private_key in .terraform/modules/aks_nodes_ssh_private_key
2022-07-22T05:52:36.6535022Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.module.key-vault-secret?ref=2.3.1 for aks_nodes_ssh_public_key...
2022-07-22T05:52:36.6537560Z - aks_nodes_ssh_public_key in .terraform/modules/aks_nodes_ssh_public_key
2022-07-22T05:52:36.6539367Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.module.metric-alert?ref=1.2.2 for aks_oidc_proxy_failed_pod...
2022-07-22T05:52:36.6543652Z - aks_oidc_proxy_failed_pod in .terraform/modules/aks_oidc_proxy_failed_pod
2022-07-22T05:52:36.6545207Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.module.metric-alert?ref=1.2.2 for aks_pip_ddos_alert...
2022-07-22T05:52:36.6553869Z - aks_pip_ddos_alert in .terraform/modules/aks_pip_ddos_alert
2022-07-22T05:52:36.6555322Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.module.role-assignment?ref=1.0.2 for aks_rg_role_assignment_managed_system_identity_network_contributor...
2022-07-22T05:52:36.8387475Z - aks_rg_role_assignment_managed_system_identity_network_contributor in .terraform/modules/aks_rg_role_assignment_managed_system_identity_network_contributor
2022-07-22T05:52:36.8399052Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.module.role-assignment?ref=1.0.2 for aks_role_assignment_oms_agent_identity_monitoring_metrics_publisher...
2022-07-22T05:52:36.8400190Z - aks_role_assignment_oms_agent_identity_monitoring_metrics_publisher in .terraform/modules/aks_role_assignment_oms_agent_identity_monitoring_metrics_publisher
2022-07-22T05:52:36.8401063Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.module.subnet?ref=1.2.0 for aks_subnet...
2022-07-22T05:52:36.8401710Z - aks_subnet in .terraform/modules/aks_subnet
2022-07-22T05:52:36.8402434Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.module.network-security-group?ref=2.0.0 for aks_subnet_nsg...
2022-07-22T05:52:36.8403111Z - aks_subnet_nsg in .terraform/modules/aks_subnet_nsg
2022-07-22T05:52:36.8403921Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.module.network-security-group-association?ref=1.0.0 for aks_subnet_nsg_group_association...
2022-07-22T05:52:36.8404915Z - aks_subnet_nsg_group_association in .terraform/modules/aks_subnet_nsg_group_association
2022-07-22T05:52:36.8405772Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.module.network-security-rule?ref=1.0.0 for azure_infrastructure_communication_nsg_rule...
2022-07-22T05:52:37.1931976Z - azure_infrastructure_communication_nsg_rule in .terraform/modules/azure_infrastructure_communication_nsg_rule
2022-07-22T05:52:37.1934240Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.module.role-assignment?ref=1.0.2 for cr_role_assignment_aks_kubelet_identity_acr_pull...
2022-07-22T05:52:37.1935630Z - cr_role_assignment_aks_kubelet_identity_acr_pull in .terraform/modules/cr_role_assignment_aks_kubelet_identity_acr_pull
2022-07-22T05:52:37.1937022Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.module.role-assignment?ref=1.0.2 for cr_role_assignment_aks_kubelet_identity_acr_shared_pull...
2022-07-22T05:52:37.1938414Z - cr_role_assignment_aks_kubelet_identity_acr_shared_pull in .terraform/modules/cr_role_assignment_aks_kubelet_identity_acr_shared_pull
2022-07-22T05:52:37.1939639Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.global.variables?ref=master for global_variables...
2022-07-22T05:52:37.3684122Z - global_variables in .terraform/modules/global_variables
2022-07-22T05:52:37.3685884Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.module.scheduled-query-rules-alert?ref=1.2.0 for idp_pods_average_cpu_utilization_percentage_above_threshold...
2022-07-22T05:52:37.5332807Z - idp_pods_average_cpu_utilization_percentage_above_threshold in .terraform/modules/idp_pods_average_cpu_utilization_percentage_above_threshold
2022-07-22T05:52:37.5334827Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.module.scheduled-query-rules-alert?ref=1.2.0 for idp_pods_average_memory_utilization_percentage_above_threshold...
2022-07-22T05:52:37.5336347Z - idp_pods_average_memory_utilization_percentage_above_threshold in .terraform/modules/idp_pods_average_memory_utilization_percentage_above_threshold
2022-07-22T05:52:37.5337767Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.module.network-security-rule?ref=1.0.0 for internet_http_to_aks_gateway_subnet_nsg_rule...
2022-07-22T05:52:37.5339285Z - internet_http_to_aks_gateway_subnet_nsg_rule in .terraform/modules/internet_http_to_aks_gateway_subnet_nsg_rule
2022-07-22T05:52:37.5340628Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.module.network-security-rule?ref=1.0.0 for internet_https_to_aks_gateway_subnet_nsg_rule...
2022-07-22T05:52:37.5341932Z - internet_https_to_aks_gateway_subnet_nsg_rule in .terraform/modules/internet_https_to_aks_gateway_subnet_nsg_rule
2022-07-22T05:52:37.5343341Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.module.scheduled-query-rules-alert?ref=1.2.0 for nginx_pods_average_cpu_utilization_percentage_above_threshold...
2022-07-22T05:52:37.5353838Z - nginx_pods_average_cpu_utilization_percentage_above_threshold in .terraform/modules/nginx_pods_average_cpu_utilization_percentage_above_threshold
2022-07-22T05:52:37.5357363Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.module.scheduled-query-rules-alert?ref=1.2.0 for nginx_pods_average_memory_utilization_percentage_above_threshold...
2022-07-22T05:52:37.5371611Z - nginx_pods_average_memory_utilization_percentage_above_threshold in .terraform/modules/nginx_pods_average_memory_utilization_percentage_above_threshold
2022-07-22T05:52:37.5373200Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.module.monitor_activity_log_alert?ref=1.1.0 for node_autoscaling...
2022-07-22T05:52:37.5380299Z - node_autoscaling in .terraform/modules/node_autoscaling
2022-07-22T05:52:37.5382237Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.module.key-vault-access-policy?ref=1.2.1 for pipeline_key_vault_access_policy_waf_user_assigned_identity...
2022-07-22T05:52:37.8055382Z - pipeline_key_vault_access_policy_waf_user_assigned_identity in .terraform/modules/pipeline_key_vault_access_policy_waf_user_assigned_identity
2022-07-22T05:52:37.8057424Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.module.private-endpoint?ref=1.0.0 for private_endpoint_aks_container_registry_shared...
2022-07-22T05:52:37.9752028Z - private_endpoint_aks_container_registry_shared in .terraform/modules/private_endpoint_aks_container_registry_shared
2022-07-22T05:52:37.9753093Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.module.private-dns-zone?ref=2.0.0 for private_link_dns_zone...
2022-07-22T05:52:38.1439231Z - private_link_dns_zone in .terraform/modules/private_link_dns_zone
2022-07-22T05:52:38.1441085Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.module.private-dns-zone-virtual-network-link?ref=1.0.0 for private_link_dns_zone_vnet_link...
2022-07-22T05:52:38.4100463Z - private_link_dns_zone_vnet_link in .terraform/modules/private_link_dns_zone_vnet_link
2022-07-22T05:52:38.4101693Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.module.subnet?ref=1.2.0 for private_link_subnet...
2022-07-22T05:52:38.4102541Z - private_link_subnet in .terraform/modules/private_link_subnet
2022-07-22T05:52:38.4103396Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.module.public-ip?ref=2.1.0 for public_ip_aks...
2022-07-22T05:52:38.6290496Z - public_ip_aks in .terraform/modules/public_ip_aks
2022-07-22T05:52:38.6291635Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.module.public-ip?ref=2.1.0 for public_ip_waf...
2022-07-22T05:52:38.6292433Z - public_ip_waf in .terraform/modules/public_ip_waf
2022-07-22T05:52:38.6293284Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.module.resource-group?ref=1.0.6 for resource_group...
2022-07-22T05:52:38.7897790Z - resource_group in .terraform/modules/resource_group
2022-07-22T05:52:38.7899434Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.module.route-table?ref=2.0.0 for route_table...
2022-07-22T05:52:38.9546820Z - route_table in .terraform/modules/route_table
2022-07-22T05:52:38.9548079Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.module.subnet-route-table-association?ref=1.0.0 for subnet_route_table_association...
2022-07-22T05:52:39.1174813Z - subnet_route_table_association in .terraform/modules/subnet_route_table_association
2022-07-22T05:52:39.1175894Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.module.metric-alert?ref=1.2.2 for traf_all_endpoints_down_alert...
2022-07-22T05:52:39.1176676Z - traf_all_endpoints_down_alert in .terraform/modules/traf_all_endpoints_down_alert
2022-07-22T05:52:39.1177487Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.module.metric-alert?ref=1.2.2 for traf_endpoint_unavailable_alert...
2022-07-22T05:52:39.1178257Z - traf_endpoint_unavailable_alert in .terraform/modules/traf_endpoint_unavailable_alert
2022-07-22T05:52:39.1179134Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.module.traffic-manager-external-endpoint?ref=3.0.0 for traffic_manager_endpoint...
2022-07-22T05:52:39.5364870Z - traffic_manager_endpoint in .terraform/modules/traffic_manager_endpoint
2022-07-22T05:52:39.5365853Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.module.virtual-network?ref=2.0.0 for vnet...
2022-07-22T05:52:39.8126329Z - vnet in .terraform/modules/vnet
2022-07-22T05:52:39.8127715Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.module.waf?ref=3.0.1 for waf...
2022-07-22T05:52:40.0292845Z - waf in .terraform/modules/waf
2022-07-22T05:52:40.0293752Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.module.metric-alert?ref=1.2.2 for waf_pip_ddos_alert...
2022-07-22T05:52:40.0325942Z - waf_pip_ddos_alert in .terraform/modules/waf_pip_ddos_alert
2022-07-22T05:52:40.0326718Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.module.waf-policy?ref=2.1.0 for waf_policy...
2022-07-22T05:52:40.7509001Z - waf_policy in .terraform/modules/waf_policy
2022-07-22T05:52:40.7510554Z Downloading git::https://dev.azure.com/org/project/_git/infra.tf.module.user-assigned-identity?ref=2.0.0 for waf_user_assigned_identity...
2022-07-22T05:52:41.0971226Z - waf_user_assigned_identity in .terraform/modules/waf_user_assigned_identity
2022-07-22T05:52:41.1375399Z 
2022-07-22T05:52:41.1379524Z Initializing the backend...
2022-07-22T05:52:41.2432754Z 
2022-07-22T05:52:41.2434604Z Successfully configured the backend "azurerm"! Terraform will automatically
2022-07-22T05:52:41.2435105Z use this backend unless the backend configuration changes.
2022-07-22T05:52:41.3857048Z 
2022-07-22T05:52:41.3859515Z Initializing provider plugins...
2022-07-22T05:52:41.3860831Z - Finding hashicorp/azurerm versions matching "3.13.0"...
2022-07-22T05:52:41.5626431Z - Finding hashicorp/external versions matching "2.2.2"...
2022-07-22T05:52:41.6010955Z - Finding hashicorp/null versions matching "3.1.1"...
2022-07-22T05:52:41.6411532Z - Finding microsoft/azuredevops versions matching "0.2.1"...
2022-07-22T05:52:41.7657833Z - Finding hashicorp/tls versions matching "3.4.0"...
2022-07-22T05:52:41.8955778Z - Installing hashicorp/external v2.2.2...
2022-07-22T05:52:42.2227587Z - Installed hashicorp/external v2.2.2 (signed by HashiCorp)
2022-07-22T05:52:42.3106086Z - Installing hashicorp/null v3.1.1...
2022-07-22T05:52:42.6220186Z - Installed hashicorp/null v3.1.1 (signed by HashiCorp)
2022-07-22T05:52:43.0532073Z - Installing microsoft/azuredevops v0.2.1...
2022-07-22T05:52:43.8635130Z - Installing hashicorp/tls v3.4.0...
2022-07-22T05:52:44.1812945Z - Installed hashicorp/tls v3.4.0 (signed by HashiCorp)
2022-07-22T05:52:44.2789358Z - Installing hashicorp/azurerm v3.13.0...
2022-07-22T05:52:46.2257859Z - Installed hashicorp/azurerm v3.13.0 (signed by HashiCorp)
2022-07-22T05:52:46.2258242Z 
2022-07-22T05:52:46.2258578Z Error: Failed to install provider
2022-07-22T05:52:46.2258729Z 
2022-07-22T05:52:46.2259081Z Error while installing microsoft/azuredevops v0.2.1: local error: tls: bad
2022-07-22T05:52:46.2259458Z record MAC
2022-07-22T05:52:46.2259572Z 
2022-07-22T05:52:46.2350669Z ##[error]Bash exited with code '1'.
2022-07-22T05:52:46.2374018Z ##[error]Bash wrote one or more lines to the standard error stream.
2022-07-22T05:52:46.2375645Z ##[error]
Error: Failed to install provider

Error while installing microsoft/azuredevops v0.2.1: local error: tls: bad
record MAC

2022-07-22T05:52:46.2383655Z ##[section]Finishing: Terraform Init

Expected Behavior

It should be successfully installed with output:

Initializing provider plugins...
- Finding hashicorp/tls versions matching "3.4.0"...
- Finding hashicorp/azurerm versions matching "3.13.0"...
- Finding hashicorp/external versions matching "2.2.2"...
- Finding hashicorp/null versions matching "3.1.1"...
- Finding microsoft/azuredevops versions matching "0.2.1"...
- Installing hashicorp/tls v3.4.0...
- Installed hashicorp/tls v3.4.0 (signed by HashiCorp)
- Installing hashicorp/azurerm v3.13.0...
- Installed hashicorp/azurerm v3.13.0 (signed by HashiCorp)
- Installing hashicorp/external v2.2.2...
- Installed hashicorp/external v2.2.2 (signed by HashiCorp)
- Installing hashicorp/null v3.1.1...
- Installed hashicorp/null v3.1.1 (signed by HashiCorp)
- Installing microsoft/azuredevops v0.2.1...
- Installed microsoft/azuredevops v0.2.1 (signed by a HashiCorp partner, key ID 6F0B91BDE98478CF)

Actual Behavior

Failed with some TLS/SSL error, always when installing `microsoft/azuredevops` provider. See detailed Debugging Logs above.

Error: Failed to install provider

Error while installing microsoft/azuredevops v0.2.1: local error: tls: bad
record MAC

Steps to Reproduce

terraform init -backend-config="access_key=${ACCESS_KEY}" ${{ parameters.terraform_backend_config }} -no-color

Additional Context

Our Infrastructure is running in the Azure Cloud using Azure YAML Pipelines within Azure DevOps. The provider microsoft/azuredevops v0.2.1 fails in the terraform init step in unpredictable cases, therefore until now the TF_DEBUG modes were not used and activated. We would like to have a reliable running CI/CD, but unfortunately this provider makes it unforseenable. Google helped with some directions as in keywords: tls`, `networkissue,Corrupted data stream along the way,Firewall / Antivirus is the usual suspect`.

Also I noticed that when installing that specific provider microsoft/azuredevops it is Signed by a HashiCorp partner with some key ID, just as a site note, that this differs somehow from the other installed providers. So may here is some reason, that something with the installing and signing process is different.

Hope that helped a bit for insights.

jbardin commented 2 years ago

Hi @sahinM,

Thanks for filing the issue. The hints from your search ar most likely correct, this is happening at a layer outside of Terraform's control, possible within the tls stream itself. Since this isn't something we can do anything about within the Terraform CLI, I'm going to close this issue. Feel free to use the community forum where there are more users familiar with the Azure Pipelines and the particulars of that infrastructure.

Thanks!

radeksimko commented 2 years ago

@sahinM FYI - we have also noticed the same error messages in our CI recently and one more piece of context I'd add here is that the difference between microsoft/azuredevops and hashicorp/* providers is where the provider assets are served/downloaded from. In case of HashiCorp-maintained providers, we usually serve them from releases.hashicorp.com, but assets of pretty much all other providers come from github.com (GitHub's Release API). There will still be some requests to registry.terraform.io for discovering the release assets location and other things, but if you experience this more commonly with providers outside of the hashicorp namespace, it's more likely that there's some kind TLS/networking issue between you and GitHub. Still nothing Terraform can address - as you and James both said, but it may help you with debugging.

I also raised https://github.com/hashicorp/terraform/pull/31524 to make this more obvious.

Hope that helps!

github-actions[bot] commented 2 years ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.