search

josenk / terraform-provider-esxi

Terraform-provider-esxi plugin
GNU General Public License v3.0
544 stars 154 forks source link

Connection failure to esxi host #133

Closed vukzeka closed 3 years ago

vukzeka commented 3 years ago

Hi @josenk thanks for providing your plugin. Very useful. I am trying to use it with latest Terraform version but getting error described bellow. All was good when I was checking earlier with previous version(Terraform 0.11 and plugin 1.5.3). Not sure if I am missing something? If any hints what to check please let me know.

Describe the bug Plugin is trowing error on testing connection although from ssh is possible to connect to esxi host using exactly same connection params(triple checked). It is not possible to connect after 3 retries and then it fails which is expected. As mentioned all good when connection is tested from ssh.

To Reproduce Steps to reproduce the behaviour:

  1. terraform init
  2. terraform plan
  3. See error 
    2020-11-10T11:25:48.446+0100 [DEBUG] plugin.terraform-provider-esxi_v1.8.1: pid-82328-esxi_remote_cmds.go:61: [runRemoteSshCommand] :Connectivity test, get vmware version
2020-11-10T11:25:48.488+0100 [DEBUG] plugin.terraform-provider-esxi_v1.8.1: pid-82328-esxi_remote_cmds.go:41: [runRemoteSshCommand] Retry connection: 3
2020-11-10T11:25:49.511+0100 [DEBUG] plugin.terraform-provider-esxi_v1.8.1: pid-82328-esxi_remote_cmds.go:41: [runRemoteSshCommand] Retry connection: 2
2020-11-10T11:25:50.536+0100 [DEBUG] plugin.terraform-provider-esxi_v1.8.1: pid-82328-esxi_remote_cmds.go:41: [runRemoteSshCommand] Retry connection: 1
2020-11-10T11:25:51.539+0100 [DEBUG] plugin.terraform-provider-esxi_v1.8.1: pid-82328-esxi_remote_cmds.go:72: [runRemoteSshCommand] Failed err: Client Connection Error
2020/11/10 11:25:51 [ERROR] eval: *terraform.EvalConfigProvider, err: Failed to connect to esxi host: Client Connection Error

Expected behaviour Expected is to get plan produced.

Terraform files versions.tf

terraform {
  required_version = ">= 0.13"
  required_providers {
    esxi = {
      source = "registry.terraform.io/josenk/esxi"
    }
  }
}

main.tf

provider "esxi" {
  esxi_hostname = var.esxi_hostname
  esxi_hostport = var.esxi_hostport
  esxi_hostssl  = var.esxi_hostssl
  esxi_username = var.esxi_username
  esxi_password = var.esxi_password
}

resource "esxi_guest" "vmxtest" {
  guest_name = "vmxtest"
  disk_store = "Datastore1"

  network_interfaces {
    virtual_network = "VM Network"
  }
}

Variables not provided but all triple checked.

Versions:

Additional context

2020/11/10 11:19:19 [INFO] backend/local: plan calling Plan 2020/11/10 11:19:19 [INFO] terraform: building graph: GraphTypePlan 2020/11/10 11:19:19 [TRACE] Executing graph transform *terraform.ConfigTransformer 2020/11/10 11:19:19 [TRACE] ConfigTransformer: Starting for path:

2020/11/10 11:19:19 [TRACE] Completed graph transform terraform.ConfigTransformer with new graph: esxi_guest.vmxtest (expand) - terraform.nodeExpandPlannableResource

2020/11/10 11:19:19 [TRACE] Executing graph transform terraform.LocalTransformer 2020/11/10 11:19:19 [TRACE] Completed graph transform terraform.LocalTransformer (no changes) 2020/11/10 11:19:19 [TRACE] Executing graph transform terraform.OutputTransformer 2020/11/10 11:19:19 [TRACE] OutputTransformer: adding ip_addr as terraform.nodeExpandOutput 2020/11/10 11:19:19 [TRACE] Completed graph transform terraform.OutputTransformer with new graph: esxi_guest.vmxtest (expand) - terraform.nodeExpandPlannableResource output.ip_addr (expand) - *terraform.nodeExpandOutput

2020/11/10 11:19:19 [TRACE] Executing graph transform terraform.OrphanResourceInstanceTransformer 2020/11/10 11:19:19 [TRACE] Completed graph transform terraform.OrphanResourceInstanceTransformer (no changes) 2020/11/10 11:19:19 [TRACE] Executing graph transform terraform.StateTransformer 2020/11/10 11:19:19 [TRACE] StateTransformer: state is empty, so nothing to do 2020/11/10 11:19:19 [TRACE] Completed graph transform terraform.StateTransformer (no changes) 2020/11/10 11:19:19 [TRACE] Executing graph transform terraform.OrphanOutputTransformer 2020/11/10 11:19:19 [TRACE] Completed graph transform terraform.OrphanOutputTransformer (no changes) 2020/11/10 11:19:19 [TRACE] Executing graph transform terraform.AttachResourceConfigTransformer 2020/11/10 11:19:19 [TRACE] AttachResourceConfigTransformer: attaching to "esxi_guest.vmxtest (expand)" (terraform.nodeExpandPlannableResource) config from main.tf:9,1-32 2020/11/10 11:19:19 [TRACE] AttachResourceConfigTransformer: attaching provider meta configs to esxi_guest.vmxtest (expand) 2020/11/10 11:19:19 [TRACE] Completed graph transform terraform.AttachResourceConfigTransformer (no changes) 2020/11/10 11:19:19 [TRACE] Executing graph transform terraform.AttachStateTransformer 2020/11/10 11:19:19 [TRACE] Completed graph transform terraform.AttachStateTransformer (no changes) 2020/11/10 11:19:19 [TRACE] Executing graph transform terraform.RootVariableTransformer 2020/11/10 11:19:19 [TRACE] Completed graph transform terraform.RootVariableTransformer with new graph: esxi_guest.vmxtest (expand) - terraform.nodeExpandPlannableResource output.ip_addr (expand) - terraform.nodeExpandOutput var.esxi_hostname - terraform.NodeRootVariable var.esxi_hostport - terraform.NodeRootVariable var.esxi_hostssl - terraform.NodeRootVariable var.esxi_password - terraform.NodeRootVariable var.esxi_username - terraform.NodeRootVariable

2020/11/10 11:19:19 [TRACE] Executing graph transform terraform.MissingProvisionerTransformer 2020/11/10 11:19:19 [TRACE] Completed graph transform terraform.MissingProvisionerTransformer (no changes) 2020/11/10 11:19:19 [TRACE] Executing graph transform terraform.ProvisionerTransformer 2020/11/10 11:19:19 [TRACE] Completed graph transform terraform.ProvisionerTransformer (no changes) 2020/11/10 11:19:19 [TRACE] Executing graph transform terraform.ModuleVariableTransformer 2020/11/10 11:19:19 [TRACE] Completed graph transform terraform.ModuleVariableTransformer (no changes) 2020/11/10 11:19:19 [TRACE] Executing graph transform terraform.graphTransformerMulti 2020/11/10 11:19:19 [TRACE] (graphTransformerMulti) Executing graph transform terraform.ProviderConfigTransformer 2020/11/10 11:19:19 [TRACE] ProviderConfigTransformer: attaching to "provider[\"registry.terraform.io/josenk/esxi\"]" provider configuration from main.tf:1,1-16 2020/11/10 11:19:19 [TRACE] (graphTransformerMulti) Completed graph transform terraform.ProviderConfigTransformer with new graph: esxi_guest.vmxtest (expand) - terraform.nodeExpandPlannableResource output.ip_addr (expand) - terraform.nodeExpandOutput provider["registry.terraform.io/josenk/esxi"] - terraform.NodeApplyableProvider var.esxi_hostname - terraform.NodeRootVariable var.esxi_hostport - terraform.NodeRootVariable var.esxi_hostssl - terraform.NodeRootVariable var.esxi_password - terraform.NodeRootVariable var.esxi_username - *terraform.NodeRootVariable

2020/11/10 11:19:19 [TRACE] (graphTransformerMulti) Executing graph transform terraform.MissingProviderTransformer 2020/11/10 11:19:19 [TRACE] (graphTransformerMulti) Completed graph transform terraform.MissingProviderTransformer (no changes) 2020/11/10 11:19:19 [TRACE] (graphTransformerMulti) Executing graph transform terraform.ProviderTransformer 2020/11/10 11:19:19 [TRACE] ProviderTransformer: exact match for provider["registry.terraform.io/josenk/esxi"] serving esxi_guest.vmxtest (expand) 2020/11/10 11:19:19 [DEBUG] ProviderTransformer: "esxi_guest.vmxtest (expand)" (terraform.nodeExpandPlannableResource) needs provider["registry.terraform.io/josenk/esxi"] 2020/11/10 11:19:19 [TRACE] (graphTransformerMulti) Completed graph transform terraform.ProviderTransformer with new graph: esxi_guest.vmxtest (expand) - terraform.nodeExpandPlannableResource provider["registry.terraform.io/josenk/esxi"] - terraform.NodeApplyableProvider output.ip_addr (expand) - terraform.nodeExpandOutput provider["registry.terraform.io/josenk/esxi"] - terraform.NodeApplyableProvider var.esxi_hostname - terraform.NodeRootVariable var.esxi_hostport - terraform.NodeRootVariable var.esxi_hostssl - terraform.NodeRootVariable var.esxi_password - terraform.NodeRootVariable var.esxi_username - terraform.NodeRootVariable

2020/11/10 11:19:19 [TRACE] (graphTransformerMulti) Executing graph transform terraform.PruneProviderTransformer 2020/11/10 11:19:19 [TRACE] (graphTransformerMulti) Completed graph transform terraform.PruneProviderTransformer (no changes) 2020/11/10 11:19:19 [TRACE] (graphTransformerMulti) Executing graph transform terraform.ParentProviderTransformer 2020/11/10 11:19:19 [TRACE] (graphTransformerMulti) Completed graph transform terraform.ParentProviderTransformer (no changes) 2020/11/10 11:19:19 [TRACE] Completed graph transform terraform.graphTransformerMulti with new graph: esxi_guest.vmxtest (expand) - terraform.nodeExpandPlannableResource provider["registry.terraform.io/josenk/esxi"] - terraform.NodeApplyableProvider output.ip_addr (expand) - terraform.nodeExpandOutput provider["registry.terraform.io/josenk/esxi"] - terraform.NodeApplyableProvider var.esxi_hostname - terraform.NodeRootVariable var.esxi_hostport - terraform.NodeRootVariable var.esxi_hostssl - terraform.NodeRootVariable var.esxi_password - terraform.NodeRootVariable var.esxi_username - terraform.NodeRootVariable

2020/11/10 11:19:19 [TRACE] Executing graph transform terraform.RemovedModuleTransformer 2020/11/10 11:19:19 [TRACE] Completed graph transform terraform.RemovedModuleTransformer (no changes) 2020/11/10 11:19:19 [TRACE] Executing graph transform terraform.AttachSchemaTransformer 2020/11/10 11:19:19 [TRACE] AttachSchemaTransformer: attaching provider config schema to provider["registry.terraform.io/josenk/esxi"] 2020/11/10 11:19:19 [TRACE] AttachSchemaTransformer: attaching resource schema to esxi_guest.vmxtest (expand) 2020/11/10 11:19:19 [TRACE] Completed graph transform terraform.AttachSchemaTransformer (no changes) 2020/11/10 11:19:19 [TRACE] Executing graph transform terraform.ModuleExpansionTransformer 2020/11/10 11:19:19 [TRACE] Completed graph transform terraform.ModuleExpansionTransformer (no changes) 2020/11/10 11:19:19 [TRACE] Executing graph transform terraform.ReferenceTransformer 2020/11/10 11:19:19 [DEBUG] ReferenceTransformer: "var.esxi_hostssl" references: [] 2020/11/10 11:19:19 [DEBUG] ReferenceTransformer: "var.esxi_username" references: [] 2020/11/10 11:19:19 [DEBUG] ReferenceTransformer: "var.esxi_password" references: [] 2020/11/10 11:19:19 [DEBUG] ReferenceTransformer: "provider[\"registry.terraform.io/josenk/esxi\"]" references: [var.esxi_hostname var.esxi_hostport var.esxi_hostssl var.esxi_password var.esxi_username] 2020/11/10 11:19:19 [DEBUG] ReferenceTransformer: "esxi_guest.vmxtest (expand)" references: [] 2020/11/10 11:19:19 [DEBUG] ReferenceTransformer: "output.ip_addr (expand)" references: [esxi_guest.vmxtest (expand)] 2020/11/10 11:19:19 [DEBUG] ReferenceTransformer: "var.esxi_hostname" references: [] 2020/11/10 11:19:19 [DEBUG] ReferenceTransformer: "var.esxi_hostport" references: [] 2020/11/10 11:19:19 [TRACE] Completed graph transform terraform.ReferenceTransformer with new graph: esxi_guest.vmxtest (expand) - terraform.nodeExpandPlannableResource provider["registry.terraform.io/josenk/esxi"] - terraform.NodeApplyableProvider output.ip_addr (expand) - terraform.nodeExpandOutput esxi_guest.vmxtest (expand) - terraform.nodeExpandPlannableResource provider["registry.terraform.io/josenk/esxi"] - terraform.NodeApplyableProvider var.esxi_hostname - terraform.NodeRootVariable var.esxi_hostport - terraform.NodeRootVariable var.esxi_hostssl - terraform.NodeRootVariable var.esxi_password - terraform.NodeRootVariable var.esxi_username - terraform.NodeRootVariable var.esxi_hostname - terraform.NodeRootVariable var.esxi_hostport - terraform.NodeRootVariable var.esxi_hostssl - terraform.NodeRootVariable var.esxi_password - terraform.NodeRootVariable var.esxi_username - *terraform.NodeRootVariable

2020/11/10 11:19:19 [TRACE] Executing graph transform terraform.attachDataResourceDependenciesTransformer 2020/11/10 11:19:19 [TRACE] Completed graph transform terraform.attachDataResourceDependenciesTransformer (no changes) 2020/11/10 11:19:19 [TRACE] Executing graph transform terraform.TargetsTransformer 2020/11/10 11:19:19 [TRACE] Completed graph transform terraform.TargetsTransformer (no changes) 2020/11/10 11:19:19 [TRACE] Executing graph transform terraform.ForcedCBDTransformer 2020/11/10 11:19:19 [TRACE] ForcedCBDTransformer: "esxi_guest.vmxtest (expand)" (terraform.nodeExpandPlannableResource) has no CBD descendent, so skipping 2020/11/10 11:19:19 [TRACE] Completed graph transform terraform.ForcedCBDTransformer (no changes) 2020/11/10 11:19:19 [TRACE] Executing graph transform terraform.CountBoundaryTransformer 2020/11/10 11:19:19 [TRACE] Completed graph transform terraform.CountBoundaryTransformer with new graph: esxi_guest.vmxtest (expand) - terraform.nodeExpandPlannableResource provider["registry.terraform.io/josenk/esxi"] - terraform.NodeApplyableProvider meta.count-boundary (EachMode fixup) - terraform.NodeCountBoundary esxi_guest.vmxtest (expand) - terraform.nodeExpandPlannableResource output.ip_addr (expand) - terraform.nodeExpandOutput provider["registry.terraform.io/josenk/esxi"] - terraform.NodeApplyableProvider var.esxi_hostname - terraform.NodeRootVariable var.esxi_hostport - terraform.NodeRootVariable var.esxi_hostssl - terraform.NodeRootVariable var.esxi_password - terraform.NodeRootVariable var.esxi_username - terraform.NodeRootVariable output.ip_addr (expand) - terraform.nodeExpandOutput esxi_guest.vmxtest (expand) - terraform.nodeExpandPlannableResource provider["registry.terraform.io/josenk/esxi"] - terraform.NodeApplyableProvider var.esxi_hostname - terraform.NodeRootVariable var.esxi_hostport - terraform.NodeRootVariable var.esxi_hostssl - terraform.NodeRootVariable var.esxi_password - terraform.NodeRootVariable var.esxi_username - terraform.NodeRootVariable var.esxi_hostname - terraform.NodeRootVariable var.esxi_hostport - terraform.NodeRootVariable var.esxi_hostssl - terraform.NodeRootVariable var.esxi_password - terraform.NodeRootVariable var.esxi_username - *terraform.NodeRootVariable

2020/11/10 11:19:19 [TRACE] Executing graph transform terraform.CloseProviderTransformer 2020/11/10 11:19:19 [TRACE] Completed graph transform terraform.CloseProviderTransformer with new graph: esxi_guest.vmxtest (expand) - terraform.nodeExpandPlannableResource provider["registry.terraform.io/josenk/esxi"] - terraform.NodeApplyableProvider meta.count-boundary (EachMode fixup) - terraform.NodeCountBoundary esxi_guest.vmxtest (expand) - terraform.nodeExpandPlannableResource output.ip_addr (expand) - terraform.nodeExpandOutput provider["registry.terraform.io/josenk/esxi"] - terraform.NodeApplyableProvider var.esxi_hostname - terraform.NodeRootVariable var.esxi_hostport - terraform.NodeRootVariable var.esxi_hostssl - terraform.NodeRootVariable var.esxi_password - terraform.NodeRootVariable var.esxi_username - terraform.NodeRootVariable output.ip_addr (expand) - terraform.nodeExpandOutput esxi_guest.vmxtest (expand) - terraform.nodeExpandPlannableResource provider["registry.terraform.io/josenk/esxi"] - terraform.NodeApplyableProvider var.esxi_hostname - terraform.NodeRootVariable var.esxi_hostport - terraform.NodeRootVariable var.esxi_hostssl - terraform.NodeRootVariable var.esxi_password - terraform.NodeRootVariable var.esxi_username - terraform.NodeRootVariable provider["registry.terraform.io/josenk/esxi"] (close) - terraform.graphNodeCloseProvider esxi_guest.vmxtest (expand) - terraform.nodeExpandPlannableResource provider["registry.terraform.io/josenk/esxi"] - terraform.NodeApplyableProvider var.esxi_hostname - terraform.NodeRootVariable var.esxi_hostport - terraform.NodeRootVariable var.esxi_hostssl - terraform.NodeRootVariable var.esxi_password - terraform.NodeRootVariable var.esxi_username - *terraform.NodeRootVariable

2020/11/10 11:19:19 [TRACE] Executing graph transform terraform.CloseProvisionerTransformer 2020/11/10 11:19:19 [TRACE] Completed graph transform terraform.CloseProvisionerTransformer (no changes) 2020/11/10 11:19:19 [TRACE] Executing graph transform terraform.CloseRootModuleTransformer 2020/11/10 11:19:19 [TRACE] Completed graph transform terraform.CloseRootModuleTransformer with new graph: esxi_guest.vmxtest (expand) - terraform.nodeExpandPlannableResource provider["registry.terraform.io/josenk/esxi"] - terraform.NodeApplyableProvider meta.count-boundary (EachMode fixup) - terraform.NodeCountBoundary esxi_guest.vmxtest (expand) - terraform.nodeExpandPlannableResource output.ip_addr (expand) - terraform.nodeExpandOutput provider["registry.terraform.io/josenk/esxi"] - terraform.NodeApplyableProvider var.esxi_hostname - terraform.NodeRootVariable var.esxi_hostport - terraform.NodeRootVariable var.esxi_hostssl - terraform.NodeRootVariable var.esxi_password - terraform.NodeRootVariable var.esxi_username - terraform.NodeRootVariable output.ip_addr (expand) - terraform.nodeExpandOutput esxi_guest.vmxtest (expand) - terraform.nodeExpandPlannableResource provider["registry.terraform.io/josenk/esxi"] - terraform.NodeApplyableProvider var.esxi_hostname - terraform.NodeRootVariable var.esxi_hostport - terraform.NodeRootVariable var.esxi_hostssl - terraform.NodeRootVariable var.esxi_password - terraform.NodeRootVariable var.esxi_username - terraform.NodeRootVariable provider["registry.terraform.io/josenk/esxi"] (close) - terraform.graphNodeCloseProvider esxi_guest.vmxtest (expand) - terraform.nodeExpandPlannableResource provider["registry.terraform.io/josenk/esxi"] - terraform.NodeApplyableProvider root - terraform.nodeCloseModule meta.count-boundary (EachMode fixup) - terraform.NodeCountBoundary provider["registry.terraform.io/josenk/esxi"] (close) - terraform.graphNodeCloseProvider var.esxi_hostname - terraform.NodeRootVariable var.esxi_hostport - terraform.NodeRootVariable var.esxi_hostssl - terraform.NodeRootVariable var.esxi_password - terraform.NodeRootVariable var.esxi_username - terraform.NodeRootVariable

2020/11/10 11:19:19 [TRACE] Executing graph transform terraform.TransitiveReductionTransformer 2020/11/10 11:19:19 [TRACE] Completed graph transform terraform.TransitiveReductionTransformer with new graph: esxi_guest.vmxtest (expand) - terraform.nodeExpandPlannableResource provider["registry.terraform.io/josenk/esxi"] - terraform.NodeApplyableProvider meta.count-boundary (EachMode fixup) - terraform.NodeCountBoundary output.ip_addr (expand) - terraform.nodeExpandOutput output.ip_addr (expand) - terraform.nodeExpandOutput esxi_guest.vmxtest (expand) - terraform.nodeExpandPlannableResource provider["registry.terraform.io/josenk/esxi"] - terraform.NodeApplyableProvider var.esxi_hostname - terraform.NodeRootVariable var.esxi_hostport - terraform.NodeRootVariable var.esxi_hostssl - terraform.NodeRootVariable var.esxi_password - terraform.NodeRootVariable var.esxi_username - terraform.NodeRootVariable provider["registry.terraform.io/josenk/esxi"] (close) - terraform.graphNodeCloseProvider esxi_guest.vmxtest (expand) - terraform.nodeExpandPlannableResource root - terraform.nodeCloseModule meta.count-boundary (EachMode fixup) - terraform.NodeCountBoundary provider["registry.terraform.io/josenk/esxi"] (close) - terraform.graphNodeCloseProvider var.esxi_hostname - terraform.NodeRootVariable var.esxi_hostport - terraform.NodeRootVariable var.esxi_hostssl - terraform.NodeRootVariable var.esxi_password - terraform.NodeRootVariable var.esxi_username - terraform.NodeRootVariable

2020/11/10 11:19:19 [DEBUG] Starting graph walk: walkPlan 2020/11/10 11:19:19 [TRACE] dag/walk: visiting "var.esxi_username" 2020/11/10 11:19:19 [TRACE] dag/walk: visiting "var.esxi_password" 2020/11/10 11:19:19 [TRACE] vertex "var.esxi_username": starting visit (terraform.NodeRootVariable) 2020/11/10 11:19:19 [TRACE] dag/walk: visiting "var.esxi_hostssl" 2020/11/10 11:19:19 [TRACE] vertex "var.esxi_username": evaluating 2020/11/10 11:19:19 [TRACE] vertex "var.esxi_password": starting visit (terraform.NodeRootVariable) 2020/11/10 11:19:19 [TRACE] dag/walk: visiting "var.esxi_hostname" 2020/11/10 11:19:19 [TRACE] vertex "var.esxi_password": evaluating 2020/11/10 11:19:19 [TRACE] [walkPlan] Entering eval tree: var.esxi_username 2020/11/10 11:19:19 [TRACE] dag/walk: visiting "var.esxi_hostport" 2020/11/10 11:19:19 [TRACE] [walkPlan] Entering eval tree: var.esxi_password 2020/11/10 11:19:19 [TRACE] eval: terraform.EvalSequence 2020/11/10 11:19:19 [TRACE] vertex "var.esxi_hostname": starting visit (terraform.NodeRootVariable) 2020/11/10 11:19:19 [TRACE] vertex "var.esxi_hostport": starting visit (terraform.NodeRootVariable) 2020/11/10 11:19:19 [TRACE] [walkPlan] Exiting eval tree: var.esxi_username 2020/11/10 11:19:19 [TRACE] vertex "var.esxi_hostport": evaluating 2020/11/10 11:19:19 [TRACE] vertex "var.esxi_hostname": evaluating 2020/11/10 11:19:19 [TRACE] [walkPlan] Entering eval tree: var.esxi_hostport 2020/11/10 11:19:19 [TRACE] vertex "var.esxi_username": visit complete 2020/11/10 11:19:19 [TRACE] eval: terraform.EvalSequence 2020/11/10 11:19:19 [TRACE] [walkPlan] Entering eval tree: var.esxi_hostname 2020/11/10 11:19:19 [TRACE] [walkPlan] Exiting eval tree: var.esxi_hostport 2020/11/10 11:19:19 [TRACE] eval: terraform.EvalSequence 2020/11/10 11:19:19 [TRACE] vertex "var.esxi_hostport": visit complete 2020/11/10 11:19:19 [TRACE] [walkPlan] Exiting eval tree: var.esxi_hostname 2020/11/10 11:19:19 [TRACE] vertex "var.esxi_hostname": visit complete 2020/11/10 11:19:19 [TRACE] vertex "var.esxi_hostssl": starting visit (terraform.NodeRootVariable) 2020/11/10 11:19:19 [TRACE] vertex "var.esxi_hostssl": evaluating 2020/11/10 11:19:19 [TRACE] eval: terraform.EvalSequence 2020/11/10 11:19:19 [TRACE] [walkPlan] Entering eval tree: var.esxi_hostssl 2020/11/10 11:19:19 [TRACE] [walkPlan] Exiting eval tree: var.esxi_password 2020/11/10 11:19:19 [TRACE] eval: terraform.EvalSequence 2020/11/10 11:19:19 [TRACE] vertex "var.esxi_password": visit complete 2020/11/10 11:19:19 [TRACE] [walkPlan] Exiting eval tree: var.esxi_hostssl 2020/11/10 11:19:19 [TRACE] vertex "var.esxi_hostssl": visit complete 2020/11/10 11:19:19 [TRACE] dag/walk: visiting "provider[\"registry.terraform.io/josenk/esxi\"]" 2020/11/10 11:19:19 [TRACE] vertex "provider[\"registry.terraform.io/josenk/esxi\"]": starting visit (terraform.NodeApplyableProvider) 2020/11/10 11:19:19 [TRACE] vertex "provider[\"registry.terraform.io/josenk/esxi\"]": evaluating 2020/11/10 11:19:19 [TRACE] [walkPlan] Entering eval tree: provider["registry.terraform.io/josenk/esxi"] 2020/11/10 11:19:19 [TRACE] eval: terraform.EvalSequence 2020/11/10 11:19:19 [TRACE] eval: terraform.EvalInitProvider 2020-11-10T11:19:19.610+0100 [INFO] plugin: configuring client automatic mTLS 2020-11-10T11:19:19.634+0100 [DEBUG] plugin: starting plugin: path=.terraform/plugins/registry.terraform.io/josenk/esxi/1.8.1/darwin_amd64/terraform-provider-esxi_v1.8.1 args=[.terraform/plugins/registry.terraform.io/josenk/esxi/1.8.1/darwin_amd64/terraform-provider-esxi_v1.8.1] 2020-11-10T11:19:19.637+0100 [DEBUG] plugin: plugin started: path=.terraform/plugins/registry.terraform.io/josenk/esxi/1.8.1/darwin_amd64/terraform-provider-esxi_v1.8.1 pid=82229 2020-11-10T11:19:19.637+0100 [DEBUG] plugin: waiting for RPC address: path=.terraform/plugins/registry.terraform.io/josenk/esxi/1.8.1/darwin_amd64/terraform-provider-esxi_v1.8.1 2020-11-10T11:19:19.647+0100 [INFO] plugin.terraform-provider-esxi_v1.8.1: configuring server automatic mTLS: timestamp=2020-11-10T11:19:19.647+0100 2020-11-10T11:19:19.674+0100 [DEBUG] plugin: using plugin: version=5 2020-11-10T11:19:19.674+0100 [DEBUG] plugin.terraform-provider-esxi_v1.8.1: plugin address: address=/var/folders/vb/1hzy2mm57cb4j8f0rj4x5wvh0000gn/T/plugin047376712 network=unix timestamp=2020-11-10T11:19:19.674+0100 2020-11-10T11:19:19.730+0100 [TRACE] plugin.stdio: waiting for stdio data 2020/11/10 11:19:19 [TRACE] BuiltinEvalContext: Initialized "provider[\"registry.terraform.io/josenk/esxi\"]" provider for provider["registry.terraform.io/josenk/esxi"] 2020/11/10 11:19:19 [TRACE] eval: terraform.EvalNoop 2020/11/10 11:19:19 [TRACE] eval: terraform.EvalOpFilter 2020/11/10 11:19:19 [TRACE] eval: terraform.EvalSequence 2020/11/10 11:19:19 [TRACE] eval: terraform.EvalGetProvider 2020/11/10 11:19:19 [TRACE] eval: terraform.EvalOpFilter 2020/11/10 11:19:19 [TRACE] eval: terraform.EvalSequence 2020/11/10 11:19:19 [TRACE] eval: terraform.EvalConfigProvider 2020/11/10 11:19:19 [TRACE] buildProviderConfig for provider["registry.terraform.io/josenk/esxi"]: using explicit config only 2020/11/10 11:19:19 [TRACE] GRPCProvider: GetSchema 2020-11-10T11:19:19.730+0100 [WARN] plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unimplemented desc = unknown service plugin.GRPCStdio" 2020/11/10 11:19:19 [TRACE] No provider meta schema returned 2020/11/10 11:19:19 [TRACE] GRPCProvider: Configure 2020-11-10T11:19:19.731+0100 [DEBUG] plugin.terraform-provider-esxi_v1.8.1: pid-82229-config.go:18: [validateEsxiCreds] 2020-11-10T11:19:19.731+0100 [DEBUG] plugin.terraform-provider-esxi_v1.8.1: pid-82229-esxi_remote_cmds.go:61: [runRemoteSshCommand] :Connectivity test, get vmware version 2020-11-10T11:19:19.752+0100 [DEBUG] plugin.terraform-provider-esxi_v1.8.1: pid-82229-esxi_remote_cmds.go:41: [runRemoteSshCommand] Retry connection: 3 2020-11-10T11:19:20.778+0100 [DEBUG] plugin.terraform-provider-esxi_v1.8.1: pid-82229-esxi_remote_cmds.go:41: [runRemoteSshCommand] Retry connection: 2 2020-11-10T11:19:21.819+0100 [DEBUG] plugin.terraform-provider-esxi_v1.8.1: pid-82229-esxi_remote_cmds.go:41: [runRemoteSshCommand] Retry connection: 1 2020-11-10T11:19:22.820+0100 [DEBUG] plugin.terraform-provider-esxi_v1.8.1: pid-82229-esxi_remote_cmds.go:72: [runRemoteSshCommand] Failed err: Client Connection Error 2020/11/10 11:19:22 [ERROR] eval: terraform.EvalConfigProvider, err: Failed to connect to esxi host: Client Connection Error 2020/11/10 11:19:22 [ERROR] eval: terraform.EvalSequence, err: Failed to connect to esxi host: Client Connection Error 2020/11/10 11:19:22 [ERROR] eval: terraform.EvalOpFilter, err: Failed to connect to esxi host: Client Connection Error 2020/11/10 11:19:22 [ERROR] eval: *terraform.EvalSequence, err: Failed to connect to esxi host: Client Connection Error 2020/11/10 11:19:22 [TRACE] [walkPlan] Exiting eval tree: provider["registry.terraform.io/josenk/esxi"] 2020/11/10 11:19:22 [TRACE] vertex "provider[\"registry.terraform.io/josenk/esxi\"]": visit complete 2020/11/10 11:19:22 [TRACE] dag/walk: upstream of "esxi_guest.vmxtest (expand)" errored, so skipping 2020/11/10 11:19:22 [TRACE] dag/walk: upstream of "output.ip_addr (expand)" errored, so skipping 2020/11/10 11:19:22 [TRACE] dag/walk: upstream of "provider[\"registry.terraform.io/josenk/esxi\"] (close)" errored, so skipping 2020/11/10 11:19:22 [TRACE] dag/walk: upstream of "meta.count-boundary (EachMode fixup)" errored, so skipping 2020/11/10 11:19:22 [TRACE] dag/walk: upstream of "root" errored, so skipping 2020/11/10 11:19:22 [INFO] backend/local: plan operation completed  2020/11/10 11:19:22 [TRACE] statemgr.Filesystem: removing lock metadata file .terraform.tfstate.lock.info Error: Failed to connect to esxi host: Client Connection Error 

 on main.tf line 1, in provider "esxi": 1: provider "esxi" {   2020/11/10 11:19:22 [TRACE] statemgr.Filesystem: unlocking terraform.tfstate using fcntl flock 2020-11-10T11:19:22.826+0100 [DEBUG] plugin: plugin process exited: path=.terraform/plugins/registry.terraform.io/josenk/esxi/1.8.1/darwin_amd64/terraform-provider-esxi_v1.8.1 pid=82229 2020-11-10T11:19:22.826+0100 [DEBUG] plugin: plugin exited

josenk commented 3 years ago

The process to login to esxi has not fundamentally changed. There have been fixes to encoding the password, and changing the retries, but otherwise it's the same. I believe there's still an issue if your password contains a space.

The biggest issue getting the initial connection working is making sure you have the correct password and make sure the account is not locked on your esxi host. Verify that you can ssh from command line to the host using that password.

It's very easy to lock the account. Edit ssd_config to make it less restrictive.

If the password contains really special characters, try changing the password to something more simple.

Does terrafom 0.11 still work?

vukzeka commented 3 years ago

Hi @josenk Thanks for reaching out.

Yes, I can connect to esxi host using ssh and all works as expected.

I just tested by downgrading terraform to 0.11 and also to use older plugin version 1.5.3 but still same issue. It looks like something else is causing this problem as I remember all was fine with 1.5.3

root access is not locked as I can connect to esxi host without any issues. Also just checked sshd_config but not seeing anything wrong there.

Do you maybe have any idea what to check further as logs are not providing any clue?

josenk commented 3 years ago

From your original post.

2020-11-10T11:25:48.446+0100 [DEBUG] plugin.terraform-provider-esxi_v1.8.1: pid-82328-esxi_remote_cmds.go:61: [runRemoteSshCommand] :Connectivity test, get vmware version
2020-11-10T11:25:48.488+0100 [DEBUG] plugin.terraform-provider-esxi_v1.8.1: pid-82328-esxi_remote_cmds.go:41: [runRemoteSshCommand] Retry connection: 3
2020-11-10T11:25:49.511+0100 [DEBUG] plugin.terraform-provider-esxi_v1.8.1: pid-82328-esxi_remote_cmds.go:41: [runRemoteSshCommand] Retry connection: 2
2020-11-10T11:25:50.536+0100 [DEBUG] plugin.terraform-provider-esxi_v1.8.1: pid-82328-esxi_remote_cmds.go:41: [runRemoteSshCommand] Retry connection: 1
2020-11-10T11:25:51.539+0100 [DEBUG] plugin.terraform-provider-esxi_v1.8.1: pid-82328-esxi_remote_cmds.go:72: [runRemoteSshCommand] Failed err: Client Connection Error
2020/11/10 11:25:51 [ERROR] eval: *terraform.EvalConfigProvider, err: Failed to connect to esxi host: Client Connection Error

These logs come from this code.

https://github.com/josenk/terraform-provider-esxi/blob/86264ae1473e088c6a3275d9ad21859a72e7e44e/esxi/config.go#L23-L27 https://github.com/josenk/terraform-provider-esxi/blob/86264ae1473e088c6a3275d9ad21859a72e7e44e/esxi/esxi_remote_cmds.go#L70-L74

So, basically.... Client Connection Error comes directly from the connectToHost go function. I pass that error directly to the output. Unfortunately, It's not very descriptive.

Since you have tried multiple versions of Terraform and multiple versions of the plugin and you have the same results, I don't believe there's any issue with the plugin... You have some type of connectivity or access problem.

My suggestions are:

vukzeka commented 3 years ago

Hi @josenk I already checked both auth.log and shell.log and there was nothing so it looked like request never reach server...

After some additional debugging today I finally found root cause esxi host had following in sshd_config PasswordAuthentication no

ssh was working correctly as it has also UsePAM yes but it was failing when called from terraform/go(ssh.Dial). https://github.com/josenk/terraform-provider-esxi/blob/86264ae1473e088c6a3275d9ad21859a72e7e44e/esxi/esxi_remote_cmds.go#L39

Probably make sense you mention same in readme as precondition, so that no one else experience this same issue in future.

Thanks for all your support and just continue great work.

Best, Zeljko

TallFurryMan commented 2 years ago

I'd also like to add that the ESXi FIPS module has a very specific set of SSH keys, which may lead to negotiation failure early in the process.

From /var/log/authd.log on ESXi 7.0u3, with matching symptoms from client side:

2021-11-15T10:07:42.389Z sshd[2310795]: FIPS mode initialized                                           
2021-11-15T10:07:42.389Z sshd[2310795]: Connection from a.b.c.d port xxx                            
2021-11-15T10:07:42.390Z sshd[2310793]: Unable to negotiate with a.b.c.d port xxx: no matching host key type found. Their offer: ssh-ed25519 [preauth]