Connection time out after 5 successive SshRemoteCommands

[sinking0h]

Remote SSH connection hangs after 5 consecutive runRemoteSshCommand commands and times out after ~15 minutes retrying connection;

After connection retry, after another 5 consecutive runRemoteSshCommand it hangs again and times out after 15 minutes;

This continues until VM is created (guest VM creation takes ~3 hours because of the timeout wait times)

Modified MaxStartups, UsePAM, and other sshd config vars to no avail;

after adding breaking point and error printing before log.Printf("[runRemoteSshCommand] Retry connection: %d\n", attempt)

the below error is shown

[DEBUG] provider.terraform-provider-esxi_v1.10.2: pid-1754-esxi_remote_cmds.go:41: ssh: handshake failed: read tcp :->:: read: connection timed out [DEBUG] provider: plugin process exited: path=./esxi/1.10.2/linux_amd64/terraform-provider-esxi_v1.10.2 pid=1754 error="exit status 1"

if the breaking point is not added, VM creation takes ~3H otherwise it stops at the first timeout.

main.tf

terraform {
  required_version = ">= 0.13"
  required_providers {
    esxi = {
      source  = "terraform-example.com/local/esxi"
      version = "1.10.2"
    }
  }
}

provider "esxi" {
  esxi_hostname      = var.hostname
  esxi_hostport      = var.port
  esxi_hostssl       = var.portssl
  esxi_username      = var.username
  esxi_password      = var.password
}

resource "esxi_guest" "vmtest" {
  guest_name         = "vmtest"
  disk_store         = "data-store-home-1"
  memsize            = 128
  guestos    = "ubuntu-32"

  power              = "on"
  ovf_source        = "https://cloud-images.ubuntu.com/xenial/current/xenial-server-cloudimg-amd64.ova"

  network_interfaces {
    virtual_network = "VM Network"
  }

}

[josenk]

You most likely have a PAM setting that is throttling the ssh connections.

[oscar-b]

@sinking0h Did you find a solution for this? I'm seeing the same behavior

Update: It turned out to be our Unifi Dream Machine firewall dropping the connections, due to package inspection thinking this was a SSH scan attack.