search

hashicorp / terraform-provider-azurerm

Terraform provider for Azure Resource Manager
https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs
Mozilla Public License 2.0
4.53k stars 4.61k forks source link

Remote-Exec SSH is Failing #616

Closed akamalov closed 6 years ago

akamalov commented 6 years ago

Environment:

Terraform v0.11.0
+ provider.azurerm v0.3.3

Greetings,

Trying to issue SSH to execute inline commands, but getting failures:

Error: Error applying plan:

1 error(s) occurred:

* azurerm_virtual_machine.bastion: 1 error(s) occurred:

* ssh: handshake failed: ssh: unable to authenticate, attempted methods [publickey none], no supported methods remain

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

From terraform debug log:

2017-12-11T11:56:08.259-0500 [DEBUG] plugin.terraform-provider-azurerm_v0.3.3_x4: 0
2017-12-11T11:56:08.259-0500 [DEBUG] plugin.terraform-provider-azurerm_v0.3.3_x4:
2017-12-11T11:56:08.259-0500 [DEBUG] plugin.terraform-provider-azurerm_v0.3.3_x4:
2017/12/11 11:56:08 [TRACE] root: eval: *terraform.EvalWriteState
2017/12/11 11:56:08 [TRACE] root: eval: *terraform.EvalApplyProvisioners
2017-12-11T11:56:08.270-0500 [DEBUG] plugin.terraform: file-provisioner (internal) 2017/12/11 11:56:08 connecting to TCP connection for SSH
2017-12-11T11:56:08.271-0500 [DEBUG] plugin.terraform: file-provisioner (internal) 2017/12/11 11:56:08 handshaking with SSH
2017-12-11T11:56:08.421-0500 [DEBUG] plugin.terraform: file-provisioner (internal) 2017/12/11 11:56:08 handshake error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
2017-12-11T11:56:08.421-0500 [DEBUG] plugin.terraform: file-provisioner (internal) 2017/12/11 11:56:08 Retryable error: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain
2017/12/11 11:56:09 [TRACE] dag/walk: vertex "provider.azurerm (close)", waiting for: "azurerm_virtual_machine.zookeeper[0]"
2017-12-11T11:56:09.432-0500 [DEBUG] plugin.terraform-provider-azurerm_v0.3.3_x4: 2017/12/11 11:56:09 [DEBUG] AzureRM Request:
2017-12-11T11:56:09.432-0500 [DEBUG] plugin.terraform-provider-azurerm_v0.3.3_x4: GET /subscriptions/330fb5ae-79cf-4305-b306-1e1dd3e09c7e/providers/Microsoft.Compute/locations/eastus/operations/XXXXX-XXXX-47de-XXXX-XXXXXXXXXXXX?api-version=2017-03-30 HTTP/1.1
2017-12-11T11:56:09.432-0500 [DEBUG] plugin.terraform-provider-azurerm_v0.3.3_x4: Host: management.azure.com
2017-12-11T11:56:09.432-0500 [DEBUG] plugin.terraform-provider-azurerm_v0.3.3_x4: User-Agent: HashiCorp-Terraform-v0.10.6

Here is the code snippet:

##################################################################################
# Prepare node using inline file upload
##################################################################################

  provisioner "file" {
    source      = "${var.ansible_dir_path}/${var.ansible_file}"
    destination = "~/${var.ansible_file}"
  }

  provisioner "remote-exec" {
    inline = [
      "git clone https://github.com/ansible/ansible.git ~/ansible"
    ]
    connection = {
                   type = "ssh"
                   user = "${var.admin_username}"
                   private_key = "${file("${var.ssh_private_key}")}"
                   timeout = "2m"
                   agent = "false"
                   #host = "${element(azurerm_public_ip.bastion.*.ip_address, count.index)}"
                   host = "${azurerm_public_ip.bastion_pip.id}"
      }
  }

  provisioner "remote-exec" {
    inline = [
      "ansible-playbook -i ~/ansible/contrib/inventory/azure_rm.py ~/${var.ansible_file}/ansible_kafka.yml -e 'ansible_user=XXXXXXX ansible_group=XXXXX kafka_user=XXXXX kafka_group=XXXXX zookeeper_user=XXXXX zookeeper_group=XXXXX'"
    ]
    connection = {
                   type = "ssh"
                   user = "${var.admin_username}"
                   #key_file = "${var.ssh_private_key}"
                   private_key = "${file("${var.ssh_private_key}")}"
                   timeout = "2m"
                   agent = "false"
                   #host = "${element(azurerm_public_ip.bastion.*.ip_address, count.index)}"
                   host = "${azurerm_public_ip.bastion_pip.id}"
      }
  }
}

Any pointers on what is it I am doing incorrectly or perhaps other way of doing it ?

akamalov commented 6 years ago

Further debug is showing the following:

2017-12-11T13:24:01.224-0500 [DEBUG] plugin.terraform: file-provisioner (internal) 2017/12/11 13:24:01 connecting to TCP connection for SSH
2017-12-11T13:24:01.224-0500 [DEBUG] plugin.terraform: file-provisioner (internal) 2017/12/11 13:24:01 [DEBUG] Connecting to bastion: XXXXXX-bastion.XXXXX.cloudapp.azure.com:22
2017/12/11 13:24:01 [TRACE] dag/walk: vertex "meta.count-boundary (count boundary fixup)", waiting for: "azurerm_virtual_machine.broker[1]"
2017-12-11T13:24:02.759-0500 [DEBUG] plugin.terraform: file-provisioner (internal) 2017/12/11 13:24:02 [DEBUG] Connecting via bastion (XXXXXX-bastion.XXXXX.cloudapp.azure.com:22) to host: :22
2017-12-11T13:24:02.816-0500 [DEBUG] plugin.terraform: file-provisioner (internal) 2017/12/11 13:24:02 connection error: ssh: rejected: administratively prohibited (open failed)
2017-12-11T13:24:02.816-0500 [DEBUG] plugin.terraform: file-provisioner (internal) 2017/12/11 13:24:02 Retryable error: ssh: rejected: administratively prohibited (open failed)
2017/12/11 13:24:04 [TRACE] dag/walk: vertex "provisioner.remote-exec (close)", waiting for: "azurerm_virtual_machine.bastion"
2017/12/11 13:24:04 [TRACE] dag/walk: vertex "provisioner.file (close)", waiting for: "azurerm_virtual_machine.bastion"
2017/12/11 13:24:04 [TRACE] dag/walk: vertex "provider.azurerm (close)", waiting for: "azurerm_virtual_machine.bastion"
2017/12/11 13:24:04 [TRACE] dag/walk: vertex "root", waiting for: "provisioner.file (close)"
2017-12-11T13:24:05.817-0500 [DEBUG] plugin.terraform: file-provisioner (internal) 2017/12/11 13:24:05 connecting to TCP connection for SSH
2017-12-11T13:24:05.817-0500 [DEBUG] plugin.terraform: file-provisioner (internal) 2017/12/11 13:24:05 [DEBUG] Connecting to bastion: XXXXXX-bastion.XXXXX.cloudapp.azure.com:22
2017-12-11T13:24:06.370-0500 [DEBUG] plugin.terraform: file-provisioner (internal) 2017/12/11 13:24:06 [DEBUG] Connecting via bastion (XXXXXX-bastion.XXXXX.cloudapp.azure.com:22) to host: :22
2017/12/11 13:24:06 [TRACE] dag/walk: vertex "meta.count-boundary (count boundary fixup)", waiting for: "azurerm_virtual_machine.broker[1]"
2017-12-11T13:24:06.602-0500 [DEBUG] plugin.terraform: file-provisioner (internal) 2017/12/11 13:24:06 connection error: ssh: rejected: administratively prohibited (open failed)
2017-12-11T13:24:06.602-0500 [DEBUG] plugin.terraform: file-provisioner (internal) 2017/12/11 13:24:06 Retryable error: ssh: rejected: administratively prohibited (open failed)
2017-12-11T13:24:09.603-0500 [DEBUG] plugin.terraform: file-provisioner (internal) 2017/12/11 13:24:09 connecting to TCP connection for SSH
2017-12-11T13:24:09.603-0500 [DEBUG] plugin.terraform: file-provisioner (internal) 2017/12/11 13:24:09 [DEBUG] Connecting to bastion: XXXXXX-bastion.XXXXX.cloudapp.azure.com:22
2017/12/11 13:24:09 [TRACE] dag/walk: vertex "provisioner.file (close)", waiting for: "azurerm_virtual_machine.bastion"
2017/12/11 13:24:09 [TRACE] dag/walk: vertex "provisioner.remote-exec (close)", waiting for: "azurerm_virtual_machine.bastion"
2017/12/11 13:24:09 [TRACE] dag/walk: vertex "provider.azurerm (close)", waiting for: "azurerm_virtual_machine.bastion"
2017/12/11 13:24:09 [TRACE] dag/walk: vertex "root", waiting for: "provisioner.file (close)"
2017-12-11T13:24:10.008-0500 [DEBUG] plugin.terraform: file-provisioner (internal) 2017/12/11 13:24:10 [DEBUG] Connecting via bastion (XXXXXX-bastion.XXXXX.cloudapp.azure.com:22) to host: :22
2017-12-11T13:24:10.234-0500 [DEBUG] plugin.terraform: file-provisioner (internal) 2017/12/11 13:24:10 connection error: ssh: rejected: administratively prohibited (open failed)
2017-12-11T13:24:10.235-0500 [DEBUG] plugin.terraform: file-provisioner (internal) 2017/12/11 13:24:10 Retryable error: ssh: rejected: administratively prohibited (open failed)
2017/12/11 13:24:11 [TRACE] dag/walk: vertex "meta.count-boundary (count boundary fixup)", waiting for: "azurerm_virtual_machine.broker[1]"
2017-12-11T13:24:13.235-0500 [DEBUG] plugin.terraform: file-provisioner (internal) 2017/12/11 13:24:13 connecting to TCP connection for SSH
2017-12-11T13:24:13.235-0500 [DEBUG] plugin.terraform: file-provisioner (internal) 2017/12/11 13:24:13 [DEBUG] Connecting to bastion: XXXXXX-bastion.XXXXX.cloudapp.azure.com:22
2017-12-11T13:24:13.460-0500 [DEBUG] plugin.terraform: file-provisioner (internal) 2017/12/11 13:24:13 [DEBUG] Connecting via bastion (XXXXXX-bastion.XXXXX.cloudapp.azure.com:22) to host: :22
2017-12-11T13:24:13.695-0500 [DEBUG] plugin.terraform: file-provisioner (internal) 2017/12/11 13:24:13 connection error: ssh: rejected: administratively prohibited (open failed)
2017-12-11T13:24:13.695-0500 [DEBUG] plugin.terraform: file-provisioner (internal) 2017/12/11 13:24:13 Retryable error: ssh: rejected: administratively prohibited (open failed)
2017/12/11 13:24:14 [TRACE] dag/walk: vertex "provisioner.remote-exec (close)", waiting for: "azurerm_virtual_machine.bastion"
2017/12/11 13:24:14 [TRACE] dag/walk: vertex "provisioner.file (close)", waiting for: "azurerm_virtual_machine.bastion"
2017/12/11 13:24:14 [TRACE] dag/walk: vertex "provider.azurerm (close)", waiting for: "azurerm_virtual_machine.bastion"

I found an existing ticket referencing the same type of an error (not sure if it is AzureRM or AWS): https://github.com/hashicorp/terraform/issues/13830

My SSH rules are open to bastion server. Not sure what is causing SSH rejected errors...

akamalov commented 6 years ago

Solved the problem. It looks like it has nothing to do with Terraform, but the actual OS image being used where SSHD is not configured to allow "AllowTcpForwarding". So, using packer I had to add the following line:

"sed -i -e \"s/^#    AllowTcpForwarding no/AllowTcpForwarding yes/g\" /etc/ssh/sshd_config"

...which configures SSH daemon to allow TCP forwarding and consequently used this image for the OS in Terraform.

ghost commented 4 years ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!