es_kibana timed out

[clong]

Hey @christophetd - congrats on launching this! I'm attempting to bring it online, but I was just met with the following:

azurerm_virtual_machine.es_kibana (local-exec): fatal: [es-kibana_31cd]: FAILED! => {"changed": false, "elapsed": 601, "msg": "timed out waiting for ping module test success: Failed to connect to the host via ssh: hunter@52.183.127.129: Permission denied (publickey)."}

azurerm_virtual_machine.es_kibana (local-exec): PLAY RECAP *********************************************************************
azurerm_virtual_machine.es_kibana (local-exec): es-kibana_31cd             : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

Error: Error running command '/bin/bash -c 'source venv/bin/activate && ANSIBLE_HOST_KEY_CHECKING=false ansible-playbook elasticsearch-kibana.yml -v'': exit status 2. Output: Using /Users/clong/git-repos/Adaz/ansible/ansible.cfg as config file

PLAY [Configure Elasticsearch and Kibana] **************************************

TASK [elasticsearch-kibana : wait_for_connection] ******************************
fatal: [es-kibana_31cd]: FAILED! => {"changed": false, "elapsed": 601, "msg": "timed out waiting for ping module test success: Failed to connect to the host via ssh: hunter@52.183.127.129: Permission denied (publickey)."}

PLAY RECAP *********************************************************************
es-kibana_31cd             : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

Is there maybe a timeout value that needs to be adjusted here?

[clong]

Tried to apply again, and it looks like it got farther except that an entry must have gotten loaded into known_hosts :-/ Maybe Ansible can be configured to ignore host keys?

TASK [elasticsearch-kibana : wait_for_connection] ******************************
fatal: [es-kibana_31cd]: FAILED! => {"changed": false, "elapsed": 600, "msg": "timed out waiting for ping module test success: Failed to connect to the host via ssh: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@\r\n@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @\r\n@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@\r\nIT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!\r\nSomeone could be eavesdropping on you right now (man-in-the-middle attack)!\r\nIt is also possible that a host key has just been changed.\r\nThe fingerprint for the ECDSA key sent by the remote host is\nSHA256:0XQYKHPPuYRSa1R6VR5BF40BxX10sIRsTPjt1dupalM.\r\nPlease contact your system administrator.\r\nAdd correct host key in /Users/clong/.ssh/known_hosts to get rid of this message.\r\nOffending ECDSA key in /Users/clong/.ssh/known_hosts:347\r\nChallenge/response authentication is disabled to avoid man-in-the-middle attacks.\r\nhunter@52.183.127.129: Permission denied (publickey)."}

PLAY RECAP *********************************************************************
es-kibana_31cd             : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

[christophetd]

Hi Chris,

Thanks a lot for the report. Here are the relevant code bits FYI:

• https://github.com/christophetd/Adaz/blob/master/terraform/elasticsearch_kibana.tf#L62-L65
• https://github.com/christophetd/Adaz/blob/master/terraform/locals.tf#L4

Ansible is then supposed to pick up your SSH key (since it's in a default location) and use it to connect to the instance. Host key checking is already disabled, c.f. https://github.com/christophetd/Adaz/blob/master/terraform/elasticsearch_kibana.tf#L71

I'm guessing the second error is due to ANSIBLE_HOST_KEY_CHECKING not being supported on Mac (?)

Questions for you:

• Can you try setting host_key_checking = False in ansible.cfg under [defaults]?
• Do you have a SSH key at ~/.ssh/id_rsa.pub? The ~ is expanded by pathexpand which should work on Mac OS as well
• Could you destroy the lab, apply the following patch to increase Ansible verbosity, and try again?

diff --git a/terraform/elasticsearch_kibana.tf b/terraform/elasticsearch_kibana.tf
index b1fdac3..916d3f4 100644
--- a/terraform/elasticsearch_kibana.tf
+++ b/terraform/elasticsearch_kibana.tf
@@ -68,7 +68,7 @@ resource "azurerm_virtual_machine" "es_kibana" {
   provisioner "local-exec" {
     working_dir = "${path.root}/../ansible"
     # Note: ANSIBLE_HOST_KEY_CHECKING needs to be set like this because 'source venv/bin/activate' will reset the environment, hence it cannot be passed via an environment {} block
-    command = "/bin/bash -c 'source venv/bin/activate && ANSIBLE_HOST_KEY_CHECKING=false ansible-playbook elasticsearch-kibana.yml -v'"
+    command = "/bin/bash -c 'source venv/bin/activate && ANSIBLE_HOST_KEY_CHECKING=false ansible-playbook elasticsearch-kibana.yml -vvv'"
   }

   tags = {

Thanks!

[christophetd]

Closing for inactivity, feel free to reopen if you run into the same issue