Fail to ssh into bastion/master with floating ip - Terraform - Openstack

[arkanmgerges]

Environment:

• Cloud provider or hardware configuration: Open stack - Remote

• OS (printf "$(uname -srm)\n$(cat /etc/os-release)\n"): --- Controller Node (Openstack) ------- Linux 4.15.0-106-generic x86_64 NAME="Ubuntu" VERSION="18.04.4 LTS (Bionic Beaver)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 18.04.4 LTS" VERSION_ID="18.04" HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=bionic UBUNTU_CODENAME=bionic

----- Deployer Machine ----- macOS Catalina version 10.15.5 MacBook Pro (15-inch, 2019) Processor 2,3 GHz 8-Core Intel Core i9 Memory 16 GB 2400 MHz DDR4 Graphics Radeon Pro 560X 4 GB Intel UHD Graphics 630 1536 MB

• Version of Ansible (ansible --version): --- Deployer Machine (mac) ----- ansible 2.9.7 config file = /Users/arkan/ansible.cfg configured module search path = ['/Users/arkan/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules'] ansible python module location = /usr/local/Cellar/ansible/2.9.7/libexec/lib/python3.8/site-packages/ansible executable location = /usr/local/bin/ansible python version = 3.8.2 (default, May 1 2020, 17:44:54) [Clang 11.0.3 (clang-1103.0.32.59)]

• Version of Python (python --version): -- Deployer machine (mac) --- Python 2.7.16

Kubespray version (commit) (git rev-parse --short HEAD): I did not reach this phase because terraform provisioning machines that can not be ssh using Ansbile

Network plugin used: I did not reach this phase because terraform provisioning machines that can not be ssh using Ansbile

Full inventory with variables (ansible -i inventory/sample/inventory.ini all -m debug -a "var=hostvars[inventory_hostname]"):

Command used to invoke ansible: ansible -i inventory/$CLUSTER/hosts -m ping all

Output of ansible run:

k8s-dev-bastion-1 | FAILED! => {
    "changed": false,
    "module_stderr": "Warning: Permanently added '192.168.40.216' (ECDSA) to the list of known hosts.\r\n/bin/sh: 1: python: not found\n",
    "module_stdout": "",
    "msg": "The module failed to execute correctly, you probably need to set the interpreter.\nSee stdout/stderr for the exact error",
    "rc": 127
}
k8s-dev-k8s-node-nf-1 | FAILED! => {
    "changed": false,
    "module_stderr": "Warning: Permanently added '172.30.0.69' (ECDSA) to the list of known hosts.\r\n/bin/sh: 1: python: not found\n",
    "module_stdout": "",
    "msg": "The module failed to execute correctly, you probably need to set the interpreter.\nSee stdout/stderr for the exact error",
    "rc": 127
}
k8s-dev-k8s-master-nf-1 | FAILED! => {
    "changed": false,
    "module_stderr": "Warning: Permanently added '172.30.2.53' (ECDSA) to the list of known hosts.\r\n/bin/sh: 1: python: not found\n",
    "module_stdout": "",
    "msg": "The module failed to execute correctly, you probably need to set the interpreter.\nSee stdout/stderr for the exact error",
    "rc": 127
}
k8s-dev-k8s-node-nf-2 | FAILED! => {
    "changed": false,
    "module_stderr": "Warning: Permanently added '172.30.1.238' (ECDSA) to the list of known hosts.\r\n/bin/sh: 1: python: not found\n",
    "module_stdout": "",
    "msg": "The module failed to execute correctly, you probably need to set the interpreter.\nSee stdout/stderr for the exact error",
    "rc": 127
}

Anything else do we need to know: This is the Terraform output https://gist.github.com/arkanmgerges/d6f0a30a08896e7c9f11d1e5a893e72d

Points that you need to know:

1. I can provision and ssh to machines without any problem from Horizon dashboard (volumes, floating ips, networking, vms ...etc) all are working
2. I can provision and ssh to machines without any problem from Command line using openstack client and all are working
3. I wrote terraform code in order to provision machines, and everything is working (floating ip, ssh to the machines, attaching volumes, ...etc)

The problem that I'm facing is the machines are created using your code, (Bastion machine, master node, and worker nodes) but when I run ansible I have errors

My one of the config is here: https://gist.github.com/arkanmgerges/80a810e67fdd3861bdc1fe1640b56ec8

[arkanmgerges]

Hi there, I solved the issue.

First I ran ansible -i hosts -m ping 'bastion' -vvvv Then I saw that the problem that my image "ubuntu 18.04" does not have python installed, so I ssh manually into bastion mashine. note

You can add into your ~/.ssh/config
IdentityFile ~/.ssh/id_rsa

Then exit the current shell and re-open it in order for the key to be loaded into your ssh-add agent (make sure that is running by running eval $(ssh-agent -s)) So when you run ssh-add -l you can see your path listed to the private key

Then ssh ubuntu@ sudo apt install -y python. <------------ here it did not work because it needs dns

So I added the following into cluster.tfvars: dns_nameservers = ["8.8.8.8", "8.8.4.4"] and run terraform apply -var-file=cluster.tfvars ../../contrib/terraform/openstack

Then I re-logged in into the bastion machine and installed python After that from my deployer machine (mac), I ran: ansible -i hosts -m ping 'bastion'

(base) Arkans-MBP:k8s-dev arkan$ ansible -i hosts -m ping 'bastion'
[DEPRECATION WARNING]: The TRANSFORM_INVALID_GROUP_CHARS settings is set to allow bad characters in group names by default, this will change, but still be user configurable on deprecation. This feature will be removed 
in version 2.10. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
[WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details
k8s-dev-bastion-1 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}