Closed js-max closed 4 years ago
Yes the statefile format has changed for TF0.12. Earlier this week I rewrote it for TF0.12.23 and Gen2 VSIs. This should also work for Gen1 VSIs.
The script assumes that it is executed from the same folder as ansible, which also contains the statefile. It can be invoked from ansible adding -i terraform_hosts.py
ansible-inventory --graph -i terraform_hosts.py
#!/usr/bin/env python
# Terraform-Ansible dynamic inventory for IBM Cloud
# Copyright (c) 2020, IBM UK
# steve_strutt@uk.ibm.com
ti_version = '0.1'
#
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
# http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# Can be used alongside static inventory files in the same directory
#
# This inventory script expects to find Terraform tags of the form
# group: ans_group associated with each tf instance to define the
# host group membership for Ansible. Multiple group tags are allowed per host
#
# terraform_inv.ini file in the same directory as this script, points to the
# location of the terraform.tfstate file to be inventoried
# [TFSTATE]
# TFSTATE_FILE = /usr/share/terraform/ibm/Demoapp2x/terraform.tfstate
#
# Validate correct execution:
# With ini file './terraform.py'
# Successful execution returns groups with lists of hosts and _meta/hostvars with a detailed
# host listing.
# Validate successful operation with ansible:
# With - 'ansible-inventory -i inventory --list'
import json
import configparser
import os
from os import getenv
from collections import defaultdict
from argparse import ArgumentParser
def parse_params():
parser = ArgumentParser('IBM Cloud Terraform inventory')
parser.add_argument('--list', action='store_true', default=True, help='List Terraform hosts')
parser.add_argument('--tfstate', '-t', action='store', dest='tfstate', help='Terraform state file in current or specified directory (terraform.tfstate default)')
parser.add_argument('--version', '-v', action='store_true', help='Show version')
args = parser.parse_args()
# read location of terrafrom state file from ini if it exists
if not args.tfstate:
args.tfstate = "terraform.tfstate"
return args
def get_tfstate(filename):
return json.load(open(filename))
class TerraformInventory:
def __init__(self):
self.args = parse_params()
if self.args.version:
print(ti_version)
elif self.args.list:
print(self.list_all())
def list_all(self):
hosts_vars = {}
attributes = {}
groups = {}
inv_output = {}
group_hosts = defaultdict(list)
hosts = self.get_tf_instances()
if hosts is not None:
for host in hosts:
hosts_vars[host[0]] = host[1]
groups = host[2]
if groups is not None:
for group in groups:
group_hosts[group].append(host[0])
for group in group_hosts:
inv_output[group] = {'hosts': group_hosts[group]}
inv_output["_meta"] = {'hostvars': hosts_vars}
return json.dumps(inv_output, indent=2)
#return json.dumps({'all': {'hosts': hosts}, '_meta': {'hostvars': hosts_vars}}, indent=2)
def get_tf_instances(self):
tfstate = get_tfstate(self.args.tfstate)
for resource in tfstate['resources']:
if (resource['type'] == 'ibm_is_instance') & (resource['mode'] == 'managed'):
for instance in resource['instances']:
tf_attrib = instance['attributes']
name = tf_attrib['name']
group = []
attributes = {
'id': tf_attrib['id'],
'image': tf_attrib['image'],
#'metadata': tf_attrib['user_data'],
'region': tf_attrib['zone'],
'ram': tf_attrib['memory'],
'cpu': tf_attrib['vcpu'][0]['count'],
'ssh_keys': tf_attrib['keys'],
'private_ipv4': tf_attrib['primary_network_interface'][0]['primary_ipv4_address'],
'ansible_host': tf_attrib['primary_network_interface'][0]['primary_ipv4_address'],
'ansible_ssh_user': 'root',
'provider': 'provider.ibm',
'tags': tf_attrib['tags'],
}
#tag of form ans_group: xxxxxxx is used to define ansible host group
for value in list(attributes["tags"]):
try:
curprefix, rest = value.split(":", 1)
except ValueError:
continue
if curprefix != "ans_group" :
continue
group.append(rest)
yield name, attributes, group
else:
continue
if __name__ == '__main__':
TerraformInventory()
Thanks @stevestrutt , will test it during today
@stevestrutt faced some problems
my env (working with virtualenv and loaded before attempt):
$ ansible --version
ansible 2.9.7
...
python version = 3.7.7 (default, Mar 10 2020, 15:43:33) [Clang 11.0.0 (clang-1100.0.33.17)]
Copied terraform.tfstate
to be inside same dir as terraform_hosts.py
$ tree
.
├── terraform.tfstate
└── terraform_hosts.py
Exec ansible-inventory --graph -i terraform_hosts.py
within that dir:
$ ansible-inventory --graph -i terraform_hosts.py
[WARNING]: * Failed to parse /Users/tf/code/ansible/inventory/terraform_hosts.py with script plugin: problem running /Users/tf/code/ansible/inventory/terraform_hosts.py --list ([Errno 13] Permission denied:
'/Users/tf/code/ansible/inventory/terraform_hosts.py')
[WARNING]: * Failed to parse /Users/tf/code/ansible/inventory/terraform_hosts.py with ini plugin: /Users/tf/code/ansible/inventory/terraform_hosts.py:6: Expected key=value host variable assignment, got: 0.1
[WARNING]: Unable to parse /Users/tf/code/ansible/inventory/terraform_hosts.py as an inventory source
[WARNING]: No inventory was parsed, only implicit localhost is available
@all:
|--@ungrouped:
Changed permissions and make py executable:
$ chmod +x terraform_hosts.py
$ ansible-inventory --graph -i terraform_hosts.py
@all:
|--@ungrouped:
Simple exec py will generate JSON file with hosts properly identified:
$ ./terraform_hosts.py
{
"_meta": {
"hostvars": {
...
}
What am I missing here?
The script is not locating your terraform.tfstate file in the working directory. This is my working folder.
Steves-MBP-2:tf12 stevestrutt$ ls -al
-rw-r--r--@ 1 stevestrutt staff 91540 5 May 16:12 terraform.tfstate
-rwxr-xr-x 1 stevestrutt staff 5146 7 May 11:27 terraform_hosts.py
and ./terraform_hosts.py
on the command line returns
{
"_meta": {
"hostvars": {
"ssh-vpc-vpc-frontend-vsi-2": {
"ssh_keys": [
"r006-e62f05cb-5fe1-4b5a-968f-db6e63afbb60"
],
"ram": 4,
"private_ipv4": "172.16.2.13",
"ansible_host": "172.16.2.13",
"image": "r006-e0039ab2-fcc8-11e9-8a36-6ffb6501dd33",
"id": "0727_5276c180-dd89-4196-98d6-efa9cbd19f65",
"provider": "provider.ibm",
"cpu": 2,
"tags": [
"ans_group:frontend",
"ans_group:topend"
...
...
You can also use the --tfstate flag to explicity reference the state file, though this only works for running the script directly.
./terraform_hosts.py --tfstate /Users/stevestrutt/ansible/tf12/terraform.tfstate
@stevestrutt it's. When I run py script it extract info, correct info.
The problem here it's calling with 'ansible-inventory' apologies for misleading you with any wrong info
When executing ansible-playbook I added the -i ./terraform_hosts.py
flag. I am executing ansible-playbook or ansible-inventory from the same directory containing the script and statefile. This works for me:
ansible-inventory --list -i terraform_hosts.py
or
ansible-inventory --list -i ./terraform_hosts.py
@stevestrutt well this might be a typical case of it works on my laptop
Created a new instance (to test on a fresh new environment) and installed ansible there (virtualenv not used, copy/pasted py inv script plus terraform state: Still unable to parse
$ ansible-inventory --list -i terraform_hosts.py
{
"_meta": {
"hostvars": {}
},
"all": {
"children": [
"ungrouped"
]
}
}
$ ansible-inventory --list -i ./terraform_hosts.py
{
"_meta": {
"hostvars": {}
},
"all": {
"children": [
"ungrouped"
]
}
}
Again, executing py script it's possible to collect data:
$ ./terraform_hosts.py
{
"_meta": {
"hostvars": {
"osp-lb-01": {
...
Something it's missing here...
Are you using specific version of ansible/python? IBM resources need a special tag?
My observation is that the script is finding a valid TF0.12 state file, as there is no error. However it is not finding any valid ibm_is_instance resources in the state file. I saw the same result when it ran against a state file that only contained null_resource records.
The new version of the script is only looking for VPC Gen1/Gen2 ibm_is_instance resources and not to older IaaS Classic (Softlayer) ibm_compute_vm_instance
. If you are provisioning the older classic VSI's the script will not report anything. If this is the case the section of code that parses the compute_vm_instance needs inserting into this version.
Now we have example updated to work for v12 https://github.com/IBM-Cloud/terraform-provider-ibm/tree/master/examples/ibm-ansible-samples
Hi there,
As stated here Terraform-Ansible dynamic inventory for IBM Cloud
It's planned to update mentioned script to support TF v0.12+?