IBM-Cloud / terraform-provider-ibm

https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs
Mozilla Public License 2.0
339 stars 662 forks source link

Ansible "dynamic" inventory with TF v0.12+ #1410

Closed js-max closed 4 years ago

js-max commented 4 years ago

Hi there,

As stated here Terraform-Ansible dynamic inventory for IBM Cloud

This script was written for Terraform 0.11.07. It will break if Hashicorp change the format of the terraform.tf file. This is expected with 0.12.0.

It's planned to update mentioned script to support TF v0.12+?

stevestrutt commented 4 years ago

Yes the statefile format has changed for TF0.12. Earlier this week I rewrote it for TF0.12.23 and Gen2 VSIs. This should also work for Gen1 VSIs.

The script assumes that it is executed from the same folder as ansible, which also contains the statefile. It can be invoked from ansible adding -i terraform_hosts.py
ansible-inventory --graph -i terraform_hosts.py

#!/usr/bin/env python

# Terraform-Ansible dynamic inventory for IBM Cloud
# Copyright (c) 2020, IBM UK
# steve_strutt@uk.ibm.com
ti_version = '0.1'
#

#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#     http://www.apache.org/licenses/LICENSE-2.0
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# Can be used alongside static inventory files in the same directory 
#
# This inventory script expects to find Terraform tags of the form 
# group: ans_group associated with each tf instance to define the 
# host group membership for Ansible. Multiple group tags are allowed per host
#   
# terraform_inv.ini file in the same directory as this script, points to the 
# location of the terraform.tfstate file to be inventoried
# [TFSTATE]
# TFSTATE_FILE = /usr/share/terraform/ibm/Demoapp2x/terraform.tfstate
# 
# Validate correct execution: 
#   With ini file './terraform.py'
# Successful execution returns groups with lists of hosts and _meta/hostvars with a detailed
# host listing.
# Validate successful operation with ansible:
#   With - 'ansible-inventory -i inventory --list'

import json
import configparser
import os
from os import getenv
from collections import defaultdict
from argparse import ArgumentParser

def parse_params():
    parser = ArgumentParser('IBM Cloud Terraform inventory')
    parser.add_argument('--list', action='store_true', default=True, help='List Terraform hosts')
    parser.add_argument('--tfstate', '-t', action='store', dest='tfstate', help='Terraform state file in current or specified directory (terraform.tfstate default)')
    parser.add_argument('--version', '-v', action='store_true', help='Show version')
    args = parser.parse_args()
    # read location of terrafrom state file from ini if it exists 
    if not args.tfstate:
        args.tfstate = "terraform.tfstate"
    return args

def get_tfstate(filename):
    return json.load(open(filename))

class TerraformInventory:
    def __init__(self):
        self.args = parse_params()
        if self.args.version:
            print(ti_version)
        elif self.args.list:
            print(self.list_all())

    def list_all(self):
        hosts_vars = {}
        attributes = {}
        groups = {}
        inv_output = {}
        group_hosts = defaultdict(list)
        hosts = self.get_tf_instances()
        if hosts is not None: 
            for host in hosts:
                hosts_vars[host[0]] = host[1]
                groups = host[2]
                if groups is not None: 
                    for group in groups:
                        group_hosts[group].append(host[0])

        for group in group_hosts:
            inv_output[group] = {'hosts': group_hosts[group]}
        inv_output["_meta"] = {'hostvars': hosts_vars} 
        return json.dumps(inv_output, indent=2)    
        #return json.dumps({'all': {'hosts': hosts}, '_meta': {'hostvars': hosts_vars}}, indent=2)

    def get_tf_instances(self):
        tfstate = get_tfstate(self.args.tfstate)
        for resource in tfstate['resources']:

            if (resource['type'] == 'ibm_is_instance') & (resource['mode'] == 'managed'):
                for instance in resource['instances']:
                    tf_attrib = instance['attributes']
                    name = tf_attrib['name']
                    group = []

                    attributes = {
                        'id': tf_attrib['id'],
                        'image': tf_attrib['image'],
                        #'metadata': tf_attrib['user_data'],
                        'region': tf_attrib['zone'],
                        'ram': tf_attrib['memory'],
                        'cpu': tf_attrib['vcpu'][0]['count'],
                        'ssh_keys': tf_attrib['keys'],
                        'private_ipv4': tf_attrib['primary_network_interface'][0]['primary_ipv4_address'],
                        'ansible_host': tf_attrib['primary_network_interface'][0]['primary_ipv4_address'],
                        'ansible_ssh_user': 'root',
                        'provider': 'provider.ibm',
                        'tags': tf_attrib['tags'],
                    }

                    #tag of form ans_group: xxxxxxx is used to define ansible host group
                    for value in list(attributes["tags"]):
                        try:
                            curprefix, rest = value.split(":", 1)
                        except ValueError:
                            continue
                        if curprefix != "ans_group" :
                            continue  
                        group.append(rest)

                    yield name, attributes, group

            else:    
                continue        

if __name__ == '__main__':
    TerraformInventory()
js-max commented 4 years ago

Thanks @stevestrutt , will test it during today

js-max commented 4 years ago

@stevestrutt faced some problems

my env (working with virtualenv and loaded before attempt):

$ ansible --version                                        
ansible 2.9.7
  ...
  python version = 3.7.7 (default, Mar 10 2020, 15:43:33) [Clang 11.0.0 (clang-1100.0.33.17)]

Copied terraform.tfstate to be inside same dir as terraform_hosts.py

$ tree   
.
├── terraform.tfstate
└── terraform_hosts.py

Exec ansible-inventory --graph -i terraform_hosts.py within that dir:

$ ansible-inventory --graph -i terraform_hosts.py          
[WARNING]:  * Failed to parse /Users/tf/code/ansible/inventory/terraform_hosts.py with script plugin: problem running /Users/tf/code/ansible/inventory/terraform_hosts.py --list ([Errno 13] Permission denied:
'/Users/tf/code/ansible/inventory/terraform_hosts.py')
[WARNING]:  * Failed to parse /Users/tf/code/ansible/inventory/terraform_hosts.py with ini plugin: /Users/tf/code/ansible/inventory/terraform_hosts.py:6: Expected key=value host variable assignment, got: 0.1
[WARNING]: Unable to parse /Users/tf/code/ansible/inventory/terraform_hosts.py as an inventory source
[WARNING]: No inventory was parsed, only implicit localhost is available
@all:
  |--@ungrouped:

Changed permissions and make py executable:

$ chmod +x terraform_hosts.py 

$ ansible-inventory --graph -i terraform_hosts.py
@all:
  |--@ungrouped:

Simple exec py will generate JSON file with hosts properly identified:

$ ./terraform_hosts.py
{
  "_meta": {
    "hostvars": {
...
}

What am I missing here?

stevestrutt commented 4 years ago

The script is not locating your terraform.tfstate file in the working directory. This is my working folder.

Steves-MBP-2:tf12 stevestrutt$ ls -al
-rw-r--r--@  1 stevestrutt  staff  91540  5 May 16:12 terraform.tfstate
-rwxr-xr-x   1 stevestrutt  staff   5146  7 May 11:27 terraform_hosts.py

and ./terraform_hosts.py on the command line returns

{
  "_meta": {
    "hostvars": {
      "ssh-vpc-vpc-frontend-vsi-2": {
        "ssh_keys": [
          "r006-e62f05cb-5fe1-4b5a-968f-db6e63afbb60"
        ],
        "ram": 4,
        "private_ipv4": "172.16.2.13",
        "ansible_host": "172.16.2.13",
        "image": "r006-e0039ab2-fcc8-11e9-8a36-6ffb6501dd33",
        "id": "0727_5276c180-dd89-4196-98d6-efa9cbd19f65",
        "provider": "provider.ibm",
        "cpu": 2,
        "tags": [
          "ans_group:frontend",
          "ans_group:topend"
...

...

You can also use the --tfstate flag to explicity reference the state file, though this only works for running the script directly.

./terraform_hosts.py --tfstate /Users/stevestrutt/ansible/tf12/terraform.tfstate

js-max commented 4 years ago

@stevestrutt it's. When I run py script it extract info, correct info.

The problem here it's calling with 'ansible-inventory' apologies for misleading you with any wrong info

stevestrutt commented 4 years ago

When executing ansible-playbook I added the -i ./terraform_hosts.py flag. I am executing ansible-playbook or ansible-inventory from the same directory containing the script and statefile. This works for me: ansible-inventory --list -i terraform_hosts.py or ansible-inventory --list -i ./terraform_hosts.py

js-max commented 4 years ago

@stevestrutt well this might be a typical case of it works on my laptop

Created a new instance (to test on a fresh new environment) and installed ansible there (virtualenv not used, copy/pasted py inv script plus terraform state: Still unable to parse

$ ansible-inventory --list -i terraform_hosts.py 
{
    "_meta": {
        "hostvars": {}
    },
    "all": {
        "children": [
            "ungrouped"
        ]
    }
}

$ ansible-inventory --list -i ./terraform_hosts.py 
{
    "_meta": {
        "hostvars": {}
    },
    "all": {
        "children": [
            "ungrouped"
        ]
    }
}

Again, executing py script it's possible to collect data:

$ ./terraform_hosts.py 
{
  "_meta": {
    "hostvars": {
      "osp-lb-01": {
...

Something it's missing here...

Are you using specific version of ansible/python? IBM resources need a special tag?

stevestrutt commented 4 years ago

My observation is that the script is finding a valid TF0.12 state file, as there is no error. However it is not finding any valid ibm_is_instance resources in the state file. I saw the same result when it ran against a state file that only contained null_resource records.

The new version of the script is only looking for VPC Gen1/Gen2 ibm_is_instance resources and not to older IaaS Classic (Softlayer) ibm_compute_vm_instance. If you are provisioning the older classic VSI's the script will not report anything. If this is the case the section of code that parses the compute_vm_instance needs inserting into this version.

hkantare commented 4 years ago

Now we have example updated to work for v12 https://github.com/IBM-Cloud/terraform-provider-ibm/tree/master/examples/ibm-ansible-samples