Unable to connect to multiple NiFi's

[jamesduffy]

• Nipyapi version: 0.10.3
• NiFi version: 1.7.1
• NiFi-Registry version: N/A
• Python version: 2.7.15
• Operating System: OSX 10.14

Description

I want to connect to different NiFi servers within the same script. I currently have a loop, but when looking at the data returned it is data from the first set server and not the other server. I have tried reloading the module each time. It would probably be easier to initialize a class instead of configuring a module. I would prefer to do something like the following:

import nipyapi

consumer_nifi = nipyapi.Server(host='consumer.nifi.int.example.com:80/nifi-api')
s3_nifi = nipyapi.Server(host='s3.nifi.int.example.com:80/nifi-api')

print(consumer_nifi.system.get_system_diagnostics())
print(s3_nifi.system.get_system_diagnostics())

What I Did

import nipyapi
import os

from pprint import pprint

nifi_clusters = ['consumer', 'cluster', 's3']
status_metrics = [
    'flow_files_in',
    'flow_files_out',
    'flow_files_queued',
    'flow_files_received',
    'flow_files_sent',
    'flow_files_transferred',
]

def clean_nifi_name(name):
    """Lowercase and remove spaces in NiFi names."""
    return name.lower().replace(' ', '_').replace('_-_', '_')

def lambda_handler(event, context):
    """Lambda handler."""
    env = os.environ['ENVIRONMENT']

    metrics_list = []

    for cluster in nifi_clusters:
        print("Getting Metrics for {}".format(cluster))
        nipyapi.config.nifi_config.host = '{}.nifi.int.example.com:80/nifi-api'.format(cluster)

        flows = nipyapi.canvas.recurse_flow().to_dict()['process_group_flow']['flow']
        process_groups = flows['process_groups']

        for process_group in process_groups:
            name = clean_nifi_name(process_group['component']['name'])

            print(name)

            for metric in status_metrics:
                point = process_group['status']['aggregate_snapshot'][metric]
                metrics_list.append({
                    'metric': "nifi.process_group.{}".format(metric),
                    'points': point,
                    'tags': [
                        "process_group:{}".format(name)
                    ],
                    'type': 'gauge'
                })

    pprint(metrics_list)

if __name__ == "__main__":
    os.environ["AWS_REGION"] = "us-west-1"
    lambda_handler(event={}, context={})

Urgency

We are unable to use this library because we need to connect to multiple NiFi server/clusters from within the same script.

[Chaffelson]

Hi @jamesduffy The architecture of the library is to connect to one environment at a time, but switch between environments as requirements dictate, which may be done within the same script.

For an example of how this works, you can look at the FDLC demo within the library, which creates two Docker environments (Dev & Prod) and procedurally creates and version controls flows between them. I suggest running the demo, and examining the script while you execute it to understand the usages of the various functions.

Let me know how you get on.

[rbramley]

Hi @jamesduffy - as @Chaffelson says the FDLC demo switches between environments e.g. nipyapi.utils.set_endpoint(prod_nifi_api_url) documented here

[Chaffelson]

No further comments in a month. Please reopen if you need more advice :)