Provide a way to connect components between process groups

[ennui93]

Background

With nifi-config, it is possible to deploy and configure flows in NiFi. However, at present, nifi-config does not provide any means of connecting disjoint flows together after they are deployed and configured.

We use a design pattern in NiFi consisting of many 'front-end' flows feeding into one common 'back-end' flow. Both of these flows can be deployed and configured by nifi-config; but connecting them together requires manual work, which is both tedious and subject to misconfiguration.

Proposed Solution

Provide a means of connecting any two connectable components (i.e. processors, ports, and tunnels). Users could access this functionality by calling a new mode on the command-line: routeConnection. This mode will take two parameters to determine where the source and destination of the series of connections. Connections between process groups will have ports created automatically.

java -jar nifi-deploy-config.jar \
  -nifi http://127.0.0.1:8079/nifi-api \
  -sourceComponent "root > front_end > front_end_1 > outputPort" \
  -destinationComponent "root > back_end > back_end_1 > inputPort" \
  -m routeConnection

[hermannpencole]

Ok, i can make it. I am to busy this week, i can look for that next week.

[ennui93]

Thanks @hermannpencole. I've been developing a solution within nifi-config this week. It should be finished next week. I can send a pull request then to see if it's something you'd be willing to accept into nifi-config then. Alternatively, I can push what I have now into my fork of nifi-config if you'd like to see the solution in its current state; but as you're busy this week, I'm guessing not. 😃

Note that I decided to change the UX slightly so the command will look more like the following:

java -jar nifi-deploy-config.jar \
  -nifi http://127.0.0.1:8079/nifi-api \
  -sourceBranch "root > front_end > front_end_1" \
  -destinationBranch "root > back_end > back_end_1" \
  -destinationPort "back_end_1-standard_input" \
  -m routeConnection

[hermannpencole]

Thanks for your participation. I will draw my inspiration from your solution. To keep the way of nifi-config, i prefer have the command

java -jar nifi-deploy-config-1.1.3.jar \
  -nifi http://ip-nifi-prod:8080/nifi-api \
  -branch "root > front_end > front_end_1" \
  -conf /tmp/config.json \
  -mode updateConfig

And have in config.json

{  
  "connections": [
       {
           "source":"root > front_end > front_end_1",
            "detsination":"root > front_end > back_end_1"
        }
    ]
}

In my example -branch is not used. But if you want, you can configure back_end_1 and after configure front_end_1 with the connections configuration in the json file.

[ennui93]

I created pull request #22, which will create connections in the updateConfig mode based upon connections specified in a JSON config.

Note that I elected to add one more KVP to the connection object called name, which determines the name of ports to be used along a given route of connections. This is necessary for our use case, which often has more than one connection running from a given set of output port in a front-end to a corresponding set of input ports in its corresponding back-end.

This has been tested against NiFi 1.0 & 1.3.

[ennui93]

22 has been merged, enabling this functionality. Thank you!

[mattyait]

@ennui93 I see you merged the latest changes of creating connection through updateConfig using the connection from json. I'm trying to use that but while executing i'm getting null pointer exception for both case if I enter connections block in json or not. Please can you check it and verify.

[https://github.com/hermannpencole/nifi-config/issues/28]()