poc - integrate nifi-registry with github

[hannemariavister]

What is the issue?

NiFi registry can be integrated with NiFi for storing, retrieving, and upgrading versioned flows from Flow Registry. With the release of NiFi Registry 0.2.0, flow contents can now be stored under a Git directory using the new GitFlowPersistenceProvider. This could be interesting topic for us to investigate.

Tutorial link: https://community.cloudera.com/t5/Community-Articles/Storing-Apache-NiFi-Versioned-Flows-in-a-Git-Repository/ta-p/248713 Youtube-link https://www.youtube.com/watch?v=kK7eVppg9Aw https://leanjavaengineering.wordpress.com/2018/07/26/quick-tip-using-git-with-nifi-registry-in-docker/

Checklist (after created issue)

• [x] Added label(s)
• [ ] Added to project
• [ ] Added to milestone

[hannemariavister]

NiFi Registry uses a pluggable persistence provider to store the content of each versioned item. Each type of versioned item, such as a versioned flow or extension bundle, has its own persistence provider.

The flow persistence provider stores the content of the flows saved to the registry. NiFi Registry provides FileSystemFlowPersistenceProvider and GitFlowPersistenceProvider, configured in provider.xml. In our case, we try to implement GitFlowPersistenceProvider to the configuration.

What is completed:

• Created a public git repository where the NiFi templates will be stored. By using artifact-stanza we are able to clone the public git repository locally on Nomad.
• Mounted the file provider.xml (with git configuration) from outside the container (on Nomad) to a standalone nifi-registry container. This configuration let us use git for storing NiFi templates.

TODO:

• Nifi-registry use the folder /flow_storage for storing NiFi templates. We use volume to store data inside the Nifi-registry container to the host (Nomad). Since we are using GitFlowPersistenceProvider, the /flow_storage must be treated as a git repository. Which means that the file .git must exists inside the container (in folder /flow_storage), this is solved by using volume. The folder /flow_storage is created by the user nifi and Nomad hasn’t write access to the directory, which means that we cannot create volume from container to host. TODO: We must find out how to give Nomad write access for creating this volume.

[hannemariavister]

The code is here: https://github.com/hannemariavister/terraform-nomad-nifi/tree/issue_30_testing_nifi_nifireg_github (For easily testing the flow, we have placed the code for this poc in terraform-nomad-nifi-repo. But in the end, the changes affect only terraform-nomad-nifiregistry-repo)

[hannemariavister]

We managed to integrate git version control to NiFi Registry after changing the image to michalklempa/nifi-registry.

Documentation (git integration for this image): https://github.com/michalklempa/docker-nifi-registry#gitflowpersistenceprovider

Docker hub: https://hub.docker.com/r/michalklempa/nifi-registry