search

guimou / nifi-on-openshift

Apache NiFi deployment on OpenShift
MIT License
12 stars 14 forks source link

readme

Apache NiFi on OpenShift

NiFi on OpenShift

Apache NiFi supports powerful and scalable directed graphs of data routing, transformation, and system mediation logic. Ref: https://nifi.apache.org/

This repo provides tools and instructions for deploying and using Apache Nifi on OpenShift, with the following characteristics:

Architecture

Several components are deployed to cover as many use cases as possible:

When deployed, a secured Route is automatically created, with reencryption using the certificate emitted by NiFi-CA. Put otherwise, it's a no touch setup, everything is taken care of.

Deployment

There are tons of parameters that you can play with in Apache NiFi. Therefore, instead of providing an Helm chart and trying to expose and maintain all those parameters to cover every specific use cases, I chose to use Kustomize definitions and provide different base deployments for different authentication mechanisms and cluster sizes. Those "base examples" are then easy to modify for your own needs.

Here are the steps to follow (from the base folder deployment):

kustomize build -o /path/to/output/folder

NOTE: with a recent version of OpenShift (=>4.8), you can also do directly oc apply -k . instead of the build + apply

Parameters

The base configuration deploys a NiFi cluster with a SingleUser authentication mechanism, and 3 nodes. You can modify the following parameters in params.env:

NOTE: don't modify the nodes_number directly in the params.env file! Use one of the overlays to modify the number of nodes (yeah, I know, that's a limitation of kustomize).

Overlays

Nodes number

Different overlays are available, named replicas-1 to replicas-5. They allow you to deploy the NiFi cluster with more or less nodes than the default 3.

OIDC authentication

To enable OIDC authentication, you can use the overlay oidc.

Supplemental parameters to change in params.env:

Full reference: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#openid_connect

LDAP Authentication

To enable LDAP authentication, you can use the overlay ldap.

Supplemental parameters to change in params.env:

Full reference: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#ldap_login_identity_provider

Other recipes

HTTP Listeners

The ListenHTTP component (like others) can open an http(s) port where you can send information directly into a workflow. To enable this functionality, you can apply the following recipe.

  1. Modify the Nifi StatefulSet at the container definition level to add the ports you want to use for the listener(s). Example:
ports:
    - name: metrics
        containerPort: 9092
        protocol: TCP
    - name: https
        containerPort: 9443
        protocol: TCP
    - name: cluster
        containerPort: 6007
        protocol: TCP
    - name: httplistener
        containerPort: 8888
        protocol: TCP
  1. Create a Service to connect to the pods on this port. Example:
kind: Service
apiVersion: v1
metadata:
  name: nifi-httplistener
  namespace: nifi
spec:
  ports:
    - name: httplistener
      protocol: TCP
      port: 8888
      targetPort: 8888
  selector:
    app: nifi
  1. Create a Route to connect to the service, with SSL or not. SSL in this example:
kind: Route
apiVersion: route.openshift.io/v1
metadata:
  name: nifi-httplistener
  namespace: nifi
spec:
  to:
    kind: Service
    name: nifi-httplistener
  port:
    targetPort: httplistener
  tls:
    termination: edge
    insecureEdgeTerminationPolicy: Redirect
  wildcardPolicy: None
  1. In NiFi, create the listener, configured on the port you defined at 1.

listenhttp

  1. (Optional) When using SSL, you have to define an SSL Context Service that will use the local certificates created during the deployment.

sslcontextservice

The following properties are used. All the stores have the same password, set at deployment, to allow the configuration of the service, which is unique in the cluster, to work on all nodes.