No data volumes or claims are generated

[Happyman0815]

I followed all the steps and Nifi is running well but it does not store templates or the current workflow in volumes.

It seems that not data volumes or Persistent Volume Claims are generated or used.

I am working on a Windows 10 environment.

[whs-dot-hk]

Hi, are you using AKS?

[Happyman0815]

I use the Kubernetes integrated in Docker Desktop (Edge) for Windows:

https://hub.docker.com/editions/community/docker-ce-desktop-windows

Runs on a Microsoft Windows [Version 10.0.17763.253] host PC.

[whs-dot-hk]

Do you mean minikube?

[Happyman0815]

Docker Desktop for Windows 18.06.0-ce-win70 CE includes a standalone Kubernetes server and client, as well as Docker CLI integration. The Kubernetes server runs locally within your Docker instance, is not configurable, and is a single-node cluster. The Kubernetes server runs within a Docker container on your local system, and is only for local testing. When Kubernetes support is enabled, you can deploy your workloads, in parallel, on Kubernetes, Swarm, and as standalone containers. Enabling or disabling the Kubernetes server does not affect your other workloads.

Source: https://docs.docker.com/docker-for-windows/kubernetes/

I think it is not Minicube.

[whs-dot-hk]

How about running

$ kubectl delete -f https://raw.githubusercontent.com/whs-dot-hk/kubernetes-nifi-refined/master/nifi.yaml

# nifi-statefulset.yaml
apiVersion: v1
kind: Service
metadata:
  name: nifi-headless
  labels:
    app: nifi
spec:
  ports:
  - port: 8080
    name: http
  clusterIP: None
  selector:
    app: nifi
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nifi
spec:
  selector:
    matchLabels:
      app: nifi
      tier: backend
      track: stable
  serviceName: nifi-headless
  replicas: 1
  template:
    metadata:
      labels:
        app: nifi
        tier: backend
        track: stable
    spec:
      containers:
        - name: nifi
          image: whshk/nifi:latest
          ports:
            - name: http
              containerPort: 8080
          volumeMounts:
            - name: nifi-data
              mountPath: /opt/nifi/nifi-current/conf
              subPath: conf
            - name: nifi-data
              mountPath: /opt/nifi/nifi-current/logs
              subPath: logs
            - name: nifi-data
              mountPath: /opt/nifi/nifi-current/database_repository
              subPath: database_repository
            - name: nifi-data
              mountPath: /opt/nifi/nifi-current/flowfile_repository
              subPath: flowfile_repository
            - name: nifi-data
              mountPath: /opt/nifi/nifi-current/content_repository
              subPath: content_repository
            - name: nifi-data
              mountPath: /opt/nifi/nifi-current/provenance_repository
              subPath: provenance_repository
            - name: nifi-data
              mountPath: /opt/nifi/nifi-current/state
              subPath: state
      securityContext:
        runAsUser: 1000
        fsGroup: 1000
  volumeClaimTemplates:
    - metadata:
        name: nifi-data
      spec:
        accessModes: [ "ReadWriteOnce" ]
        storageClassName: ""
        resources:
          requests:
storage: 10Gi

$ kubectl create -f nifi-statefulset.yaml

Does it work for you? Please report back.

[Happyman0815]

Hi whs-dot-hk,

First of all: I have not used the nifi-statefulset.yaml and this was the biggest issue.

What I did

I deleted the "nifi.yaml" (as u said) and tried it with the original "nifi-statefulset.yaml" which addresses the storageClassName "default". Result: Persistent Volume Claim was created. Good - but my Kubernetes had then problems to bind the "default" class because it was not created. Symptom: Persistent Volume Claim was trapped in a Pending state.

After creating the following "default" Storage Class:

{ 
  "kind": "StorageClass",
  "apiVersion": "storage.k8s.io/v1",
  "metadata": {
    "name": "default",
    "annotations": {
      "storageclass.kubernetes.io/is-default-class": "true"
    }
  },
  "provisioner": "docker.io/hostpath",
  "reclaimPolicy": "Delete",
  "volumeBindingMode": "Immediate"
}

It worked fine for me. All Nifi workflows seem to be persistent now.

Recommendation

I'm not an expert in Kubernetes and just needed something which is quickly running. Although these problems would have been (probably) easy solvable for an expert, I would adapt the readme a bit in a way that also non experts like me are able to run it without issues.

Thanks for your support whs-dot-hk!

PS: Don't create nifi-statefulset.yaml AND nifi.yaml. It will freeze Kubernetes (CPU load 100 %).

[whs-dot-hk]

I will keep that in mind. And will add a Storage Class example after I try it with a kops cluster this weekend.