search

konpyutaika / nifikop

The NiFiKop NiFi Kubernetes operator makes it easy to run Apache NiFi on Kubernetes. Apache NiFI is a free, open-source solution that support powerful and scalable directed graphs of data routing, transformation, and system mediation logic.
https://konpyutaika.github.io/nifikop/
Apache License 2.0
122 stars 40 forks source link

nifikop 1.6.0 duplicates flow on canvas (deploys two instead of one) #346

Closed mostafamohajeri closed 4 months ago

mostafamohajeri commented 6 months ago

What steps will reproduce the bug?

Hi, thanks for the help and the work, here's the method to reproduce

  1. In minikube
  2. follow the starting guide
  3. start minikube minikube start --driver=docker --cpus=2 --memory=8000 --extra-config=kubelet.cgroup-driver=systemd
  4. install cert-manager kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.7.2/cert-manager.yaml
  5. install nifikop helm install nifikop oci://ghcr.io/konpyutaika/helm-charts/nifikop --namespace=nifi --version 1.6.0 --set image.tag=v1.6.0-release --set resources.requests.memory=256Mi --set resources.requests.cpu=250m --set resources.limits.memory=256Mi --set resources.limits.cpu=250m --set namespaces={"nifi"} --create-namespace
  6. deploy zookeeper helm install zookeeper bitnami/zookeeper --namespace=zookeeper --set resources.requests.memory=256Mi --set resources.requests.cpu=250m --set resources.limits.memory=256Mi --set resources.limits.cpu=250m --set global.storageClass=standard --set networkPolicy.enabled=true --set replicaCount=1 --create-namespace

deploy everything:

  1. kubectl apply -f cluster.yaml -n nifi
  2. kubectl apply -f paramcontext.yaml -n nifi
  3. kubectl apply -f registry.yaml -n nifi
  4. kubectl apply -f flow.yaml -n nifi

cluster

apiVersion: nifi.konpyutaika.com/v1
kind: NifiCluster
metadata:
  name: simplenifi
spec:
  service:
    headlessEnabled: true
    labels:
      cluster-name: simplenifi
  zkAddress: "zookeeper.zookeeper:2181"
  zkPath: /simplenifi
  externalServices:
    - metadata:
        labels:
          cluster-name: driver-simplenifi
      name: driver-ip
      spec:
        portConfigs:
          - internalListenerName: http
            port: 18443
        type: LoadBalancer
  clusterImage: "apache/nifi:1.23.2"
  initContainerImage: "bash:5.2.2"
  oneNifiNodePerNode: true
  readOnlyConfig:
    nifiProperties:
      overrideConfigs: |
        nifi.sensitive.props.key=thisIsABadSensitiveKeyPassword
  pod:
    labels:
      cluster-name: simplenifi
  nodeConfigGroups:
    default_group:
      imagePullPolicy: IfNotPresent
      isNode: true
      serviceAccountName: default
      storageConfigs:
        - mountPath: "/opt/nifi/nifi-current/logs"
          name: logs
          pvcSpec:
            accessModes:
              - ReadWriteOnce
            storageClassName: "standard"
            resources:
              requests:
                storage: 1Gi
      resourcesRequirements:
        limits:
          cpu: "1"
          memory: 1Gi
        requests:
          cpu: "1"
          memory: 1Gi
  nodes:
    - id: 1
      nodeConfigGroup: "default_group"
  propagateLabels: true
  nifiClusterTaskSpec:
    retryDurationMinutes: 10
  listenersConfig:
    internalListeners:
      - containerPort: 8443
        type: http
        name: http
      - containerPort: 6007
        type: cluster
        name: cluster
      - containerPort: 10000
        type: s2s
        name: s2s
      - containerPort: 9090
        type: prometheus
        name: prometheus
      - containerPort: 6342
        type: load-balance
        name: load-balance

param context

# kubectl create secret generic secret-params --from-literal=secret1=yop --from-literal=secret2=yep
apiVersion: nifi.konpyutaika.com/v1
kind: NifiParameterContext
metadata:
  name: dataflow-parameters
spec:
  description: "It is a test"
  clusterRef:
    name: simplenifi
    namespace: nifi
  parameters:
    - name: test
      value: toto
      description: tutu
    - name: test2
      value: toto
      description: toto

registry client

apiVersion: nifi.konpyutaika.com/v1
kind: NifiRegistryClient
metadata:
  name: registryclient
spec:
  clusterRef:
    name: simplenifi
    namespace: nifi
  description: "cool demo"
  uri: "http://reg.local:18080/"

flow

apiVersion: nifi.konpyutaika.com/v1
kind: NifiDataflow
metadata:
  name: testing1234
spec:
#  parentProcessGroupID: "d55f90d4-018c-1000-0000-00004a2c0a5a"
  bucketId: "571b4f9e-7839-412e-be75-3b06aa55dc3e"
  flowId: "7880d07c-4991-4a53-ac44-95ffc9f28850"
  flowVersion: 2
  syncMode: once
  skipInvalidControllerService: true
  skipInvalidComponent: true
  clusterRef:
    name: simplenifi
    namespace: nifi
  registryClientRef:
    name: registryclient
    namespace: nifi
  flowPosition:
    posX: 12
    posY: 200
  parameterContextRef:
    name: dataflow-parameters
    namespace: nifi
  updateStrategy: drain

What is the expected behavior?

one flow being deployed on the canvas

What do you see instead?

two copies of the flow end up on the canvas

Only one seems to be the correct one:

nifikop log after deploy (not touching the canvas/ui):

{"level":"info","time":"2024-01-05T11:31:27.608Z","logger":"controllers.NifiDataflow","caller":"controllers/controller_common.go:36","msg":"failed to patch status for NifiDataflow testing1234"}
{"level":"error","time":"2024-01-05T11:31:27.608Z","caller":"controller/controller.go:329","msg":"Reconciler error","controller":"nifidataflow","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiDataflow","nifiDataflow":{"name":"testing1234","namespace":"nifi"},"namespace":"nifi","name":"testing1234","reconcileID":"ad3d2c79-c535-419f-9523-e58636d66ca5","error":"Operation cannot be fulfilled on nifidataflows.nifi.konpyutaika.com \"testing1234\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:235"}
{"level":"info","time":"2024-01-05T11:31:27.704Z","logger":"controllers.NifiDataflow","caller":"controllers/nifidataflow_controller.go:412","msg":"Adding Finalizer for NifiDataflow testing1234"}
{"level":"info","time":"2024-01-05T11:31:27.713Z","logger":"controllers.NifiDataflow","caller":"controllers/controller_common.go:36","msg":"failed to update NifiDataflow testing1234"}
{"level":"error","time":"2024-01-05T11:31:27.713Z","caller":"controller/controller.go:329","msg":"Reconciler error","controller":"nifidataflow","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiDataflow","nifiDataflow":{"name":"testing1234","namespace":"nifi"},"namespace":"nifi","name":"testing1234","reconcileID":"ffa329e0-4b4d-4fc9-ba25-a5309dd327b3","error":"Operation cannot be fulfilled on nifidataflows.nifi.konpyutaika.com \"testing1234\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:235"}
{"level":"info","time":"2024-01-05T11:31:27.742Z","logger":"controllers.NifiDataflow","caller":"controllers/nifidataflow_controller.go:412","msg":"Adding Finalizer for NifiDataflow testing1234"}
{"level":"info","time":"2024-01-05T11:31:27.831Z","logger":"controllers.NifiDataflow","caller":"controllers/controller_common.go:36","msg":"failed to patch status for NifiDataflow testing1234"}
{"level":"error","time":"2024-01-05T11:31:27.831Z","caller":"controller/controller.go:329","msg":"Reconciler error","controller":"nifidataflow","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiDataflow","nifiDataflow":{"name":"testing1234","namespace":"nifi"},"namespace":"nifi","name":"testing1234","reconcileID":"a6eee858-299f-45a8-8584-1c2b1c80e399","error":"Operation cannot be fulfilled on nifidataflows.nifi.konpyutaika.com \"testing1234\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:235"}
{"level":"info","time":"2024-01-05T11:31:28.001Z","logger":"controllers.NifiDataflow","caller":"controllers/controller_common.go:36","msg":"failed to update NifiDataflow testing1234"}
{"level":"error","time":"2024-01-05T11:31:28.001Z","caller":"controller/controller.go:329","msg":"Reconciler error","controller":"nifidataflow","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiDataflow","nifiDataflow":{"name":"testing1234","namespace":"nifi"},"namespace":"nifi","name":"testing1234","reconcileID":"4ddf87ad-4a81-4442-8842-75bb532d8863","error":"Operation cannot be fulfilled on nifidataflows.nifi.konpyutaika.com \"testing1234\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:235"}
{"level":"info","time":"2024-01-05T11:31:32.244Z","logger":"controllers.NifiCluster","caller":"controllers/nificluster_controller.go:135","msg":"NifiCluster starting reconciliation","clusterName":"simplenifi"}
{"level":"info","time":"2024-01-05T11:31:32.357Z","logger":"controllers.NifiCluster","caller":"nifi/nifi.go:284","msg":"Successfully reconciled cluster","component":"nifi","clusterName":"simplenifi","clusterNamespace":"nifi"}
{"level":"info","time":"2024-01-05T11:31:43.943Z","logger":"controllers.NifiDataflow","caller":"controllers/controller_common.go:36","msg":"failed to update status for NifiDataflow testing1234"}
{"level":"error","time":"2024-01-05T11:31:43.943Z","caller":"controller/controller.go:329","msg":"Reconciler error","controller":"nifidataflow","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiDataflow","nifiDataflow":{"name":"testing1234","namespace":"nifi"},"namespace":"nifi","name":"testing1234","reconcileID":"b0df95d6-6ba5-4319-b9cd-8b09f4b53292","error":"Operation cannot be fulfilled on nifidataflows.nifi.konpyutaika.com \"testing1234\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:235"}
{"level":"info","time":"2024-01-05T11:31:44.200Z","logger":"controllers.NifiDataflow","caller":"controllers/controller_common.go:36","msg":"failed to update NifiDataflow testing1234"}
{"level":"error","time":"2024-01-05T11:31:44.200Z","caller":"controller/controller.go:329","msg":"Reconciler error","controller":"nifidataflow","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiDataflow","nifiDataflow":{"name":"testing1234","namespace":"nifi"},"namespace":"nifi","name":"testing1234","reconcileID":"a4421d7f-2aab-49ee-bc37-783386964fdc","error":"Operation cannot be fulfilled on nifidataflows.nifi.konpyutaika.com \"testing1234\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:235"}
{"level":"info","time":"2024-01-05T11:31:52.378Z","logger":"controllers.NifiCluster","caller":"controllers/nificluster_controller.go:135","msg":"NifiCluster starting reconciliation","clusterName":"simplenifi"}
{"level":"info","time":"2024-01-05T11:31:52.470Z","logger":"controllers.NifiCluster","caller":"nifi/nifi.go:284","msg":"Successfully reconciled cluster","component":"nifi","clusterName":"simplenifi","clusterNamespace":"nifi"}

Possible solution

nifikop v1.5.0 works as expected already

even the same cluster and resources if `helm del' version 1.6.0 and install 1.5.0 next flow deployments work fine. The vice-versa is also true, removing 1.5.0 and installing 1.6.0 brings the duplicating behavior back

have tried deleting minikube profile and other restart/re-installs

NiFiKop version

v1.6.0

Golang version

not installed

Kubernetes version

Client Version: v1.29.0 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.28.3

NiFi version

1.23.2

Additional context

Thanks a lot for the help

mh013370 commented 6 months ago

Is it possible to show a screenshot of what the canvas looks like?

I have actually seen this happen before, but I haven't tracked down what causes it.

mh013370 commented 6 months ago

I don't see this on any of our v1.6.0 nifikop driven NiFi clusters, but they were all upgraded with flows already in place. Perhaps this happens under a new flow deployment scenario.

mostafamohajeri commented 6 months ago

hi @mh013370 , thanks for the response indeed if I delete the duplicate and keep doing kubectl replace flow.yaml the duplicate does not come but with apply and delete/create it comes back I will create a screenshot, but the two stack on eachother so normally you won't see them (except for the darkened shadow) unless i move one. I already tested setting the position (hence the 12, 200 xy)

mostafamohajeri commented 6 months ago

please see the two images, one is the two flows stacked, the flow on top is the "false" one, its name is still the same as the name in the registry (testing) and the other screenshot is after I moved it which shows the "true" flow which has the name of the resource (testing1234), it enables/starts and even if I move it the nifi operator reconciles it back

image image(1)

mh013370 commented 6 months ago

Yes, i have seen exactly this scenario in the past. And like you describe, it doesn't impact anything because the duplicate doesn't get enabled/started by the operator. It should be addressed nonetheless.

Would it be possible to enable debug logging and share the log output? I'm curious what is causing the creation of two process groups in the UI.

https://github.com/konpyutaika/nifikop/blob/master/helm/nifikop/values.yaml#L57

mostafamohajeri commented 6 months ago

logs a few lines before I deployed the flow until after

{"level":"debug","time":"2024-01-05T14:56:06.851Z","logger":"events","caller":"recorder/recorder.go:103","msg":"Reconciling parameter context dataflow-parameters","type":"Normal","object":{"kind":"NifiParameterContext","namespace":"nifi","name":"dataflow-parameters","uid":"9086ef82-f950-4611-910d-ffce93353062","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"32040"},"reason":"Reconciling"}
{"level":"debug","time":"2024-01-05T14:56:06.875Z","logger":"events","caller":"recorder/recorder.go:103","msg":"Synchronizing parameter context dataflow-parameters","type":"Normal","object":{"kind":"NifiParameterContext","namespace":"nifi","name":"dataflow-parameters","uid":"9086ef82-f950-4611-910d-ffce93353062","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"32040"},"reason":"Synchronizing"}
{"level":"debug","time":"2024-01-05T14:56:06.909Z","logger":"events","caller":"recorder/recorder.go:103","msg":"Synchronized parameter context dataflow-parameters","type":"Normal","object":{"kind":"NifiParameterContext","namespace":"nifi","name":"dataflow-parameters","uid":"9086ef82-f950-4611-910d-ffce93353062","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"32040"},"reason":"Synchronized"}
{"level":"debug","time":"2024-01-05T14:56:06.922Z","logger":"controllers.NifiParameterContext","caller":"controllers/nifiparametercontext_controller.go:339","msg":"Ensured Parameter Context","parameterContext":"dataflow-parameters"}
{"level":"debug","time":"2024-01-05T14:56:06.922Z","logger":"events","caller":"recorder/recorder.go:103","msg":"Reconciling parameter context dataflow-parameters","type":"Normal","object":{"kind":"NifiParameterContext","namespace":"nifi","name":"dataflow-parameters","uid":"9086ef82-f950-4611-910d-ffce93353062","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"32040"},"reason":"Reconciled"}
{"level":"debug","time":"2024-01-05T14:56:21.622Z","logger":"events","caller":"recorder/recorder.go:103","msg":"Reconciling registry client registryclient","type":"Normal","object":{"kind":"NifiRegistryClient","namespace":"nifi","name":"registryclient","uid":"9dc325b4-02a4-497c-b6f9-2cd48e3114e7","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"31926"},"reason":"Reconciling"}
{"level":"debug","time":"2024-01-05T14:56:21.653Z","logger":"events","caller":"recorder/recorder.go:103","msg":"Synchronizing registry client registryclient","type":"Normal","object":{"kind":"NifiRegistryClient","namespace":"nifi","name":"registryclient","uid":"9dc325b4-02a4-497c-b6f9-2cd48e3114e7","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"31926"},"reason":"Synchronizing"}
{"level":"debug","time":"2024-01-05T14:56:21.681Z","logger":"events","caller":"recorder/recorder.go:103","msg":"Synchronized registry client registryclient","type":"Normal","object":{"kind":"NifiRegistryClient","namespace":"nifi","name":"registryclient","uid":"9dc325b4-02a4-497c-b6f9-2cd48e3114e7","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"31926"},"reason":"Synchronized"}
{"level":"debug","time":"2024-01-05T14:56:21.690Z","logger":"controllers.NifiRegistryClient","caller":"controllers/nifiregistryclient_controller.go:271","msg":"Ensured Registry Client","registryClient":"registryclient"}
{"level":"debug","time":"2024-01-05T14:56:21.691Z","logger":"events","caller":"recorder/recorder.go:103","msg":"Reconciling registry client registryclient","type":"Normal","object":{"kind":"NifiRegistryClient","namespace":"nifi","name":"registryclient","uid":"9dc325b4-02a4-497c-b6f9-2cd48e3114e7","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"31926"},"reason":"Reconciled"}
{"level":"debug","time":"2024-01-05T14:56:21.924Z","logger":"events","caller":"recorder/recorder.go:103","msg":"Reconciling parameter context dataflow-parameters","type":"Normal","object":{"kind":"NifiParameterContext","namespace":"nifi","name":"dataflow-parameters","uid":"9086ef82-f950-4611-910d-ffce93353062","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"32040"},"reason":"Reconciling"}
{"level":"debug","time":"2024-01-05T14:56:21.958Z","logger":"events","caller":"recorder/recorder.go:103","msg":"Synchronizing parameter context dataflow-parameters","type":"Normal","object":{"kind":"NifiParameterContext","namespace":"nifi","name":"dataflow-parameters","uid":"9086ef82-f950-4611-910d-ffce93353062","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"32040"},"reason":"Synchronizing"}
{"level":"debug","time":"2024-01-05T14:56:21.991Z","logger":"events","caller":"recorder/recorder.go:103","msg":"Synchronized parameter context dataflow-parameters","type":"Normal","object":{"kind":"NifiParameterContext","namespace":"nifi","name":"dataflow-parameters","uid":"9086ef82-f950-4611-910d-ffce93353062","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"32040"},"reason":"Synchronized"}
{"level":"debug","time":"2024-01-05T14:56:22.012Z","logger":"controllers.NifiParameterContext","caller":"controllers/nifiparametercontext_controller.go:339","msg":"Ensured Parameter Context","parameterContext":"dataflow-parameters"}
{"level":"debug","time":"2024-01-05T14:56:22.012Z","logger":"events","caller":"recorder/recorder.go:103","msg":"Reconciling parameter context dataflow-parameters","type":"Normal","object":{"kind":"NifiParameterContext","namespace":"nifi","name":"dataflow-parameters","uid":"9086ef82-f950-4611-910d-ffce93353062","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"32040"},"reason":"Reconciled"}
{"level":"info","time":"2024-01-05T14:56:22.245Z","logger":"controllers.NifiCluster","caller":"controllers/nificluster_controller.go:135","msg":"NifiCluster starting reconciliation","clusterName":"simplenifi"}
{"level":"debug","time":"2024-01-05T14:56:22.245Z","logger":"controllers.NifiCluster","caller":"nifi/nifi.go:85","msg":"reconciling","component":"nifi","clusterName":"simplenifi","clusterNamespace":"nifi"}
{"level":"debug","time":"2024-01-05T14:56:22.245Z","logger":"controllers.NifiCluster","caller":"k8sutil/resource.go:31","msg":"reconciling","kind":"*v1.Service","name":"simplenifi-headless"}
{"level":"debug","time":"2024-01-05T14:56:22.245Z","logger":"events","caller":"recorder/recorder.go:103","msg":"NifiCluster starting reconciliation","type":"Normal","object":{"kind":"NifiCluster","namespace":"nifi","name":"simplenifi","uid":"2bf91600-19f2-4857-aecd-6193874867b8","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"31858"},"reason":"ClusterReconciling"}
{"level":"debug","time":"2024-01-05T14:56:22.249Z","logger":"controllers.NifiCluster","caller":"k8sutil/resource.go:132","msg":"resource is in sync","kind":"*v1.Service"}
{"level":"debug","time":"2024-01-05T14:56:22.249Z","logger":"controllers.NifiCluster","caller":"k8sutil/resource.go:116","msg":"resource updated","name":"simplenifi-headless","namespace":"nifi","kind":""}
{"level":"debug","time":"2024-01-05T14:56:22.249Z","logger":"controllers.NifiCluster","caller":"nifi/nifi.go:504","msg":"searching for pvc with label because name is empty","nifiCluster":"simplenifi","nodeId":"1","kind":"*v1.PersistentVolumeClaim"}
{"level":"debug","time":"2024-01-05T14:56:22.251Z","logger":"controllers.NifiCluster","caller":"k8sutil/resource.go:132","msg":"resource is in sync","kind":"*v1.PersistentVolumeClaim"}
{"level":"debug","time":"2024-01-05T14:56:22.255Z","logger":"controllers.NifiCluster","caller":"k8sutil/resource.go:31","msg":"reconciling","kind":"*v1.Secret","name":"simplenifi-config-1"}
{"level":"debug","time":"2024-01-05T14:56:22.294Z","logger":"controllers.NifiCluster","caller":"k8sutil/resource.go:132","msg":"resource is in sync","kind":"*v1.Secret"}
{"level":"debug","time":"2024-01-05T14:56:22.294Z","logger":"controllers.NifiCluster","caller":"k8sutil/resource.go:116","msg":"resource updated","name":"simplenifi-config-1","namespace":"nifi","kind":""}
{"level":"debug","time":"2024-01-05T14:56:22.294Z","logger":"controllers.NifiCluster","caller":"nifi/nifi.go:588","msg":"searching for pod with label because name is empty","clusterName":"simplenifi","nodeId":"1","kind":"*v1.Pod"}
{"level":"debug","time":"2024-01-05T14:56:22.304Z","logger":"controllers.NifiCluster","caller":"nifi/nifi.go:703","msg":"pod resource is in sync","clusterName":"simplenifi","podName":""}
{"level":"debug","time":"2024-01-05T14:56:22.304Z","logger":"controllers.NifiCluster","caller":"k8sutil/resource.go:31","msg":"reconciling","kind":"*v1.Service","name":"driver-ip"}
{"level":"debug","time":"2024-01-05T14:56:22.305Z","logger":"controllers.NifiCluster","caller":"k8sutil/resource.go:132","msg":"resource is in sync","kind":"*v1.Service"}
{"level":"debug","time":"2024-01-05T14:56:22.306Z","logger":"controllers.NifiCluster","caller":"k8sutil/resource.go:116","msg":"resource updated","name":"driver-ip","namespace":"nifi","kind":""}
{"level":"debug","time":"2024-01-05T14:56:22.369Z","logger":"controllers.NifiCluster","caller":"k8sutil/status.go:298","msg":"Root process group id updated","clusterName":"simplenifi","id":"da0a1f66-018c-1000-3b1a-d0ba0ea7acab"}
{"level":"info","time":"2024-01-05T14:56:22.415Z","logger":"controllers.NifiCluster","caller":"nifi/nifi.go:284","msg":"Successfully reconciled cluster","component":"nifi","clusterName":"simplenifi","clusterNamespace":"nifi"}
{"level":"debug","time":"2024-01-05T14:56:22.416Z","logger":"controllers.NifiCluster","caller":"controllers/nificluster_controller.go:172","msg":"ensuring finalizers on nificluster","clusterName":"simplenifi"}
{"level":"debug","time":"2024-01-05T14:56:28.428Z","logger":"events","caller":"recorder/recorder.go:103","msg":"Reconciling dataflow testing1234 based on flow {bucketId : 571b4f9e-7839-412e-be75-3b06aa55dc3e, flowId: 7880d07c-4991-4a53-ac44-95ffc9f28850, version: 2}","type":"Normal","object":{"kind":"NifiDataflow","namespace":"nifi","name":"testing1234","uid":"2f254259-349e-44e2-879c-d6cd3eae37e9","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"34435"},"reason":"Reconciling"}
{"level":"debug","time":"2024-01-05T14:56:28.428Z","logger":"events","caller":"recorder/recorder.go:103","msg":"Creating dataflow testing1234 based on flow {bucketId : 571b4f9e-7839-412e-be75-3b06aa55dc3e, flowId: 7880d07c-4991-4a53-ac44-95ffc9f28850, version: 2}","type":"Normal","object":{"kind":"NifiDataflow","namespace":"nifi","name":"testing1234","uid":"2f254259-349e-44e2-879c-d6cd3eae37e9","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"34435"},"reason":"Creating"}
{"level":"info","time":"2024-01-05T14:56:28.612Z","logger":"controllers.NifiDataflow","caller":"controllers/controller_common.go:36","msg":"failed to patch status for NifiDataflow testing1234"}
{"level":"error","time":"2024-01-05T14:56:28.613Z","caller":"controller/controller.go:329","msg":"Reconciler error","controller":"nifidataflow","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiDataflow","nifiDataflow":{"name":"testing1234","namespace":"nifi"},"namespace":"nifi","name":"testing1234","reconcileID":"ba807e74-7ef9-4016-bcfb-7ee44d1d91bb","error":"Operation cannot be fulfilled on nifidataflows.nifi.konpyutaika.com \"testing1234\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:235"}
{"level":"debug","time":"2024-01-05T14:56:28.615Z","logger":"events","caller":"recorder/recorder.go:103","msg":"Reconciling dataflow testing1234 based on flow {bucketId : 571b4f9e-7839-412e-be75-3b06aa55dc3e, flowId: 7880d07c-4991-4a53-ac44-95ffc9f28850, version: 2}","type":"Normal","object":{"kind":"NifiDataflow","namespace":"nifi","name":"testing1234","uid":"2f254259-349e-44e2-879c-d6cd3eae37e9","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"34435"},"reason":"Reconciling"}
{"level":"debug","time":"2024-01-05T14:56:28.615Z","logger":"events","caller":"recorder/recorder.go:103","msg":"Creating dataflow testing1234 based on flow {bucketId : 571b4f9e-7839-412e-be75-3b06aa55dc3e, flowId: 7880d07c-4991-4a53-ac44-95ffc9f28850, version: 2}","type":"Normal","object":{"kind":"NifiDataflow","namespace":"nifi","name":"testing1234","uid":"2f254259-349e-44e2-879c-d6cd3eae37e9","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"34435"},"reason":"Creating"}
{"level":"info","time":"2024-01-05T14:56:28.823Z","logger":"controllers.NifiDataflow","caller":"controllers/nifidataflow_controller.go:412","msg":"Adding Finalizer for NifiDataflow testing1234"}
{"level":"debug","time":"2024-01-05T14:56:28.823Z","logger":"events","caller":"recorder/recorder.go:103","msg":"Created dataflow testing1234 based on flow {bucketId : 571b4f9e-7839-412e-be75-3b06aa55dc3e, flowId: 7880d07c-4991-4a53-ac44-95ffc9f28850, version: 2}","type":"Normal","object":{"kind":"NifiDataflow","namespace":"nifi","name":"testing1234","uid":"2f254259-349e-44e2-879c-d6cd3eae37e9","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"34440"},"reason":"Created"}
{"level":"info","time":"2024-01-05T14:56:28.834Z","logger":"controllers.NifiDataflow","caller":"controllers/controller_common.go:36","msg":"failed to update NifiDataflow testing1234"}
{"level":"error","time":"2024-01-05T14:56:28.834Z","caller":"controller/controller.go:329","msg":"Reconciler error","controller":"nifidataflow","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiDataflow","nifiDataflow":{"name":"testing1234","namespace":"nifi"},"namespace":"nifi","name":"testing1234","reconcileID":"3711cf1c-24c6-4971-ac0e-1a290a5fb0e8","error":"Operation cannot be fulfilled on nifidataflows.nifi.konpyutaika.com \"testing1234\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:235"}
{"level":"debug","time":"2024-01-05T14:56:28.836Z","logger":"events","caller":"recorder/recorder.go:103","msg":"Reconciling dataflow testing1234 based on flow {bucketId : 571b4f9e-7839-412e-be75-3b06aa55dc3e, flowId: 7880d07c-4991-4a53-ac44-95ffc9f28850, version: 2}","type":"Normal","object":{"kind":"NifiDataflow","namespace":"nifi","name":"testing1234","uid":"2f254259-349e-44e2-879c-d6cd3eae37e9","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"34440"},"reason":"Reconciling"}
{"level":"info","time":"2024-01-05T14:56:28.879Z","logger":"controllers.NifiDataflow","caller":"controllers/nifidataflow_controller.go:412","msg":"Adding Finalizer for NifiDataflow testing1234"}
{"level":"info","time":"2024-01-05T14:56:29.015Z","logger":"controllers.NifiDataflow","caller":"controllers/controller_common.go:36","msg":"failed to patch status for NifiDataflow testing1234"}
{"level":"error","time":"2024-01-05T14:56:29.016Z","caller":"controller/controller.go:329","msg":"Reconciler error","controller":"nifidataflow","controllerGroup":"nifi.konpyutaika.com","controllerKind":"NifiDataflow","nifiDataflow":{"name":"testing1234","namespace":"nifi"},"namespace":"nifi","name":"testing1234","reconcileID":"5e82b5cb-a31c-49a6-bc76-ad6512123387","error":"Operation cannot be fulfilled on nifidataflows.nifi.konpyutaika.com \"testing1234\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:235"}
{"level":"debug","time":"2024-01-05T14:56:29.017Z","logger":"events","caller":"recorder/recorder.go:103","msg":"Reconciling dataflow testing1234 based on flow {bucketId : 571b4f9e-7839-412e-be75-3b06aa55dc3e, flowId: 7880d07c-4991-4a53-ac44-95ffc9f28850, version: 2}","type":"Normal","object":{"kind":"NifiDataflow","namespace":"nifi","name":"testing1234","uid":"2f254259-349e-44e2-879c-d6cd3eae37e9","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"34443"},"reason":"Reconciling"}
{"level":"debug","time":"2024-01-05T14:56:29.185Z","logger":"events","caller":"recorder/recorder.go:103","msg":"Reconciling dataflow testing1234 based on flow {bucketId : 571b4f9e-7839-412e-be75-3b06aa55dc3e, flowId: 7880d07c-4991-4a53-ac44-95ffc9f28850, version: 2}","type":"Normal","object":{"kind":"NifiDataflow","namespace":"nifi","name":"testing1234","uid":"2f254259-349e-44e2-879c-d6cd3eae37e9","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"34445"},"reason":"Reconciling"}
{"level":"debug","time":"2024-01-05T14:56:29.231Z","logger":"events","caller":"recorder/recorder.go:103","msg":"Syncing dataflow testing1234 based on flow {bucketId : 571b4f9e-7839-412e-be75-3b06aa55dc3e, flowId: 7880d07c-4991-4a53-ac44-95ffc9f28850, version: 2}","type":"Normal","object":{"kind":"NifiDataflow","namespace":"nifi","name":"testing1234","uid":"2f254259-349e-44e2-879c-d6cd3eae37e9","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"34445"},"reason":"Synchronizing"}
{"level":"debug","time":"2024-01-05T14:56:29.521Z","logger":"events","caller":"recorder/recorder.go:103","msg":"Reconciling dataflow testing1234 based on flow {bucketId : 571b4f9e-7839-412e-be75-3b06aa55dc3e, flowId: 7880d07c-4991-4a53-ac44-95ffc9f28850, version: 2}","type":"Normal","object":{"kind":"NifiDataflow","namespace":"nifi","name":"testing1234","uid":"2f254259-349e-44e2-879c-d6cd3eae37e9","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"34445"},"reason":"Reconciling"}
{"level":"debug","time":"2024-01-05T14:56:29.614Z","logger":"events","caller":"recorder/recorder.go:103","msg":"Syncing dataflow testing1234 based on flow {bucketId : 571b4f9e-7839-412e-be75-3b06aa55dc3e, flowId: 7880d07c-4991-4a53-ac44-95ffc9f28850, version: 2}","type":"Normal","object":{"kind":"NifiDataflow","namespace":"nifi","name":"testing1234","uid":"2f254259-349e-44e2-879c-d6cd3eae37e9","apiVersion":"nifi.konpyutaika.com/v1","resourceVersion":"34445"},"reason":"Synchronizing"}
mostafamohajeri commented 6 months ago

I am guessing the Creating dataflow testing1234 based on flow ... should not be happening twice

juldrixx commented 6 months ago

It appears that the operator failed to update the resource status to store the dataflow ID. Therefore, during the next reconciliation loop, the operator thinks that the dataflow has not been deployed. So he deploys it again.

But it shoudn't because we changed the update with patch. But maybe it the cause. @mh013370 what do you think?

mostafamohajeri commented 4 months ago

Fixed in 1.7.0