Attach an existing process group to a version

[gaving]

• Nipyapi version: master
• NiFi version: 1.8.0
• NiFi-Registry version: 0.30

Hi - Is there any way to attach a process group to an existing version from version control?

I've trying to integrate building a flow as part of a CI process, pushing the newly built flow to a registry bucket on every commit.

I'm getting the error Error creating snapshot: A Versioned flow snapshot with the same version already exists: 1 on successive builds I presume is because the flow is being 'built' separately and then I'm trying to use save_flow_ver.

For example I have the following bit of code:-

try:
    group = "Extract"
    process_group = nipyapi.canvas.get_process_group(group)

    flow = nipyapi.versioning.get_flow_in_bucket(
        bucket_id=bucket.identifier, identifier=group
    )

    if flow is not None:

        # somehow tell this it's an existing flow
        # let the new flow think it's an update without checking it out first
        # can it be checked out then swap its contents?

        nipyapi.versioning.save_flow_ver(
            process_group=process_group,
            registry_client=reg_client,
            bucket=bucket,
            flow_id=flow.identifier,
            comment=f"Updated {group} group",
        )
    else:
        nipyapi.versioning.save_flow_ver(
            process_group=process_group,
            registry_client=reg_client,
            bucket=bucket,
            flow_name=group,
            desc=f"{group} group",
            comment="Initial commit",
        )
except AttributeError as e:
    log.error(f"Cannot find unique process group named '{group}'")
except ValueError as e:
    log.error(e)

Can I somehow "check out" the flow, modify it, and save it back to the registry?

Had a look at fdlc.py but couldn't see anything obvious.

Thanks for any help!

[Chaffelson]

I don't believe this is currently possible, but I have discussed it with @kevdoran in the past and he may know more about whether the community is working on an enhancement to enable it.

[gaving]

@Chaffelson Thanks for the quick update, I wondered if I was missing something obvious.

Tried a few things:-

• Going down the route of trying to extract the version keys from a checked out flow and assign them back to the newly built flow.
• Deploying the flow and then deleting all the processors under it then building the new flow inside the group, but would also have to remove controller services, connections, processors, ports, etc.

Both approaches seemed way too hacky.

Only non-ideal solution I can think of from a CI point of view would be to append commit hashes to flow names or something, but this obviously breaks the registry workflow on the canvas for other developers when a new flow was built and pushed from CI.

[jrittenh]

@gaving If I'm understanding correctly, you have NiFi->Process Group->Processor(s) that exist unversioned. You also have Registry->Bucket->Flow that matches the unversioned group. Am I getting that right?

If that's accurate, a workaround might be to create a new process group, attach it to the registry, pull the existing version, then wipe the internals of the group and copy in the internals from the existing unversioned group.

[gaving]

Hi @jrittenh -

Yeah, you understand my issue correctly. I tried the workaround as you suggest but wiping the internals looked quite overly hacky e.g. I didn't see a method for that so I'd have to loop over the contained processors, connections, ports, to nuke them one by one.

Seemed a bit too experimental to base our teams workflow off it so figured I'd post to see if there was a better way.

I could take another look at that approach, though...

[kevdoran]

I think the hacky approaches you came up with are the best we can do for now. There is no way to attach an untracked flow/PG to one that is already tracked.

[Chaffelson]

Flagging as wontfix as it's due to the way that NiFi works, not something I can resolve within a client.

[Chaffelson]

@gaving I'm going to close this as part of a housekeeping sweep up, please reopen it if you feel we can address this better in the Python client somehow.

[gaving]

@chaffelson no worries, I never got back to trying a workaround but if I ever try again I’ll update my findings here