Add support for Run once (RUN_ONCE) state of processors and process groups

[ihor-sokoliuk-exa]

• Nipyapi version: 0.19.0
• NiFi version: 1.16+
• NiFi-Registry version: None
• Python version: 3.9
• Operating System: Docker

Description

NiFi has added a RUN_ONCE state for a processor. https://github.com/apache/nifi/pull/1458/files But nipyapi lib doesn't support it. As a result, it fails when trying to get a processor/process group that was started with the Run once option.

return nipyapi.canvas.get_process_group(identifier=pg_name, identifier_type='name') File "/usr/local/lib/python3.9/dist-packages/nipyapi/utils.py", line 652, in wrapper return f(*args, kwargs) File "/usr/local/lib/python3.9/dist-packages/nipyapi/canvas.py", line 149, in get_process_group obj = list_all_process_groups() File "/usr/local/lib/python3.9/dist-packages/nipyapi/canvas.py", line 190, in list_all_process_groups root_flow = recurse_flow(pg_id) File "/usr/local/lib/python3.9/dist-packages/nipyapi/canvas.py", line 69, in recurse_flow this_flow = get_flow(this_pg_id) File "/usr/local/lib/python3.9/dist-packages/nipyapi/canvas.py", line 95, in get_flow return nipyapi.nifi.FlowApi().get_flow(pg_id) File "/usr/local/lib/python3.9/dist-packages/nipyapi/nifi/apis/flow_api.py", line 1989, in get_flow (data) = self.get_flow_with_http_info(id, kwargs) File "/usr/local/lib/python3.9/dist-packages/nipyapi/nifi/apis/flow_api.py", line 2060, in get_flow_with_http_info return self.api_client.call_api('/flow/process-groups/{id}', 'GET', File "/usr/local/lib/python3.9/dist-packages/nipyapi/nifi/api_client.py", line 330, in call_api return self.call_api(resource_path, method, File "/usr/local/lib/python3.9/dist-packages/nipyapi/nifi/api_client.py", line 163, in __call_api return_data = self.deserialize(response_data, response_type) File "/usr/local/lib/python3.9/dist-packages/nipyapi/nifi/api_client.py", line 244, in deserialize return self.deserialize(data, response_type) File "/usr/local/lib/python3.9/dist-packages/nipyapi/nifi/api_client.py", line 287, in deserialize return self.deserialize_model(data, klass) File "/usr/local/lib/python3.9/dist-packages/nipyapi/nifi/api_client.py", line 663, in deserialize_model kwargs[attr] = self.deserialize(value, attr_type) File "/usr/local/lib/python3.9/dist-packages/nipyapi/nifi/api_client.py", line 287, in deserialize return self.deserialize_model(data, klass) File "/usr/local/lib/python3.9/dist-packages/nipyapi/nifi/api_client.py", line 663, in deserialize_model kwargs[attr] = self.deserialize(value, attr_type) File "/usr/local/lib/python3.9/dist-packages/nipyapi/nifi/api_client.py", line 287, in deserialize return self.deserialize_model(data, klass) File "/usr/local/lib/python3.9/dist-packages/nipyapi/nifi/api_client.py", line 656, in deserialize_model deserialized_value = self.deserialize(value, attr_type) File "/usr/local/lib/python3.9/dist-packages/nipyapi/nifi/api_client.py", line 264, in deserialize return [self.deserialize(sub_data, sub_kls) File "/usr/local/lib/python3.9/dist-packages/nipyapi/nifi/api_client.py", line 264, in return [self.deserialize(sub_data, sub_kls) File "/usr/local/lib/python3.9/dist-packages/nipyapi/nifi/api_client.py", line 287, in deserialize return self.deserialize_model(data, klass) File "/usr/local/lib/python3.9/dist-packages/nipyapi/nifi/api_client.py", line 663, in __deserialize_model kwargs[attr] = self.deserialize(value, attr_type) File "/usr/local/lib/python3.9/dist-packages/nipyapi/nifi/api_client.py", line 287, in deserialize return self.deserialize_model(data, klass) File "/usr/local/lib/python3.9/dist-packages/nipyapi/nifi/api_client.py", line 665, in deserialize_model instance = klass(**kwargs) File "/usr/local/lib/python3.9/dist-packages/nipyapi/nifi/models/processor_dto.py", line 132, in init__ self.state = state File "/usr/local/lib/python3.9/dist-packages/nipyapi/nifi/models/processor_dto.py", line 349, in state raise ValueError( ValueError: Invalid value for state (RUN_ONCE), must be one of ['RUNNING', 'STOPPED', 'DISABLED']

What I Did

Just run nipyapi.canvas.get_process_group(identifier=pg_name, identifier_type='name') when there is at least one processor in the RUN_ONCE state. For example, run once the ExecuteStreamCommand that runs the pythons script where it gathers the processor that runs this script.

Urgency

It is not blocking so much in automated flows, but it blocks debugging sometimes.

[ottobackwards]

That PR was closed due to inactivity, not merged. I'll try to find the right thing, but regardless, that enum is generated by the swagger / openapi, so I think the fix is a new release of nipyapi against the new version

[ottobackwards]

https://github.com/apache/nifi/commit/aa726040c567818cef097fe10905badc82996bb2 is the one

[ottobackwards]

The fix version is 1.14.0

[ottobackwards]

So, nipyapi should have picked this up based on the README version. @Chaffelson ? Any ideas?

[ottobackwards]

@ihor-sokoliuk-exa I have created https://issues.apache.org/jira/browse/NIFI-10207 and will be posting a fix pr. This won't be picked up in nipyapi until there is another nifi release followed by an updated nipyapi release. I don't know if there is any workaround

[ihor-sokoliuk-exa]

Thank you, @ottobackwards Will wait for new nifi release

[Chaffelson]

IFF the NiFi API accepts the payload, then we can manually patch the NiPy client to generate it as a workaround. IFF NiFi rejects the payload nothing can be done until a new release.

[greyp9]

NiFi accepts the payload.

[ottobackwards]

The PR has landed, this should be in the next release

[ihor-sokoliuk-exa]

Hi @ottobackwards @Chaffelson Great news! NiFi 1.17.0 version was released a few days ago. And your changes are in there already.

https://github.com/apache/nifi/blob/rel/nifi-1.17.0/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-client-dto/src/main/java/org/apache/nifi/web/api/entity/ProcessorRunStatusEntity.java#L43

Do you have any plans for a new nipyapi release?

[ottobackwards]

That is up to @Chaffelson,

[Chaffelson]

Investigating now