search

hermannpencole / nifi-config

Automatically deploy and configure Template on Nifi
Apache License 2.0
55 stars 39 forks source link

Undeploy fails when there is an external connection to the process group #37

Open ennui93 opened 6 years ago

ennui93 commented 6 years ago

It is not possible to run nifi-config in its undeploy mode when the process group in which a template is being removed is connected to other process groups. Tested on release nifi-deploy-config-1.1.30 & 17411a859ddf450df32a1cd5a26503aaf6f129d9. There are two cases:

Incoming connection

2018-01-09 09:41:08 | INFO  | [pool-29-thread-1] c.g.h.n.c.s.ProcessGroupService:303 - Cannot delete Process Group because Input Port 3d3a458f-7365-1c62-aa8f-864fe0bf3085 has at least one incoming connection from a component outside of the Process Group. Delete this connection first.
Exception in thread "main" com.github.hermannpencole.nifi.config.model.ConfigException: com.github.hermannpencole.nifi.swagger.ApiException: Conflict
    at com.github.hermannpencole.nifi.config.utils.FunctionUtils.runTimeout(FunctionUtils.java:40)
    at com.github.hermannpencole.nifi.config.utils.FunctionUtils.runWhile(FunctionUtils.java:65)
    at com.github.hermannpencole.nifi.config.service.ProcessGroupService.delete(ProcessGroupService.java:296)
    at com.github.hermannpencole.nifi.config.service.TemplateService.undeploy(TemplateService.java:125)
    at com.github.hermannpencole.nifi.config.Main.main(Main.java:149)
Caused by: com.github.hermannpencole.nifi.swagger.ApiException: Conflict
    at com.github.hermannpencole.nifi.swagger.ApiClient.handleResponse(ApiClient.java:1104)
    at com.github.hermannpencole.nifi.swagger.ApiClient.execute(ApiClient.java:1027)
    at com.github.hermannpencole.nifi.swagger.client.ProcessGroupsApi.removeProcessGroupWithHttpInfo(ProcessGroupsApi.java:2882)
    at com.github.hermannpencole.nifi.swagger.client.ProcessGroupsApi.removeProcessGroup(ProcessGroupsApi.java:2866)
    at com.github.hermannpencole.nifi.config.service.ProcessGroupService.lambda$delete$11(ProcessGroupService.java:301)
    at com.github.hermannpencole.nifi.config.utils.FunctionUtils.runWhile(FunctionUtils.java:54)
    at com.github.hermannpencole.nifi.config.utils.FunctionUtils.lambda$runWhile$0(FunctionUtils.java:65)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Outgoing connection

2018-01-09 09:36:16 | INFO  | [pool-12-thread-1] c.g.h.n.c.s.ProcessGroupService:303 - Cannot delete Process Group because Output Port 3ed34099-3a0d-1d93-06ad-0a8d19f7f805 has at least one outgoing connection to a component outside of the Process Group. Delete this connection first.
Exception in thread "main" com.github.hermannpencole.nifi.config.model.ConfigException: com.github.hermannpencole.nifi.swagger.ApiException: Conflict
    at com.github.hermannpencole.nifi.config.utils.FunctionUtils.runTimeout(FunctionUtils.java:40)
    at com.github.hermannpencole.nifi.config.utils.FunctionUtils.runWhile(FunctionUtils.java:65)
    at com.github.hermannpencole.nifi.config.service.ProcessGroupService.delete(ProcessGroupService.java:296)
    at com.github.hermannpencole.nifi.config.service.TemplateService.undeploy(TemplateService.java:125)
    at com.github.hermannpencole.nifi.config.Main.main(Main.java:149)
Caused by: com.github.hermannpencole.nifi.swagger.ApiException: Conflict
    at com.github.hermannpencole.nifi.swagger.ApiClient.handleResponse(ApiClient.java:1104)
    at com.github.hermannpencole.nifi.swagger.ApiClient.execute(ApiClient.java:1027)
    at com.github.hermannpencole.nifi.swagger.client.ProcessGroupsApi.removeProcessGroupWithHttpInfo(ProcessGroupsApi.java:2882)
    at com.github.hermannpencole.nifi.swagger.client.ProcessGroupsApi.removeProcessGroup(ProcessGroupsApi.java:2866)
    at com.github.hermannpencole.nifi.config.service.ProcessGroupService.lambda$delete$11(ProcessGroupService.java:301)
    at com.github.hermannpencole.nifi.config.utils.FunctionUtils.runWhile(FunctionUtils.java:54)
    at com.github.hermannpencole.nifi.config.utils.FunctionUtils.lambda$runWhile$0(FunctionUtils.java:65)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
ennui93 commented 6 years ago

Suggested enhancement would be to remove the incoming/outgoing connections to a given process group on undeploy, a behavior which should probably be configurable via a command-line option so as not to break cases where users may choose to deploy multiple templates into the same process group.

hermannpencole commented 6 years ago

Thanks, Are you in a hurry for correction ?

ennui93 commented 6 years ago

I authored a stopgap solution. However, it does not accept any command-line parameter to toggle the behavior as discussed above. Additionally, it stops all components connected to a target process group, which may not be desirable for some users' scenarios since the connected components are not re-started again after the process group is removed. Lastly, the connected components to be stopped are limited to processors and local ports, and thus this will [potentially] not work for remote ports.

See: https://github.com/hermannpencole/nifi-config/compare/master...ennui93:issue-37

I have tested this against NiFi 1.0.1; but have not observed the behavior in the case of live data flowing through the system. There could be a bug whereby FlowFiles end up stuck in the queue of a connection to be deleted, and thus the undeploy operation fails when attempting to remove the non-empty connection. An improvement would be to stop connected components prior to stopping elements of the template itself.