search

hermannpencole / nifi-config

Automatically deploy and configure Template on Nifi
Apache License 2.0
55 stars 39 forks source link

Not possible to undeploy the template when a queue is not empty #7

Closed bbossola-tesco closed 7 years ago

bbossola-tesco commented 7 years ago

If any queue in the processor is not empty at the time of the command then the processor would not be deployed because of a conflict.

Stacktrace from the client:

2017-07-27 11:08:05 | ERROR | [main] c.g.h.n.c.Main:121 - Conflict com.github.hermannpencole.nifi.swagger.ApiException: Conflict at com.github.hermannpencole.nifi.swagger.ApiClient.handleResponse(ApiClient.java:1066) at com.github.hermannpencole.nifi.swagger.ApiClient.execute(ApiClient.java:989) at com.github.hermannpencole.nifi.swagger.client.ProcessGroupsApi.removeProcessGroupWithHttpInfo(ProcessGroupsApi.java:2882) at com.github.hermannpencole.nifi.swagger.client.ProcessGroupsApi.removeProcessGroup(ProcessGroupsApi.java:2866) at com.github.hermannpencole.nifi.config.service.TemplateService.undeploy(TemplateService.java:98) at com.github.hermannpencole.nifi.config.Main.main(Main.java:116) Exception in thread "main" com.github.hermannpencole.nifi.config.model.ConfigException: Conflict: Queue not empty for 3876beb9-2781-4a19-1321-c98080218d05 at com.github.hermannpencole.nifi.config.Main.main(Main.java:122) Caused by: com.github.hermannpencole.nifi.swagger.ApiException: Conflict at com.github.hermannpencole.nifi.swagger.ApiClient.handleResponse(ApiClient.java:1066) at com.github.hermannpencole.nifi.swagger.ApiClient.execute(ApiClient.java:989) at com.github.hermannpencole.nifi.swagger.client.ProcessGroupsApi.removeProcessGroupWithHttpInfo(ProcessGroupsApi.java:2882) at com.github.hermannpencole.nifi.swagger.client.ProcessGroupsApi.removeProcessGroup(ProcessGroupsApi.java:2866) at com.github.hermannpencole.nifi.config.service.TemplateService.undeploy(TemplateService.java:98) at com.github.hermannpencole.nifi.config.Main.main(Main.java:116)

hermannpencole commented 7 years ago

Yes it's normal, that warns you that a queue is not empty. If you empty queue, you will lose the data. I can add option --force for undeploy to force undeploy and empty queue but it's dangerous.

bbossola-tesco commented 7 years ago

Okay thanks, I think it's fair. I think then this opens another issue, in the sense, how do I know that a process group has been completely "flushed" and it's ready for redeployment? Ideally an option as "flush+deploy" with a timeout, or something like that.... dunno how that could be structured tough, in terms of "how". I am okay for the deployment task to fail if it takes too much, but it should be in a position to succeed if the conditions are good (i.e. the queues can be emptied timely). I do not want to have a human being to access NiFi. manually shutting down component by component, making sure that the queues are empty, and then click the "deploy" button on the CI.

I can also code properly with TDD, so I could PR to this project but I'd like to have your guidance on the matter.

hermannpencole commented 7 years ago

Hello, The idea is to stop only processors that do not consume queue at first. And after, wait the queues are empty. And only for finish, stop the group processor.

hermannpencole commented 7 years ago

Release 1.1.6 is created and have the correction. If the correction is ok for you, close the issue. If not say what is wrong. thank you for your collaboration

bbossola-tesco commented 7 years ago

Yup, sounds sensible, everything works. Happy to close the issue (and apologies for the delay but I was on holiday)