We were preparing and saving the next configuration in celery's on_success() method but the method suppresses any exceptions that occur within it which is why there were no error messages in any of the log files for some failures. I moved that code into a separate helper method.
TaskUtil.get_task_status assigns an overall status of the progress of the celery tasks and it's set to SUCCESS if all subtasks succeed. In rare cases we returned SUCCESS incorrectly because subtasks are not added to the TaskMeta table immediately and thus we were considering only a subset of them.
Also added more log messages and other minor things
Changes: