Wrong order while installing errata and forcing reboot for virtual machines

[AnonJo]

Describe the bug When installing errata via katprep_maintenance and using the flag --reboot-systems for virtual machines running on a VMware enviroment, the installation task is started but the virtual machines are directly restarted without waiting for the installation task to finish.

To Reproduce Steps to reproduce the behavior:

1. Enter command katprep_maintenance --reboot-systems -C containter .auth errata-report.json execute
2. See error Check VMware environment for the reboot task and check that the tasks in the Satellite systems are still running and being executed after the reboot.

Expected behavior Katprep should wait for the installation of the errata and eventually the packets to succeed and then do a reboot. If the tasks fail it should not reboot the virtual machine.

Environment (please complete the following information):

• OS version: [e.g. CentOS 7.4] Red Hat Enterprise Linux Server release 7.5 (Maipo)
• Red Hat Satellite or Foreman/Katello version [e.g. 1.17.1/3.6.0] Red Hat Satellite 6.3.2

Debug-Output:

DEBUG:ForemanAPIClient:API version 2 found
DEBUG:katprep_maintenance:Patching host 'host.local'...
DEBUG:ForemanAPIClient:GET request to https://satelliteserver.local/api/v2/hosts (payload: )
DEBUG:ForemanAPIClient:host host.local seems to have ID #13
DEBUG:ForemanAPIClient:PUT request to https://satelliteserver.local/api/v2/hosts/13/errata/apply (payload: {"errata_ids": ["RHSA-2018:2384", "RHEA-2018:2299", "RHSA-2018:2285", "RHSA-2018:2242"]})
DEBUG:ForemanAPIClient:GET request to https://satelliteserver.local/api/v2/hosts (payload: )
DEBUG:ForemanAPIClient:host host.local seems to have ID #13
DEBUG:ForemanAPIClient:PUT request to https://satelliteserver.local/api/v2/hosts/13/power (payload: {"power_action": "soft"})```

[stdevel]

Hello @AnonJo,

I'm going to reproduce that error. Just one thing I have in mind regarding Red Hat Satellite 6.x: are the affected virtual machines linked with the VM objects? You can check this by having a look at the hosts in the "Hosts" view. If they are linked correctly, you should see the appropriate VM power state (e.g. green icon for powered-on VM). If you just see a red icon and a hint like "Power actions are not available" then the host/VM objects are not linked correctly.

[AnonJo]

Yeah, they are linked. If not Red Hat Satellite couldn't reboot it.

[stdevel]

Thanks for your feedback! I will investigate and update on this.

[stdevel]

Confirmed on Foreman 1.19 / Katello 3.8. Really looks like Foreman does not queue the second API call (for triggering the reboot) to be run after the first API call (updating the system).

Just opened a thread in the developer board to discuss this: https://community.theforeman.org/t/parallel-foreman-katello-api-calls-are-not-ordered-do-not-have-dependencies/11266

The workaround would be skipping the -r / --reboot-systems parameter or supplying the -R / --reboot. After patching your systems you could suplly the -r / --reboot-systmes parameter to trigger the reboot.

Will update this when I have feedback from upstream.

[stdevel]

The official feedback is a recommendation to implement a mechanism within katprep in order to detect non-finished tasks before executing the call: https://community.theforeman.org/t/parallel-foreman-katello-api-calls-are-not-ordered-do-not-have-dependencies/11266/3

Currently, I see the following possibilities:

1. implement a dedicated katprep_maintenance sub-command, e.g. reboot as we do not have a process running in the backup polling for the task status
2. run the reboot task as part of the cleanup sub-command
3. make the execute stage poll for task status before triggering reboot

Option 1 would be the easiest solution, but it also adds complexity and is contrary to the plan to simplify katprep - see issue #101. Option 2 is also not the best idea as not everybody wants to remove snapshots/downtimes right after rebooting (e.g. because of changes needed to be tested) Option 3 is a bigger change but it seems to be worth it. An idea would be to implement an updating line at the bottom, e.g. Updating x hosts | Errata updates: x/y completed | Package updates: x/y completed | Reboot x/y completed

[stdevel]

When implementing Option 3, this is also related to #87. So implementing polling is necessary.

Would be a great idea to have fancy progress bars with the progressbar2 Python module.

Investigating in branch feature-polling-issue87_93.