[adoption]Add a new role for pre adoption validation

[gibizer]

The new edpm_pre_adoption_validation role is intended to be used as dataplane service in a separate deployment during adoption that runs before the main deployment of the adoption process. This role does not change anything on the EDPM nodes just reads their current (pre-adoption) configuration and check if the intended new (post-adoption) configuration will not cause any trouble.

The first check introduced by this commit ensures that the openstack service hostname will not change during adoption on the EDPM nodes. Such change would cause adoption failure in a later stage, after the point of no return, as nova-compute will refuse to start up if it detects the host rename. Additionally neutron services might fail too / or create database corruption due to the host rename.

Implements: OSPRH-5713

[openshift-ci[bot]]

Skipping CI for Draft Pull Request. If you want CI signal for your change, please convert it to an actual PR. You can still manually trigger a test run with /test all

[softwarefactory-project-zuul[bot]]

Build failed (check pipeline). Post recheck (without leading slash) to rerun all jobs. Make sure the failure cause has been resolved before you rerun jobs.

https://review.rdoproject.org/zuul/buildset/4ed0890443374b7d9c8195a235be0d2b

:heavy_check_mark: openstack-k8s-operators-content-provider SUCCESS in 2h 06m 19s :heavy_check_mark: podified-multinode-edpm-deployment-crc SUCCESS in 1h 46m 58s :x: cifmw-crc-podified-edpm-baremetal RETRY_LIMIT in 16m 53s :heavy_check_mark: edpm-ansible-molecule-edpm_bootstrap SUCCESS in 5m 43s :heavy_check_mark: edpm-ansible-molecule-edpm_podman SUCCESS in 4m 41s :heavy_check_mark: edpm-ansible-molecule-edpm_module_load SUCCESS in 4m 03s :heavy_check_mark: edpm-ansible-molecule-edpm_kernel SUCCESS in 9m 44s :heavy_check_mark: edpm-ansible-molecule-edpm_libvirt SUCCESS in 8m 01s :heavy_check_mark: edpm-ansible-molecule-edpm_nova SUCCESS in 8m 11s :heavy_check_mark: edpm-ansible-molecule-edpm_frr SUCCESS in 5m 32s :heavy_check_mark: edpm-ansible-molecule-edpm_iscsid SUCCESS in 5m 05s :heavy_check_mark: edpm-ansible-molecule-edpm_ovn_bgp_agent SUCCESS in 6m 00s :heavy_check_mark: edpm-ansible-molecule-edpm_ovs SUCCESS in 4m 32s

[softwarefactory-project-zuul[bot]]

Build failed (check pipeline). Post recheck (without leading slash) to rerun all jobs. Make sure the failure cause has been resolved before you rerun jobs.

https://review.rdoproject.org/zuul/buildset/6fb9431c4f874341896778b7f3f4bab7

:heavy_check_mark: openstack-k8s-operators-content-provider SUCCESS in 2h 36m 30s :x: podified-multinode-edpm-deployment-crc FAILURE in 1h 34m 52s :heavy_check_mark: cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 46m 50s :heavy_check_mark: edpm-ansible-molecule-edpm_bootstrap SUCCESS in 7m 12s :heavy_check_mark: edpm-ansible-molecule-edpm_podman SUCCESS in 5m 41s :heavy_check_mark: edpm-ansible-molecule-edpm_module_load SUCCESS in 4m 46s :heavy_check_mark: edpm-ansible-molecule-edpm_kernel SUCCESS in 12m 14s :heavy_check_mark: edpm-ansible-molecule-edpm_libvirt SUCCESS in 10m 30s :heavy_check_mark: edpm-ansible-molecule-edpm_nova SUCCESS in 9m 20s :heavy_check_mark: edpm-ansible-molecule-edpm_frr SUCCESS in 6m 52s :heavy_check_mark: edpm-ansible-molecule-edpm_iscsid SUCCESS in 3m 56s :heavy_check_mark: edpm-ansible-molecule-edpm_ovn_bgp_agent SUCCESS in 6m 22s :heavy_check_mark: edpm-ansible-molecule-edpm_ovs SUCCESS in 4m 50s

[gibizer]

The usage of the new role can be seen in https://github.com/openstack-k8s-operators/data-plane-adoption/pull/374

[softwarefactory-project-zuul[bot]]

Build failed (check pipeline). Post recheck (without leading slash) to rerun all jobs. Make sure the failure cause has been resolved before you rerun jobs.

https://review.rdoproject.org/zuul/buildset/bfecef84cf404c4897d1db994cffb207

:heavy_check_mark: openstack-k8s-operators-content-provider SUCCESS in 2h 54m 15s :x: podified-multinode-edpm-deployment-crc RETRY_LIMIT in 19m 21s :heavy_check_mark: cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 55m 12s :heavy_check_mark: edpm-ansible-molecule-edpm_bootstrap SUCCESS in 6m 12s :heavy_check_mark: edpm-ansible-molecule-edpm_podman SUCCESS in 5m 35s :heavy_check_mark: edpm-ansible-molecule-edpm_module_load SUCCESS in 4m 23s :heavy_check_mark: edpm-ansible-molecule-edpm_kernel SUCCESS in 8m 42s :heavy_check_mark: edpm-ansible-molecule-edpm_libvirt SUCCESS in 8m 54s :heavy_check_mark: edpm-ansible-molecule-edpm_nova SUCCESS in 8m 01s :heavy_check_mark: edpm-ansible-molecule-edpm_frr SUCCESS in 7m 14s :heavy_check_mark: edpm-ansible-molecule-edpm_iscsid SUCCESS in 3m 49s :heavy_check_mark: edpm-ansible-molecule-edpm_ovn_bgp_agent SUCCESS in 6m 50s :heavy_check_mark: edpm-ansible-molecule-edpm_ovs SUCCESS in 4m 26s

[gibizer]

recheck

[softwarefactory-project-zuul[bot]]

Build failed (check pipeline). Post recheck (without leading slash) to rerun all jobs. Make sure the failure cause has been resolved before you rerun jobs.

https://review.rdoproject.org/zuul/buildset/46aa2dfb72154888acf0dc9c822ed863

:x: openstack-k8s-operators-content-provider FAILURE in 8m 03s :warning: podified-multinode-edpm-deployment-crc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider :warning: cifmw-crc-podified-edpm-baremetal SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider :heavy_check_mark: edpm-ansible-molecule-edpm_bootstrap SUCCESS in 6m 39s :heavy_check_mark: edpm-ansible-molecule-edpm_podman SUCCESS in 5m 40s :heavy_check_mark: edpm-ansible-molecule-edpm_module_load SUCCESS in 4m 54s :heavy_check_mark: edpm-ansible-molecule-edpm_kernel SUCCESS in 12m 12s :heavy_check_mark: edpm-ansible-molecule-edpm_libvirt SUCCESS in 9m 01s :heavy_check_mark: edpm-ansible-molecule-edpm_nova SUCCESS in 10m 35s :heavy_check_mark: edpm-ansible-molecule-edpm_frr SUCCESS in 7m 05s :heavy_check_mark: edpm-ansible-molecule-edpm_iscsid SUCCESS in 4m 53s :heavy_check_mark: edpm-ansible-molecule-edpm_ovn_bgp_agent SUCCESS in 7m 15s :heavy_check_mark: edpm-ansible-molecule-edpm_ovs SUCCESS in 5m 24s

[SeanMooney]

check-rdo content provider failed

[softwarefactory-project-zuul[bot]]

Build failed (check pipeline). Post recheck (without leading slash) to rerun all jobs. Make sure the failure cause has been resolved before you rerun jobs.

https://review.rdoproject.org/zuul/buildset/b609b90e919244558e0729eff51bb14d

:x: openstack-k8s-operators-content-provider FAILURE in 8m 07s :warning: podified-multinode-edpm-deployment-crc SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider :warning: cifmw-crc-podified-edpm-baremetal SKIPPED Skipped due to failed job openstack-k8s-operators-content-provider :heavy_check_mark: edpm-ansible-molecule-edpm_bootstrap SUCCESS in 5m 32s :heavy_check_mark: edpm-ansible-molecule-edpm_podman SUCCESS in 5m 00s :heavy_check_mark: edpm-ansible-molecule-edpm_module_load SUCCESS in 4m 58s :heavy_check_mark: edpm-ansible-molecule-edpm_kernel SUCCESS in 11m 38s :heavy_check_mark: edpm-ansible-molecule-edpm_libvirt SUCCESS in 9m 32s :heavy_check_mark: edpm-ansible-molecule-edpm_nova SUCCESS in 10m 03s :heavy_check_mark: edpm-ansible-molecule-edpm_frr SUCCESS in 6m 59s :heavy_check_mark: edpm-ansible-molecule-edpm_iscsid SUCCESS in 4m 34s :heavy_check_mark: edpm-ansible-molecule-edpm_ovn_bgp_agent SUCCESS in 7m 34s :heavy_check_mark: edpm-ansible-molecule-edpm_ovs SUCCESS in 5m 07s

[SeanMooney]

the content provider failed again but im not going to recheck it until monday

[bogdando]

recheck

[bogdando]

lgtm

[softwarefactory-project-zuul[bot]]

Build failed (check pipeline). Post recheck (without leading slash) to rerun all jobs. Make sure the failure cause has been resolved before you rerun jobs.

https://review.rdoproject.org/zuul/buildset/12617c64143a4fe5870cc2e56390acb8

:heavy_check_mark: openstack-k8s-operators-content-provider SUCCESS in 2h 08m 24s :x: podified-multinode-edpm-deployment-crc FAILURE in 1h 18m 38s :heavy_check_mark: cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 46m 25s :heavy_check_mark: edpm-ansible-molecule-edpm_bootstrap SUCCESS in 6m 15s :heavy_check_mark: edpm-ansible-molecule-edpm_podman SUCCESS in 4m 41s :heavy_check_mark: edpm-ansible-molecule-edpm_module_load SUCCESS in 4m 17s :heavy_check_mark: edpm-ansible-molecule-edpm_kernel SUCCESS in 8m 31s :heavy_check_mark: edpm-ansible-molecule-edpm_libvirt SUCCESS in 7m 54s :heavy_check_mark: edpm-ansible-molecule-edpm_nova SUCCESS in 8m 08s :heavy_check_mark: edpm-ansible-molecule-edpm_frr SUCCESS in 6m 02s :heavy_check_mark: edpm-ansible-molecule-edpm_iscsid SUCCESS in 3m 57s :heavy_check_mark: edpm-ansible-molecule-edpm_ovn_bgp_agent SUCCESS in 6m 36s :heavy_check_mark: edpm-ansible-molecule-edpm_ovs SUCCESS in 4m 57s

[jistr]

lgtm

[softwarefactory-project-zuul[bot]]

Build failed (check pipeline). Post recheck (without leading slash) to rerun all jobs. Make sure the failure cause has been resolved before you rerun jobs.

https://review.rdoproject.org/zuul/buildset/0af26a3aa76e495abf924551d0c36778

:heavy_check_mark: openstack-k8s-operators-content-provider SUCCESS in 38m 01s :x: podified-multinode-edpm-deployment-crc RETRY_LIMIT in 19m 04s :x: cifmw-crc-podified-edpm-baremetal RETRY_LIMIT in 16m 04s :heavy_check_mark: edpm-ansible-molecule-edpm_bootstrap SUCCESS in 6m 20s :heavy_check_mark: edpm-ansible-molecule-edpm_podman SUCCESS in 4m 38s :heavy_check_mark: edpm-ansible-molecule-edpm_module_load SUCCESS in 4m 58s :heavy_check_mark: edpm-ansible-molecule-edpm_kernel SUCCESS in 12m 05s :heavy_check_mark: edpm-ansible-molecule-edpm_libvirt SUCCESS in 9m 15s :heavy_check_mark: edpm-ansible-molecule-edpm_nova SUCCESS in 10m 26s :heavy_check_mark: edpm-ansible-molecule-edpm_frr SUCCESS in 6m 53s :heavy_check_mark: edpm-ansible-molecule-edpm_iscsid SUCCESS in 4m 39s :heavy_check_mark: edpm-ansible-molecule-edpm_ovn_bgp_agent SUCCESS in 6m 55s :heavy_check_mark: edpm-ansible-molecule-edpm_ovs SUCCESS in 5m 06s

[gibizer]

recheck

ERROR! Error when getting available collection versions for ansible.utils from default (https://galaxy.ansible.com/api/) (HTTP Code: 504, Message: Gateway Time-out Code: Unknown)

[softwarefactory-project-zuul[bot]]

Build failed (check pipeline). Post recheck (without leading slash) to rerun all jobs. Make sure the failure cause has been resolved before you rerun jobs.

https://review.rdoproject.org/zuul/buildset/8f0f4b00ec6c4e19bd2073fd2c4a1a5a

:heavy_check_mark: openstack-k8s-operators-content-provider SUCCESS in 41m 31s :x: podified-multinode-edpm-deployment-crc RETRY_LIMIT in 9m 10s :x: cifmw-crc-podified-edpm-baremetal RETRY_LIMIT in 12m 39s :heavy_check_mark: edpm-ansible-molecule-edpm_bootstrap SUCCESS in 6m 01s :heavy_check_mark: edpm-ansible-molecule-edpm_podman SUCCESS in 4m 44s :heavy_check_mark: edpm-ansible-molecule-edpm_module_load SUCCESS in 4m 04s :heavy_check_mark: edpm-ansible-molecule-edpm_kernel SUCCESS in 9m 29s :heavy_check_mark: edpm-ansible-molecule-edpm_libvirt SUCCESS in 7m 52s :heavy_check_mark: edpm-ansible-molecule-edpm_nova SUCCESS in 8m 11s :heavy_check_mark: edpm-ansible-molecule-edpm_frr SUCCESS in 5m 57s :heavy_check_mark: edpm-ansible-molecule-edpm_iscsid SUCCESS in 4m 11s :heavy_check_mark: edpm-ansible-molecule-edpm_ovn_bgp_agent SUCCESS in 6m 31s :heavy_check_mark: edpm-ansible-molecule-edpm_ovs SUCCESS in 4m 47s

[gibizer]

recheck hm, do we have a broken CI in general?

ERROR! couldn't resolve module/action 'cifmw.general.ci_script'. This often indicates a misspelling, missing collection, or incorrect module path.

The error appears to be in '/home/zuul/src/github.com/openstack-k8s-operators/ci-framework/roles/os_must_gather/tasks/main.yml': line 39, column 3, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

- name: Check for oc command
  ^ here

[softwarefactory-project-zuul[bot]]

Build failed (check pipeline). Post recheck (without leading slash) to rerun all jobs. Make sure the failure cause has been resolved before you rerun jobs.

https://review.rdoproject.org/zuul/buildset/124e45e08e4f4eab9af7b4e55452b81f

:heavy_check_mark: openstack-k8s-operators-content-provider SUCCESS in 2h 15m 36s :x: podified-multinode-edpm-deployment-crc RETRY_LIMIT in 11m 02s :heavy_check_mark: cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 52m 06s :heavy_check_mark: edpm-ansible-molecule-edpm_bootstrap SUCCESS in 5m 41s :heavy_check_mark: edpm-ansible-molecule-edpm_podman SUCCESS in 4m 44s :heavy_check_mark: edpm-ansible-molecule-edpm_module_load SUCCESS in 4m 07s :heavy_check_mark: edpm-ansible-molecule-edpm_kernel SUCCESS in 9m 48s :heavy_check_mark: edpm-ansible-molecule-edpm_libvirt SUCCESS in 7m 45s :heavy_check_mark: edpm-ansible-molecule-edpm_nova SUCCESS in 8m 21s :heavy_check_mark: edpm-ansible-molecule-edpm_frr SUCCESS in 5m 49s :heavy_check_mark: edpm-ansible-molecule-edpm_iscsid SUCCESS in 4m 08s :heavy_check_mark: edpm-ansible-molecule-edpm_ovn_bgp_agent SUCCESS in 6m 28s :heavy_check_mark: edpm-ansible-molecule-edpm_ovs SUCCESS in 4m 42s

[softwarefactory-project-zuul[bot]]

Build failed (check pipeline). Post recheck (without leading slash) to rerun all jobs. Make sure the failure cause has been resolved before you rerun jobs.

https://review.rdoproject.org/zuul/buildset/eedba8bdb9ed4e8eb049c8c2d09dfeea

:heavy_check_mark: openstack-k8s-operators-content-provider SUCCESS in 2h 03m 51s :x: podified-multinode-edpm-deployment-crc FAILURE in 1h 26m 51s :heavy_check_mark: cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 44m 53s :heavy_check_mark: edpm-ansible-molecule-edpm_bootstrap SUCCESS in 6m 06s :heavy_check_mark: edpm-ansible-molecule-edpm_podman SUCCESS in 5m 08s :heavy_check_mark: edpm-ansible-molecule-edpm_module_load SUCCESS in 4m 34s :heavy_check_mark: edpm-ansible-molecule-edpm_kernel SUCCESS in 10m 00s :heavy_check_mark: edpm-ansible-molecule-edpm_libvirt SUCCESS in 8m 18s :heavy_check_mark: edpm-ansible-molecule-edpm_nova SUCCESS in 8m 46s :heavy_check_mark: edpm-ansible-molecule-edpm_frr SUCCESS in 6m 24s :heavy_check_mark: edpm-ansible-molecule-edpm_iscsid SUCCESS in 4m 18s :heavy_check_mark: edpm-ansible-molecule-edpm_ovn_bgp_agent SUCCESS in 6m 45s :heavy_check_mark: edpm-ansible-molecule-edpm_ovs SUCCESS in 5m 09s

[gibizer]

recheck retry limit

[gibizer]

/retest-all

[softwarefactory-project-zuul[bot]]

Build failed (check pipeline). Post recheck (without leading slash) to rerun all jobs. Make sure the failure cause has been resolved before you rerun jobs.

https://review.rdoproject.org/zuul/buildset/67f289118eb64419ad67681a829bffec

:heavy_check_mark: openstack-k8s-operators-content-provider SUCCESS in 2h 51m 43s :x: podified-multinode-edpm-deployment-crc FAILURE in 1h 16m 23s :heavy_check_mark: cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 41m 14s :heavy_check_mark: edpm-ansible-molecule-edpm_bootstrap SUCCESS in 6m 07s :heavy_check_mark: edpm-ansible-molecule-edpm_podman SUCCESS in 4m 54s :heavy_check_mark: edpm-ansible-molecule-edpm_module_load SUCCESS in 5m 00s :heavy_check_mark: edpm-ansible-molecule-edpm_kernel SUCCESS in 14m 06s :heavy_check_mark: edpm-ansible-molecule-edpm_libvirt SUCCESS in 8m 34s :heavy_check_mark: edpm-ansible-molecule-edpm_nova SUCCESS in 8m 32s :heavy_check_mark: edpm-ansible-molecule-edpm_frr SUCCESS in 6m 12s :heavy_check_mark: edpm-ansible-molecule-edpm_iscsid SUCCESS in 4m 06s :heavy_check_mark: edpm-ansible-molecule-edpm_ovn_bgp_agent SUCCESS in 6m 28s :heavy_check_mark: edpm-ansible-molecule-edpm_ovs SUCCESS in 4m 53s

[gibizer]

This worked in the adoption job finally https://logserver.rdoproject.org/74/374/aaca4659383d76fa31b0b217f71dcc9fa4d7e79f/github-check/data-plane-adoption-osp-17-to-extracted-crc-minimal-no-ceph/3fee568/controller/ci-framework-data/logs/openstack-k8s-operators-openstack-must-gather/namespaces/openstack/pods/pre-adoption-validation-openstack-pre-adoption-openstack-rfbxh/logs/openstackansibleee.log

[gibizer]

recheck

[gibizer]

/unhold We saw this working in the adoption job so this is good to go

[openshift-ci[bot]]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: fao89, gibizer, SeanMooney

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[OWNERS](https://github.com/openstack-k8s-operators/edpm-ansible/blob/main/OWNERS)~~ [SeanMooney,fao89,gibizer] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment

[softwarefactory-project-zuul[bot]]

Build failed (check pipeline). Post recheck (without leading slash) to rerun all jobs. Make sure the failure cause has been resolved before you rerun jobs.

https://review.rdoproject.org/zuul/buildset/0ec073ee06394288bcfe516cc380dc50

:heavy_check_mark: openstack-k8s-operators-content-provider SUCCESS in 2h 51m 15s :x: podified-multinode-edpm-deployment-crc FAILURE in 59m 16s :heavy_check_mark: cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 45m 46s :heavy_check_mark: edpm-ansible-molecule-edpm_bootstrap SUCCESS in 6m 49s :heavy_check_mark: edpm-ansible-molecule-edpm_podman SUCCESS in 4m 38s :heavy_check_mark: edpm-ansible-molecule-edpm_module_load SUCCESS in 4m 51s :heavy_check_mark: edpm-ansible-molecule-edpm_kernel SUCCESS in 9m 13s :heavy_check_mark: edpm-ansible-molecule-edpm_libvirt SUCCESS in 8m 39s :heavy_check_mark: edpm-ansible-molecule-edpm_nova SUCCESS in 8m 03s :heavy_check_mark: edpm-ansible-molecule-edpm_frr SUCCESS in 5m 54s :heavy_check_mark: edpm-ansible-molecule-edpm_iscsid SUCCESS in 4m 10s :heavy_check_mark: edpm-ansible-molecule-edpm_ovn_bgp_agent SUCCESS in 6m 22s :heavy_check_mark: edpm-ansible-molecule-edpm_ovs SUCCESS in 4m 36s

[gibizer]

recheck

2024-04-11 07:36:59.590514 | controller | TASK [set_openstack_containers : Get operator environment variable after update output_dir={{ cifmw_set_openstack_containers_basedir }}/artifacts, script=oc set env {{ operator_pod_name }} -n {{ cifmw_install_yamls_defaults['OPERATOR_NAMESPACE'] }} --list > operator_env.txt, chdir={{ cifmw_set_openstack_containers_basedir }}/artifacts, creates=operator_env.txt] ***
2024-04-11 07:36:59.590526 | controller | Thursday 11 April 2024  07:36:59 -0400 (0:00:00.030)       0:11:56.349 ********
2024-04-11 07:36:59.927582 | controller | Follow script's output here: /home/zuul/ci-framework-data/logs/ci_script_010_get_operator_environment.log
2024-04-11 07:36:59.927635 | controller | An exception occurred during task execution. To see the full traceback, use -vvv. The error was: NoneType: None
2024-04-11 07:36:59.927643 | controller | fatal: [localhost]: FAILED! => {"changed": true, "msg": "non-zero return code", "rc": 1, "stderr": "", "stderr_lines": [], "stdout": "~/ci-framework-data/artifacts ~/ci-framework-data/artifacts\nError from server (NotFound): pods \"openstack-ansibleee-operator-controller-manager-77b56db5b-cd647\" not found\n", "stdout_lines": ["~/ci-framework-data/artifacts ~/ci-framework-data/artifacts", "Error from server (NotFound): pods \"openstack-ansibleee-operator-controller-manager-77b56db5b-cd647\" not found"]}