Ansible Networking Workshop error

JamesG-Projects commented 5 months ago

Problem Summary

Workshop Name: Ansible Automation Platform 2 Networking Automation Workshop

Workshop URL: http://m4j6m.example.opentlc.com/

Description:

The workshop provides 3 links -

VS Code access
Automation Controller
SSH access

In some workshop user instances, when trying to access the VS Code URL, you get the message "This site cannot provide a secure connection. student3.m4j6m.example.opentlc.com sent an invalid response. ERR_SSL_PROTOCOLERROR". To circumvent this error you can change "https" to "http", and the login page will load. However_, if you then input the password, it says the password is incorrect and you cannot access the VS Code browser link.

In the event that a workshop user can successfully access the https link, this issue does not happen. Reproducing the issue is unreliable. Some instances work and some do not.

This is not an issue for the Automation Controller or the SSH access, only the VS Code link.

Issue Type

Bug

Extra vars file

N/A

Ansible Playbook Output

N/A

Ansible Version

[student@ansible-1 ~]$ ansible --version ansible [core 2.15.5] config file = /etc/ansible/ansible.cfg configured module search path = ['/home/student/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules'] ansible python module location = /usr/local/lib/python3.9/site-packages/ansible ansible collection location = /home/student/.ansible/collections:/usr/share/ansible/collections executable location = /usr/local/bin/ansible python version = 3.9.16 (main, Sep 12 2023, 00:00:00) [GCC 11.3.1 20221121 (Red Hat 11.3.1-4)] (/usr/bin/python3) jinja version = 3.1.2 libyaml = True

Ansible Configuration

[student@ansible-1 ~]$ ansible-config dump --only-changed ACTION_WARNINGS(/etc/ansible/ansible.cfg) = False COLLECTIONS_ON_ANSIBLE_VERSION_MISMATCH(/etc/ansible/ansible.cfg) = ignore CONFIG_FILE() = /etc/ansible/ansible.cfg DEFAULT_HOST_LIST(/etc/ansible/ansible.cfg) = ['/home/student/lab_inventory/hosts'] DEFAULT_STDOUT_CALLBACK(/etc/ansible/ansible.cfg) = yaml DEFAULT_TIMEOUT(/etc/ansible/ansible.cfg) = 60 DEPRECATION_WARNINGS(/etc/ansible/ansible.cfg) = False DEVEL_WARNING(/etc/ansible/ansible.cfg) = False HOST_KEY_CHECKING(/etc/ansible/ansible.cfg) = False INTERPRETER_PYTHON(/etc/ansible/ansible.cfg) = auto_silent PERSISTENT_COMMAND_TIMEOUT(/etc/ansible/ansible.cfg) = 200 PERSISTENT_CONNECT_TIMEOUT(/etc/ansible/ansible.cfg) = 200 RETRY_FILES_ENABLED(/etc/ansible/ansible.cfg) = False SYSTEM_WARNINGS(/etc/ansible/ansible.cfg) = False

Ansible Execution Node

CLI Ansible (Ansible Core)

Operating System

Windows & MacOS

IPvSean commented 5 months ago

I am trying to see what is going on, recreating now in mye nv

IPvSean commented 4 months ago

OK I have been hammering on this most of Friday and today (Monday July 8th). I don't have a fix yet.... but these types of unreliable errors that are hard to reproduce are a giant pain.... it is really hard to figure out what is going on. In parallel I am ordering a bunch of workbenches from demo.redhat.com to see if I can get one to mess up.

Unrelated... but also causing pains are the Cisco IOS-XE support for SSH keys... I am testing a fix right now to move us to ED25519 vs ssh-rsa. My first few tests are favorable and will get rid of a lot of errors people have with ssh legacy SSH keys.

Anyways.... some findings for this specific issue with code-server and the workshop:

The UUID for the failed workshop is m4j6m
I can see a failure on July 2nd, but then a success rerun July 4th

The task that fails on July 2nd->

TASK [infra.controller_configuration.groups : Managing Controller Groups | Wait for finish the Controller Groups management] ***
changed: [m4j6m-student1-ansible-1] => (item=Create/Update Controller Group cisco | Wait for finish the Controller Group creation)
changed: [m4j6m-student2-ansible-1] => (item=Create/Update Controller Group cisco | Wait for finish the Controller Group creation)
changed: [m4j6m-student3-ansible-1] => (item=Create/Update Controller Group cisco | Wait for finish the Controller Group creation)
changed: [m4j6m-student4-ansible-1] => (item=Create/Update Controller Group cisco | Wait for finish the Controller Group creation)
changed: [m4j6m-student6-ansible-1] => (item=Create/Update Controller Group cisco | Wait for finish the Controller Group creation)
changed: [m4j6m-student5-ansible-1] => (item=Create/Update Controller Group cisco | Wait for finish the Controller Group creation)
changed: [m4j6m-student7-ansible-1] => (item=Create/Update Controller Group cisco | Wait for finish the Controller Group creation)
changed: [m4j6m-student8-ansible-1] => (item=Create/Update Controller Group cisco | Wait for finish the Controller Group creation)
changed: [m4j6m-student11-ansible-1] => (item=Create/Update Controller Group cisco | Wait for finish the Controller Group creation)
changed: [m4j6m-student12-ansible-1] => (item=Create/Update Controller Group cisco | Wait for finish the Controller Group creation)
changed: [m4j6m-student10-ansible-1] => (item=Create/Update Controller Group cisco | Wait for finish the Controller Group creation)
changed: [m4j6m-student9-ansible-1] => (item=Create/Update Controller Group cisco | Wait for finish the Controller Group creation)
changed: [m4j6m-student13-ansible-1] => (item=Create/Update Controller Group cisco | Wait for finish the Controller Group creation)
changed: [m4j6m-student14-ansible-1] => (item=Create/Update Controller Group cisco | Wait for finish the Controller Group creation)
changed: [m4j6m-student15-ansible-1] => (item=Create/Update Controller Group cisco | Wait for finish the Controller Group creation)
changed: [m4j6m-student16-ansible-1] => (item=Create/Update Controller Group cisco | Wait for finish the Controller Group creation)
changed: [m4j6m-student17-ansible-1] => (item=Create/Update Controller Group cisco | Wait for finish the Controller Group creation)
changed: [m4j6m-student20-ansible-1] => (item=Create/Update Controller Group cisco | Wait for finish the Controller Group creation)
changed: [m4j6m-student18-ansible-1] => (item=Create/Update Controller Group cisco | Wait for finish the Controller Group creation)
changed: [m4j6m-student21-ansible-1] => (item=Create/Update Controller Group cisco | Wait for finish the Controller Group creation)
changed: [m4j6m-student19-ansible-1] => (item=Create/Update Controller Group cisco | Wait for finish the Controller Group creation)
changed: [m4j6m-student1-ansible-1] => (item=Create/Update Controller Group arista | Wait for finish the Controller Group creation)
changed: [m4j6m-student3-ansible-1] => (item=Create/Update Controller Group arista | Wait for finish the Controller Group creation)
changed: [m4j6m-student2-ansible-1] => (item=Create/Update Controller Group arista | Wait for finish the Controller Group creation)
changed: [m4j6m-student7-ansible-1] => (item=Create/Update Controller Group arista | Wait for finish the Controller Group creation)
changed: [m4j6m-student11-ansible-1] => (item=Create/Update Controller Group arista | Wait for finish the Controller Group creation)
changed: [m4j6m-student4-ansible-1] => (item=Create/Update Controller Group arista | Wait for finish the Controller Group creation)
changed: [m4j6m-student5-ansible-1] => (item=Create/Update Controller Group arista | Wait for finish the Controller Group creation)
changed: [m4j6m-student6-ansible-1] => (item=Create/Update Controller Group arista | Wait for finish the Controller Group creation)
changed: [m4j6m-student8-ansible-1] => (item=Create/Update Controller Group arista | Wait for finish the Controller Group creation)
changed: [m4j6m-student12-ansible-1] => (item=Create/Update Controller Group arista | Wait for finish the Controller Group creation)
changed: [m4j6m-student10-ansible-1] => (item=Create/Update Controller Group arista | Wait for finish the Controller Group creation)
changed: [m4j6m-student13-ansible-1] => (item=Create/Update Controller Group arista | Wait for finish the Controller Group creation)
changed: [m4j6m-student15-ansible-1] => (item=Create/Update Controller Group arista | Wait for finish the Controller Group creation)
changed: [m4j6m-student14-ansible-1] => (item=Create/Update Controller Group arista | Wait for finish the Controller Group creation)
changed: [m4j6m-student19-ansible-1] => (item=Create/Update Controller Group arista | Wait for finish the Controller Group creation)
changed: [m4j6m-student16-ansible-1] => (item=Create/Update Controller Group arista | Wait for finish the Controller Group creation)
changed: [m4j6m-student9-ansible-1] => (item=Create/Update Controller Group arista | Wait for finish the Controller Group creation)
changed: [m4j6m-student17-ansible-1] => (item=Create/Update Controller Group arista | Wait for finish the Controller Group creation)
changed: [m4j6m-student21-ansible-1] => (item=Create/Update Controller Group arista | Wait for finish the Controller Group creation)
changed: [m4j6m-student18-ansible-1] => (item=Create/Update Controller Group arista | Wait for finish the Controller Group creation)
changed: [m4j6m-student20-ansible-1] => (item=Create/Update Controller Group arista | Wait for finish the Controller Group creation)
changed: [m4j6m-student1-ansible-1] => (item=Create/Update Controller Group juniper | Wait for finish the Controller Group creation)
changed: [m4j6m-student3-ansible-1] => (item=Create/Update Controller Group juniper | Wait for finish the Controller Group creation)
changed: [m4j6m-student2-ansible-1] => (item=Create/Update Controller Group juniper | Wait for finish the Controller Group creation)
changed: [m4j6m-student11-ansible-1] => (item=Create/Update Controller Group juniper | Wait for finish the Controller Group creation)
changed: [m4j6m-student7-ansible-1] => (item=Create/Update Controller Group juniper | Wait for finish the Controller Group creation)
changed: [m4j6m-student4-ansible-1] => (item=Create/Update Controller Group juniper | Wait for finish the Controller Group creation)
changed: [m4j6m-student5-ansible-1] => (item=Create/Update Controller Group juniper | Wait for finish the Controller Group creation)
changed: [m4j6m-student10-ansible-1] => (item=Create/Update Controller Group juniper | Wait for finish the Controller Group creation)
changed: [m4j6m-student8-ansible-1] => (item=Create/Update Controller Group juniper | Wait for finish the Controller Group creation)
changed: [m4j6m-student12-ansible-1] => (item=Create/Update Controller Group juniper | Wait for finish the Controller Group creation)
changed: [m4j6m-student6-ansible-1] => (item=Create/Update Controller Group juniper | Wait for finish the Controller Group creation)
changed: [m4j6m-student13-ansible-1] => (item=Create/Update Controller Group juniper | Wait for finish the Controller Group creation)
changed: [m4j6m-student19-ansible-1] => (item=Create/Update Controller Group juniper | Wait for finish the Controller Group creation)
changed: [m4j6m-student15-ansible-1] => (item=Create/Update Controller Group juniper | Wait for finish the Controller Group creation)
changed: [m4j6m-student17-ansible-1] => (item=Create/Update Controller Group juniper | Wait for finish the Controller Group creation)
changed: [m4j6m-student14-ansible-1] => (item=Create/Update Controller Group juniper | Wait for finish the Controller Group creation)
changed: [m4j6m-student20-ansible-1] => (item=Create/Update Controller Group juniper | Wait for finish the Controller Group creation)
changed: [m4j6m-student16-ansible-1] => (item=Create/Update Controller Group juniper | Wait for finish the Controller Group creation)
changed: [m4j6m-student21-ansible-1] => (item=Create/Update Controller Group juniper | Wait for finish the Controller Group creation)
changed: [m4j6m-student18-ansible-1] => (item=Create/Update Controller Group juniper | Wait for finish the Controller Group creation)
changed: [m4j6m-student9-ansible-1] => (item=Create/Update Controller Group juniper | Wait for finish the Controller Group creation)
changed: [m4j6m-student1-ansible-1] => (item=Create/Update Controller Group routers | Wait for finish the Controller Group creation)
changed: [m4j6m-student3-ansible-1] => (item=Create/Update Controller Group routers | Wait for finish the Controller Group creation)
changed: [m4j6m-student2-ansible-1] => (item=Create/Update Controller Group routers | Wait for finish the Controller Group creation)
changed: [m4j6m-student11-ansible-1] => (item=Create/Update Controller Group routers | Wait for finish the Controller Group creation)
changed: [m4j6m-student7-ansible-1] => (item=Create/Update Controller Group routers | Wait for finish the Controller Group creation)
changed: [m4j6m-student5-ansible-1] => (item=Create/Update Controller Group routers | Wait for finish the Controller Group creation)
changed: [m4j6m-student4-ansible-1] => (item=Create/Update Controller Group routers | Wait for finish the Controller Group creation)
changed: [m4j6m-student10-ansible-1] => (item=Create/Update Controller Group routers | Wait for finish the Controller Group creation)
changed: [m4j6m-student8-ansible-1] => (item=Create/Update Controller Group routers | Wait for finish the Controller Group creation)
changed: [m4j6m-student12-ansible-1] => (item=Create/Update Controller Group routers | Wait for finish the Controller Group creation)
changed: [m4j6m-student19-ansible-1] => (item=Create/Update Controller Group routers | Wait for finish the Controller Group creation)
changed: [m4j6m-student13-ansible-1] => (item=Create/Update Controller Group routers | Wait for finish the Controller Group creation)
changed: [m4j6m-student15-ansible-1] => (item=Create/Update Controller Group routers | Wait for finish the Controller Group creation)
FAILED - RETRYING: [m4j6m-student6-ansible-1]: Managing Controller Groups | Wait for finish the Controller Groups management (30 retries left).
changed: [m4j6m-student17-ansible-1] => (item=Create/Update Controller Group routers | Wait for finish the Controller Group creation)
changed: [m4j6m-student20-ansible-1] => (item=Create/Update Controller Group routers | Wait for finish the Controller Group creation)
changed: [m4j6m-student21-ansible-1] => (item=Create/Update Controller Group routers | Wait for finish the Controller Group creation)
changed: [m4j6m-student14-ansible-1] => (item=Create/Update Controller Group routers | Wait for finish the Controller Group creation)
changed: [m4j6m-student16-ansible-1] => (item=Create/Update Controller Group routers | Wait for finish the Controller Group creation)
changed: [m4j6m-student18-ansible-1] => (item=Create/Update Controller Group routers | Wait for finish the Controller Group creation)
changed: [m4j6m-student9-ansible-1] => (item=Create/Update Controller Group routers | Wait for finish the Controller Group creation)
failed: [m4j6m-student1-ansible-1] (item=Create/Update Controller Group network | Wait for finish the Controller Group creation) => {"__group_job_async_results_item": {"__controller_groups_item": {"children": ["routers"], "inventory": "Workshop Inventory", "name": "network", "variables": {"restore_credential": "Workshop Credential", "restore_inventory": "Workshop Inventory", "restore_project": "Workshop Project"}}, "ansible_job_id": "j156787352862.21826", "ansible_loop_var": "__controller_groups_item", "changed": false, "failed": 0, "finished": 0, "results_file": "/root/.ansible_async/j156787352862.21826", "started": 1}, "ansible_job_id": "j156787352862.21826", "ansible_loop_var": "__group_job_async_results_item", "attempts": 1, "changed": false, "finished": 1, "msg": "Could not find groups with name routers", "results_file": "/root/.ansible_async/j156787352862.21826", "started": 1, "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
failed: [m4j6m-student3-ansible-1] (item=Create/Update Controller Group network | Wait for finish the Controller Group creation) => {"__group_job_async_results_item": {"__controller_groups_item": {"children": ["routers"], "inventory": "Workshop Inventory", "name": "network", "variables": {"restore_credential": "Workshop Credential", "restore_inventory": "Workshop Inventory", "restore_project": "Workshop Project"}}, "ansible_job_id": "j859423898414.21486", "ansible_loop_var": "__controller_groups_item", "changed": false, "failed": 0, "finished": 0, "results_file": "/root/.ansible_async/j859423898414.21486", "started": 1}, "ansible_job_id": "j859423898414.21486", "ansible_loop_var": "__group_job_async_results_item", "attempts": 1, "changed": false, "finished": 1, "msg": "Could not find groups with name routers", "results_file": "/root/.ansible_async/j859423898414.21486", "started": 1, "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
failed: [m4j6m-student2-ansible-1] (item=Create/Update Controller Group network | Wait for finish the Controller Group creation) => {"__group_job_async_results_item": {"__controller_groups_item": {"children": ["routers"], "inventory": "Workshop Inventory", "name": "network", "variables": {"restore_credential": "Workshop Credential", "restore_inventory": "Workshop Inventory", "restore_project": "Workshop Project"}}, "ansible_job_id": "j989485026492.21631", "ansible_loop_var": "__controller_groups_item", "changed": false, "failed": 0, "finished": 0, "results_file": "/root/.ansible_async/j989485026492.21631", "started": 1}, "ansible_job_id": "j989485026492.21631", "ansible_loop_var": "__group_job_async_results_item", "attempts": 1, "changed": false, "finished": 1, "msg": "Could not find groups with name routers", "results_file": "/root/.ansible_async/j989485026492.21631", "started": 1, "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
failed: [m4j6m-student7-ansible-1] (item=Create/Update Controller Group network | Wait for finish the Controller Group creation) => {"__group_job_async_results_item": {"__controller_groups_item": {"children": ["routers"], "inventory": "Workshop Inventory", "name": "network", "variables": {"restore_credential": "Workshop Credential", "restore_inventory": "Workshop Inventory", "restore_project": "Workshop Project"}}, "ansible_job_id": "j466050216929.21465", "ansible_loop_var": "__controller_groups_item", "changed": false, "failed": 0, "finished": 0, "results_file": "/root/.ansible_async/j466050216929.21465", "started": 1}, "ansible_job_id": "j466050216929.21465", "ansible_loop_var": "__group_job_async_results_item", "attempts": 1, "changed": false, "finished": 1, "msg": "Could not find groups with name routers", "results_file": "/root/.ansible_async/j466050216929.21465", "started": 1, "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
failed: [m4j6m-student11-ansible-1] (item=Create/Update Controller Group network | Wait for finish the Controller Group creation) => {"__group_job_async_results_item": {"__controller_groups_item": {"children": ["routers"], "inventory": "Workshop Inventory", "name": "network", "variables": {"restore_credential": "Workshop Credential", "restore_inventory": "Workshop Inventory", "restore_project": "Workshop Project"}}, "ansible_job_id": "j739768768972.21481", "ansible_loop_var": "__controller_groups_item", "changed": false, "failed": 0, "finished": 0, "results_file": "/root/.ansible_async/j739768768972.21481", "started": 1}, "ansible_job_id": "j739768768972.21481", "ansible_loop_var": "__group_job_async_results_item", "attempts": 1, "changed": false, "finished": 1, "msg": "Could not find groups with name routers", "results_file": "/root/.ansible_async/j739768768972.21481", "started": 1, "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
changed: [m4j6m-student5-ansible-1] => (item=Create/Update Controller Group network | Wait for finish the Controller Group creation)
changed: [m4j6m-student4-ansible-1] => (item=Create/Update Controller Group network | Wait for finish the Controller Group creation)
changed: [m4j6m-student10-ansible-1] => (item=Create/Update Controller Group network | Wait for finish the Controller Group creation)
changed: [m4j6m-student8-ansible-1] => (item=Create/Update Controller Group network | Wait for finish the Controller Group creation)
changed: [m4j6m-student12-ansible-1] => (item=Create/Update Controller Group network | Wait for finish the Controller Group creation)
failed: [m4j6m-student19-ansible-1] (item=Create/Update Controller Group network | Wait for finish the Controller Group creation) => {"__group_job_async_results_item": {"__controller_groups_item": {"children": ["routers"], "inventory": "Workshop Inventory", "name": "network", "variables": {"restore_credential": "Workshop Credential", "restore_inventory": "Workshop Inventory", "restore_project": "Workshop Project"}}, "ansible_job_id": "j365342199221.21364", "ansible_loop_var": "__controller_groups_item", "changed": false, "failed": 0, "finished": 0, "results_file": "/root/.ansible_async/j365342199221.21364", "started": 1}, "ansible_job_id": "j365342199221.21364", "ansible_loop_var": "__group_job_async_results_item", "attempts": 1, "changed": false, "finished": 1, "msg": "Could not find groups with name routers", "results_file": "/root/.ansible_async/j365342199221.21364", "started": 1, "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
changed: [m4j6m-student13-ansible-1] => (item=Create/Update Controller Group network | Wait for finish the Controller Group creation)
changed: [m4j6m-student15-ansible-1] => (item=Create/Update Controller Group network | Wait for finish the Controller Group creation)
changed: [m4j6m-student17-ansible-1] => (item=Create/Update Controller Group network | Wait for finish the Controller Group creation)
changed: [m4j6m-student20-ansible-1] => (item=Create/Update Controller Group network | Wait for finish the Controller Group creation)
changed: [m4j6m-student21-ansible-1] => (item=Create/Update Controller Group network | Wait for finish the Controller Group creation)
changed: [m4j6m-student16-ansible-1] => (item=Create/Update Controller Group network | Wait for finish the Controller Group creation)
changed: [m4j6m-student14-ansible-1] => (item=Create/Update Controller Group network | Wait for finish the Controller Group creation)
failed: [m4j6m-student18-ansible-1] (item=Create/Update Controller Group network | Wait for finish the Controller Group creation) => {"__group_job_async_results_item": {"__controller_groups_item": {"children": ["routers"], "inventory": "Workshop Inventory", "name": "network", "variables": {"restore_credential": "Workshop Credential", "restore_inventory": "Workshop Inventory", "restore_project": "Workshop Project"}}, "ansible_job_id": "j171681188278.21375", "ansible_loop_var": "__controller_groups_item", "changed": false, "failed": 0, "finished": 0, "results_file": "/root/.ansible_async/j171681188278.21375", "started": 1}, "ansible_job_id": "j171681188278.21375", "ansible_loop_var": "__group_job_async_results_item", "attempts": 1, "changed": false, "finished": 1, "msg": "Could not find groups with name routers", "results_file": "/root/.ansible_async/j171681188278.21375", "started": 1, "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
changed: [m4j6m-student9-ansible-1] => (item=Create/Update Controller Group network | Wait for finish the Controller Group creation)
changed: [m4j6m-student6-ansible-1] => (item=Create/Update Controller Group routers | Wait for finish the Controller Group creation)
changed: [m4j6m-student6-ansible-1] => (item=Create/Update Controller Group network | Wait for finish the Controller Group creation)

It is weird that it is failing for 7 (seven) hosts but the others are fine, because they should all be identical. It also.... "sorta' matches that "some hosts" are broken..... There seems to be some sort of "order of operations" with using controller configuration collection to pre-provision groups.

However... I am not sure why this would affect visual studio code (or code-server in this case). These roles are not connected in anyway...

when I look at the code-server code everything ran correctly....

My thoughts:

I am not 100% positive that the populate_controller role failure affects code-server at all... I need to replicate both failures. It is just interesting how random hosts were affected... and its also hard to reproduce.
It is possible that some memory or storage issue affected the hosts... so when hammered with API requests the machine just failed... which would then fail code-server... again I can't prove this yet...
I originally thought this was a lets encrypt issue which would explain the intermittent failures but nothing in logs is pointing at that so far.

I wish I was 100% certain what is going on... this workshop type is in the process of being migrated to agnosticd and the demo.redhat.com native software so I have not been maintaining it as frequently as I used to... so this is also probably slowing down mean time to resolution. Keep reporting any issue you are seeing and I will continue to hammer on this.

IPvSean commented 4 months ago

OK it is related b/c its a handler.... the handler to restart code-server is never run

I was able to recreate

RUNNING HANDLER [../roles/code_server_native : Restart code-server] ************
changed: [kwtpc-student3-ansible-1]
changed: [kwtpc-student2-ansible-1]
changed: [kwtpc-student4-ansible-1]
[WARNING]: Could not match supplied host pattern, ignoring: automation_hub

In this example, student1 and studen5 are missing from the code-server restart, b/c they failed on the identical task

failed: [kwtpc-student1-ansible-1] (item=Create/Update Controller Group network | Wait for finish the Controller Group creation) => {"__group_job_async_results_item": {"__controller_groups_item": {"children": ["routers"], "inventory": "Workshop Inventory", "name": "network", "variables": {"restore_credential": "Workshop Credential", "restore_inventory": "Workshop Inventory", "restore_project": "Workshop Project"}}, "ansible_job_id": "j685691272827.21906", "ansible_loop_var": "__controller_groups_item", "changed": false, "failed": 0, "finished": 0, "results_file": "/root/.ansible_async/j685691272827.21906", "started": 1}, "ansible_job_id": "j685691272827.21906", "ansible_loop_var": "__group_job_async_results_item", "attempts": 1, "changed": false, "finished": 1, "msg": "Could not find groups with name routers", "results_file": "/root/.ansible_async/j685691272827.21906", "started": 1, "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

IPvSean commented 4 months ago

So the reason this fails is because when I use the controller_configuration content collection

controller_groups:
  - name: cisco
    inventory: "Workshop Inventory"
    variables:
      ansible_network_os: ios
      ansible_connection: network_cli
  - name: arista
    inventory: "Workshop Inventory"
    variables:
      ansible_network_os: eos
      ansible_connection: network_cli
      ansible_become: true
      ansible_become_method: enable
  - name: juniper
    inventory: "Workshop Inventory"
    variables:
      ansible_network_os: junos
      ansible_connection: netconf
  - name: routers
    inventory: "Workshop Inventory"
    children:
      - cisco
      - arista
      - juniper
  - name: network
    inventory: "Workshop Inventory"
    children:
      - routers
    variables:
      restore_inventory: "Workshop Inventory"
      restore_credential: "Workshop Credential"
      restore_project: "Workshop Project"

It is a dictionary...

dictionaries are commonly not ordered.... so the problem

failed: [kwtpc-student1-ansible-1] (item=Create/Update Controller Group network | Wait for finish the Controller Group creation) => {"__group_job_async_results_item": {"__controller_groups_item": {"children": ["routers"], "inventory": "Workshop Inventory", "name": "network", "variables": {"restore_credential": "Workshop Credential", "restore_inventory": "Workshop Inventory", "restore_project": "Workshop Project"}}, "ansible_job_id": "j685691272827.21906", "ansible_loop_var": "__controller_groups_item", "changed": false, "failed": 0, "finished": 0, "results_file": "/root/.ansible_async/j685691272827.21906", "started": 1}, "ansible_job_id": "j685691272827.21906", "ansible_loop_var": "__group_job_async_results_item", "attempts": 1, "changed": false, "finished": 1, "msg": "Could not find groups with name routers", "results_file": "/root/.ansible_async/j685691272827.21906", "started": 1, "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

happens SOMETIMES because it will try to configure the group network before the group routers has been created (which has like a 1 in 5 chance... of happening) or 20% it seems.... looking at rough math above.... 20% of 20 student is 4 and we had 7 hosts messed up, so this roughly now makes sense to me.

I have gotten rid of those two groups from using controller configuration and forced them into manual adds at the end in the right order. Will send fixes tomorrow.

ansible / workshops