RHEL 9 - vanilla ascender install using K3s - app-web is crashing

[linuxcube]

Hi Guys, I am trying to do a vanilla deployment of ascender (latest)_ under a RHEL 9.4 VM.

The k3s gets deploy but somehow my app-web container end in crash state, I checked the logs but didnt see anything interesting there. See the output of my containers below:

[sysadmin@ascender-dev ~]$ kubectl get pods -n ascender NAME READY STATUS RESTARTS AGE ascender-app-migration-24.0.1-5zpn5 0/1 Completed 0 39m ascender-app-postgres-15-0 1/1 Running 0 41m ascender-app-task-d4c5c7f5d-fb4zn 4/4 Running 0 40m ascender-app-web-7ddd546dc6-5bmkh 2/3 CrashLoopBackOff 16 (116s ago) 40m awx-operator-controller-manager-8f8f4df5d-mfsrr 2/2 Running 0 42m

Any clues as to what is happening on this case?

[linuxcube]

It looks like something is causing the issue in the App-Web deployment, see the step where it fails:

TASK [ascender_install : Wait until Ascender API is Up (This may take between 10-20 mins)] ***** FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (200 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (199 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (198 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (197 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (196 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (195 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (194 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (193 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (192 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (191 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (190 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (189 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (188 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (187 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (186 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (185 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (184 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (183 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (182 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (181 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (180 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (179 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (178 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (177 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (176 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (175 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (174 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (173 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (172 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (171 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (170 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (169 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (168 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (167 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (166 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (165 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (164 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (163 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (162 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (161 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (160 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (159 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (158 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (157 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (156 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (155 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (154 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (153 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (152 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (151 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (150 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (149 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (148 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (147 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (146 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (145 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (144 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (143 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (142 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (141 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (140 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (139 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (138 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (137 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (136 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (135 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (134 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (133 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (132 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (131 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (130 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (129 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (128 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (127 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (126 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (125 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (124 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (123 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (122 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (121 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (120 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (119 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (118 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (117 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (116 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (115 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (114 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (113 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (112 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (111 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (110 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (109 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (108 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (107 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (106 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (105 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (104 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (103 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (102 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (101 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (100 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (99 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (98 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (97 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (96 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (95 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (94 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (93 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (92 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (91 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (90 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (89 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (88 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (87 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (86 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (85 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (84 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (83 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (82 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (81 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (80 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (79 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (78 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (77 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (76 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (75 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (74 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (73 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (72 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (71 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (70 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (69 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (68 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (67 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (66 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (65 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (64 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (63 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (62 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (61 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (60 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (59 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (58 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (57 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (56 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (55 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (54 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (53 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (52 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (51 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (50 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (49 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (48 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (47 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (46 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (45 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (44 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (43 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (42 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (41 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (40 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (39 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (38 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (37 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (36 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (35 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (34 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (33 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (32 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (31 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (30 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (29 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (28 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (27 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (26 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (25 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (24 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (23 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (22 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (21 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (20 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (19 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (18 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (17 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (16 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (15 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (14 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (13 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (12 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (11 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (10 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (9 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (8 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (7 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (6 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (5 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (4 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (3 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (2 retries left). FAILED - RETRYING: [localhost]: Wait until Ascender API is Up (This may take between 10-20 mins) (1 retries left). fatal: [localhost]: FAILED! => {"attempts": 200, "changed": false, "content": "", "elapsed": 0, "msg": "Status code was -1 and not [200]: Request failed: <urlopen error [Errno 111] Connection refused>", "redirected": false, "status": -1, "url": "http://ascender-dev.local:80/api/v2/ping/"}

PLAY RECAP ***** ascender_host : ok=16 changed=7 unreachable=0 failed=0 skipped=10 rescued=0 ignored=0
localhost : ok=18 changed=9 unreachable=0 failed=1 skipped=7 rescued=0 ignored=0

ERROR OCCURRED DURING SETUP

[michaelford85]

Two possibilities come to mind:

• Perhaps the version of ascender (The ASCENDER_VERSION variable) is incorrect.
• Your DNS resolution for your ASCENDER_HOSTNAME is not completing.

I just ran the k3s installer on a fresh Rocky 9.4 t2.xlarge instance from AWS in us-east-1 (AMI: ami-09fb459fad4613d55), and specified to use /etc/hosts for DNS. I am pasting my custom.config.yml file for you to compare to.

Also, you might try running $ kubectl describe pod ascender-app-web-7ddd546dc6-5bmkh -n ascender to determine if there is an error pulling one of the container images.

---
# This variable specificies which Kubernetes platform Ascender and its components will be installed on.
k8s_platform: k3s
# Boolean indicating whether to set up a new k3s cluster (true) or use an existing k3s cluster (false)
kube_install: true
# Offline Install - Whether to use local assets to complete the install
k8s_offline: false
# Boolean indicating whether or not the kubeconfig file needs to be downloaded to the Ansible controller
download_kubeconfig: true
# Determines whether to use HTTP or HTTPS for Ascender and Ledger.
# If set to https, you MUST provide certificate/key options for the Installer to use.
k8s_lb_protocol: http
# Routable IP address for the K3s Master/Worker node
# required for DNS and k3s install
k3s_master_node_ip: "127.0.0.1"
# Boolean indicating whether to use the local /etc/hosts file for DNS resolution to access Ascender
use_etc_hosts: true
# A directory in which to place both temporary artifacts
# and timestamped Kubernetes Manifests to make Ascender/Ledger easy
# to uninstall
tmp_dir: "{{ playbook_dir}}/../ascender_install_artifacts"
# DNS resolvable hostname for Ascender service. This is required for install.
ASCENDER_HOSTNAME: ascender.example.com
# Namespace for Ascender Kubernetes objects
ASCENDER_NAMESPACE: ascender
# Administrator username for Ascender
ASCENDER_ADMIN_USER: admin
# Administrator password for Ascender
ASCENDER_ADMIN_PASSWORD: "myadminpassword"
# The image tag indicating the version of Ascender you wish to install
ASCENDER_VERSION: 24.0.1
# The version of the AWX Operator used to install Ascender and its components
ANSIBLE_OPERATOR_VERSION: 2.19.0
# Determines whether to keep the secrets required to encrypt within Ascender (important when backing up)
ascender_garbage_collect_secrets: false
# External PostgreSQL database name used for Ascender (this DB must exist)
ascender_replicas: 1
# The Ascender web container image pull policy (If unsure, choose IfNotPresent)
image_pull_policy: Always
# Boolean indicating whether to add standard playbooks into Ascender after installation
ascender_setup_playbooks: false
# Determines whether or not Ledger will be installed
LEDGER_INSTALL: true
# DNS resolvable hostname for Ledger service. This is required for install
LEDGER_HOSTNAME: ledger.example.com
# Number of replicas for the Ledger web container
ledger_web_replicas: 1
# Number of replicas for the Ledger Parser container
ledger_parser_replicas: 1
# The image tag indicating the version of Ledger you wish to install
LEDGER_VERSION: latest
# The Kubernetes namespace in which Ledger objects will live
LEDGER_NAMESPACE: ledger
# Admin password for Ledger (the username is admin by default)
LEDGER_ADMIN_PASSWORD: myadminpassword
# Password for Ledger database
LEDGER_DB_PASSWORD: mydbpassword

[linuxcube]

Hi @michaelford85 , Below find a copy of my /etc/hosts file:

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 10.10.200.24 ascender-dev.local 10.10.200.24 ledger-dev.local

Below is the output of "kubectl describe pod ascender-app-web-7ddd546dc6-5bmkh -n ascender" as requested:

Name: ascender-app-web-7ddd546dc6-5bmkh Namespace: ascender Priority: 0 Service Account: ascender-app Node: ascender-dev.local/10.10.200.24 Start Time: Sun, 14 Jul 2024 08:51:01 -0700 Labels: app.kubernetes.io/component=awx app.kubernetes.io/managed-by=awx-operator app.kubernetes.io/name=ascender-app-web app.kubernetes.io/operator-version=2.19.1 app.kubernetes.io/part-of=ascender-app app.kubernetes.io/version=24.0.1 pod-template-hash=7ddd546dc6 Annotations: checksum-configmaps-config: f4c5b18925028951c710480d9eaabe3d91487037 checksum-secret-bundle_cacert: da39a3ee5e6b4b0d3255bfef95601890afd80709 checksum-secret-ldap_cacert: da39a3ee5e6b4b0d3255bfef95601890afd80709 checksum-secret-receptor_ca: ba262090262d6e2c73b027f3c734e43ecc4a6547 checksum-secret-receptor_work_signing: 12050719f016ff67c7b0a8282768bc8af8cf1f60 checksum-secret-route_tls: da39a3ee5e6b4b0d3255bfef95601890afd80709 checksum-secret-secret_key: 42d6fa90f7e15a97872941b6b260845edaf3b7d7 checksum-secrets-app_credentials: f588348430f83dc3b070fdbc018c90acea7f34d2 checksum-storage-persistent: adc83b19e793491b1c6ea0fd8b46cd9f32e592fc kubectl.kubernetes.io/default-container: ascender-app-web Status: Running IP: 10.42.0.17 IPs: IP: 10.42.0.17 Controlled By: ReplicaSet/ascender-app-web-7ddd546dc6 Containers: redis: Container ID: containerd://f292a400849c5a7d71c5ccc3ef11cece008838810b9dc789aaa20330d898fed8 Image: docker.io/redis:7 Image ID: docker.io/library/redis@sha256:fb534a36ac2034a6374933467d971fbcbfa5d213805507f560d564851a720355 Port: Host Port: Args: redis-server /etc/redis.conf State: Running Started: Sun, 14 Jul 2024 08:51:07 -0700 Ready: True Restart Count: 0 Requests: cpu: 50m memory: 64Mi Environment: Mounts: /data from ascender-app-redis-data (rw) /etc/redis.conf from ascender-app-redis-config (ro,path="redis.conf") /var/run/redis from ascender-app-redis-socket (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-w2prp (ro) ascender-app-web: Container ID: containerd://ffea441e503fc62ec6777592e53c6dc42863d496d16711502f5a3f15fad536c6 Image: ghcr.io/ctrliq/ascender:24.0.1 Image ID: ghcr.io/ctrliq/ascender@sha256:d7f67a9830517ae9188e7e1418893c1a936e0be2957178301221e977d95f0f12 Port: 8052/TCP Host Port: 0/TCP Args: /usr/bin/launch_awx_web.sh State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Completed Exit Code: 0 Started: Sun, 14 Jul 2024 11:02:25 -0700 Finished: Sun, 14 Jul 2024 11:02:46 -0700 Ready: False Restart Count: 33 Requests: cpu: 100m memory: 128Mi Environment: AWX_COMPONENT: web SUPERVISOR_CONFIG_PATH: /etc/supervisord_web.conf MY_POD_NAMESPACE: ascender (v1:metadata.namespace) MY_POD_IP: (v1:status.podIP) UWSGI_MOUNT_PATH: / Mounts: /etc/nginx/nginx.conf from ascender-app-nginx-conf (ro,path="nginx.conf") /etc/receptor/tls/ca/mesh-CA.crt from ascender-app-receptor-ca (ro,path="tls.crt") /etc/receptor/tls/ca/mesh-CA.key from ascender-app-receptor-ca (ro,path="tls.key") /etc/receptor/work_public_key.pem from ascender-app-receptor-work-signing (ro,path="work-public-key.pem") /etc/tower/SECRET_KEY from ascender-app-secret-key (ro,path="SECRET_KEY") /etc/tower/conf.d/credentials.py from ascender-app-application-credentials (ro,path="credentials.py") /etc/tower/conf.d/execution_environments.py from ascender-app-application-credentials (ro,path="execution_environments.py") /etc/tower/conf.d/ldap.py from ascender-app-application-credentials (ro,path="ldap.py") /etc/tower/settings.py from ascender-app-settings (ro,path="settings.py") /etc/tower/uwsgi.ini from ascender-app-uwsgi-config (ro,path="uwsgi.conf") /var/run/awx-rsyslog from rsyslog-socket (rw) /var/run/redis from ascender-app-redis-socket (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-w2prp (ro) ascender-app-rsyslog: Container ID: containerd://26ef7f1820af7bb5237ff51428505049f4281fb9fcd1ca4b2a2d966472bb755e Image: ghcr.io/ctrliq/ascender:24.0.1 Image ID: ghcr.io/ctrliq/ascender@sha256:d7f67a9830517ae9188e7e1418893c1a936e0be2957178301221e977d95f0f12 Port: Host Port: Args: /usr/bin/launch_awx_rsyslog.sh State: Running Started: Sun, 14 Jul 2024 08:51:26 -0700 Ready: True Restart Count: 0 Requests: cpu: 100m memory: 128Mi Environment: SUPERVISOR_CONFIG_PATH: /etc/supervisord_rsyslog.conf Mounts: /etc/tower/SECRET_KEY from ascender-app-secret-key (ro,path="SECRET_KEY") /etc/tower/conf.d/credentials.py from ascender-app-application-credentials (ro,path="credentials.py") /etc/tower/settings.py from ascender-app-settings (ro,path="settings.py") /var/run/awx-rsyslog from rsyslog-socket (rw) /var/run/redis from ascender-app-redis-socket (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-w2prp (ro) Conditions: Type Status PodReadyToStartContainers True Initialized True Ready False ContainersReady False PodScheduled True Volumes: ascender-app-receptor-ca: Type: Secret (a volume populated by a Secret) SecretName: ascender-app-receptor-ca Optional: false ascender-app-receptor-work-signing: Type: Secret (a volume populated by a Secret) SecretName: ascender-app-receptor-work-signing Optional: false ascender-app-application-credentials: Type: Secret (a volume populated by a Secret) SecretName: ascender-app-app-credentials Optional: false ascender-app-secret-key: Type: Secret (a volume populated by a Secret) SecretName: ascender-app-secret-key Optional: false ascender-app-settings: Type: ConfigMap (a volume populated by a ConfigMap) Name: ascender-app-awx-configmap Optional: false ascender-app-nginx-conf: Type: ConfigMap (a volume populated by a ConfigMap) Name: ascender-app-awx-configmap Optional: false ascender-app-redis-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: ascender-app-awx-configmap Optional: false ascender-app-uwsgi-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: ascender-app-awx-configmap Optional: false ascender-app-redis-socket: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium:
SizeLimit: ascender-app-redis-data: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium:
SizeLimit: rsyslog-socket: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium:
SizeLimit: receptor-socket: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium:
SizeLimit: ascender-app-receptor-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: ascender-app-awx-configmap Optional: false kube-api-access-w2prp: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message

---

Warning BackOff 15s (x513 over 120m) kubelet Back-off restarting failed container ascender-app-web in pod ascender-app-web-7ddd546dc6-5bmkh_ascender(156714a7-547d-4ab3-8167-7e748d86a815)

[linuxcube]

Below is the custom code of my ascender script:

---
# This variable specificies which Kubernetes platform Ascender and its components will be installed on.
k8s_platform: k3s
# Boolean indicating whether to set up a new k3s cluster (true) or use an existing k3s cluster (false)
kube_install: true
# Offline Install - Whether to use local assets to complete the install
k8s_offline: false
# Boolean indicating whether or not the kubeconfig file needs to be downloaded to the Ansible controller
download_kubeconfig: true
# Determines whether to use HTTP or HTTPS for Ascender and Ledger.
# If set to https, you MUST provide certificate/key options for the Installer to use.
k8s_lb_protocol: http
# Routable IP address for the K3s Master/Worker node
# required for DNS and k3s install
k3s_master_node_ip: "10.10.200.24"
# Boolean indicating whether to use the local /etc/hosts file for DNS resolution to access Ascender
use_etc_hosts: true
# A directory in which to place both temporary artifacts
# and timestamped Kubernetes Manifests to make Ascender/Ledger easy
# to uninstall
tmp_dir: "{{ playbook_dir}}/../ascender_install_artifacts"
# DNS resolvable hostname for Ascender service. This is required for install.
ASCENDER_HOSTNAME: ascender-dev.local
# Namespace for Ascender Kubernetes objects
ASCENDER_NAMESPACE: ascender
# Administrator username for Ascender
ASCENDER_ADMIN_USER: admin
# Administrator password for Ascender
ASCENDER_ADMIN_PASSWORD: "mypassword"
# The image tag indicating the version of Ascender you wish to install
ASCENDER_VERSION: 24.0.1
# The version of the AWX Operator used to install Ascender and its components
ANSIBLE_OPERATOR_VERSION: 2.19.0
# Determines whether to keep the secrets required to encrypt within Ascender (important when backing up)
ascender_garbage_collect_secrets: true
# External PostgreSQL database name used for Ascender (this DB must exist)
ascender_replicas: 1
# The Ascender web container image pull policy (If unsure, choose IfNotPresent)
image_pull_policy: IfNotPresent
# Boolean indicating whether to add standard playbooks into Ascender after installation
ascender_setup_playbooks: false
# Determines whether or not Ledger will be installed
LEDGER_INSTALL: true
# DNS resolvable hostname for Ledger service. This is required for install
LEDGER_HOSTNAME: ledger-dev.local
# Number of replicas for the Ledger web container
ledger_web_replicas: 1
# Number of replicas for the Ledger Parser container
ledger_parser_replicas: 1
# The image tag indicating the version of Ledger you wish to install
LEDGER_VERSION: latest
# The Kubernetes namespace in which Ledger objects will live
LEDGER_NAMESPACE: ledger
# Admin password for Ledger (the username is admin by default)
LEDGER_ADMIN_PASSWORD: mypassword
# Password for Ledger database
LEDGER_DB_PASSWORD: mypassword

[linuxcube]

@michaelford85 , This is what I see in the kubectl logs:

2024-07-14 18:36:11,352 INFO     [-] daphne.server Listening on TCP address 127.0.0.1:8051
2024-07-14 18:36:11,352 INFO     Listening on TCP address 127.0.0.1:8051
2024-07-14 18:36:12,354 INFO spawned: 'nginx' with pid 58
2024-07-14 18:36:12,354 INFO spawned: 'nginx' with pid 58
2024/07/14 18:36:12 [emerg] 58#58: socket() [::]:8052 failed (97: Address family not supported by protocol)
nginx: [emerg] socket() [::]:8052 failed (97: Address family not supported by protocol)
2024-07-14 18:36:12,360 WARN exited: nginx (exit status 1; not expected)
2024-07-14 18:36:12,360 WARN exited: nginx (exit status 1; not expected)
2024-07-14 18:36:15,367 INFO spawned: 'nginx' with pid 59
2024-07-14 18:36:15,367 INFO spawned: 'nginx' with pid 59
2024/07/14 18:36:15 [emerg] 59#59: socket() [::]:8052 failed (97: Address family not supported by protocol)
nginx: [emerg] socket() [::]:8052 failed (97: Address family not supported by protocol)
2024-07-14 18:36:16,373 WARN exited: nginx (exit status 1; not expected)
2024-07-14 18:36:16,373 WARN exited: nginx (exit status 1; not expected)
2024-07-14 18:36:17,375 INFO gave up: nginx entered FATAL state, too many start retries too quickly
2024-07-14 18:36:17,375 INFO gave up: nginx entered FATAL state, too many start retries too quickly
2024-07-14 18:36:18,377 WARN received SIGQUIT indicating exit request
2024-07-14 18:36:18,377 WARN received SIGQUIT indicating exit request
Processing Event: ver:3.0 server:supervisor serial:0 pool:superwatcher poolserial:0 eventname:PROCESS_STATE_FATAL len:62
2024-07-14 18:36:18,378 INFO waiting for superwatcher, uwsgi, daphne, awx-cache-clear, ws-heartbeat to die
2024-07-14 18:36:18,378 INFO waiting for superwatcher, uwsgi, daphne, awx-cache-clear, ws-heartbeat to die
...brutally killing workers...
2024-07-14 18:36:18,377 INFO     [-] daphne.server Killed 0 pending application instances
2024-07-14 18:36:18,377 INFO     Killed 0 pending application instances
2024-07-14 18:36:18,890 WARN stopped: awx-cache-clear (terminated by SIGTERM)
2024-07-14 18:36:18,890 WARN stopped: awx-cache-clear (terminated by SIGTERM)
2024-07-14 18:36:18,890 INFO stopped: ws-heartbeat (exit status 0)
2024-07-14 18:36:18,890 INFO stopped: ws-heartbeat (exit status 0)
2024-07-14 18:36:18,986 INFO stopped: daphne (exit status 0)
2024-07-14 18:36:18,986 INFO stopped: daphne (exit status 0)
worker 1 buried after 1 seconds
worker 2 buried after 1 seconds
worker 3 buried after 1 seconds
worker 4 buried after 1 seconds
worker 5 buried after 1 seconds
binary reloading uWSGI...
chdir() to /var/lib/awx
closing all non-uwsgi socket fds > 2 (max_fd = 1048576)...
found fd 3 mapped to socket 0 (127.0.0.1:8050)
running /var/lib/awx/venv/awx/bin/uwsgi
[uWSGI] getting INI configuration from /etc/tower/uwsgi.ini
*** Starting uWSGI 2.0.21 (64bit) on [Sun Jul 14 18:36:19 2024] ***
compiled with version: 11.4.1 20231218 (Red Hat 11.4.1-3) on 05 June 2024 19:43:51
os: Linux-5.14.0-427.24.1.el9_4.x86_64 #1 SMP PREEMPT_DYNAMIC Sun Jun 23 11:48:35 EDT 2024
nodename: ascender-app-web-7ddd546dc6-5r5bc
machine: x86_64
clock source: unix
detected number of CPU cores: 4
current working directory: /var/lib/awx
detected binary path: /var/lib/awx/venv/awx/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
your memory page size is 4096 bytes
detected max file descriptor number: 1048576
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 inherited INET address 127.0.0.1:8050 fd 3
Python version: 3.9.18 (main, Jan 24 2024, 00:00:00)  [GCC 11.4.1 20231218 (Red Hat 11.4.1-3)]
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x2191f10
your server socket listen backlog is limited to 128 connections
your mercy for graceful operations on workers is 60 seconds
mapped 609552 bytes (595 KB) for 5 cores
*** Operational MODE: preforking ***
*** uWSGI is running in multiple interpreter mode ***
gracefully (RE)spawned uWSGI master process (pid: 16)
spawned uWSGI worker 1 (pid: 60, cores: 1)
spawned uWSGI worker 2 (pid: 61, cores: 1)
spawned uWSGI worker 3 (pid: 62, cores: 1)
spawned uWSGI worker 4 (pid: 63, cores: 1)
spawned uWSGI worker 5 (pid: 64, cores: 1)
mounting awx.wsgi:application on /
mounting awx.wsgi:application on /
mounting awx.wsgi:application on /
mounting awx.wsgi:application on /
mounting awx.wsgi:application on /
2024-07-14 18:36:21,600 INFO waiting for superwatcher, uwsgi to die
2024-07-14 18:36:21,600 INFO waiting for superwatcher, uwsgi to die
WSGI app 0 (mountpoint='/') ready in 2 seconds on interpreter 0x2191f10 pid: 61 (default app)
WSGI app 0 (mountpoint='/') ready in 2 seconds on interpreter 0x2191f10 pid: 63 (default app)
WSGI app 0 (mountpoint='/') ready in 2 seconds on interpreter 0x2191f10 pid: 60 (default app)
WSGI app 0 (mountpoint='/') ready in 2 seconds on interpreter 0x2191f10 pid: 64 (default app)
WSGI app 0 (mountpoint='/') ready in 2 seconds on interpreter 0x2191f10 pid: 62 (default app)
2024-07-14 18:36:25,310 INFO waiting for superwatcher, uwsgi to die
2024-07-14 18:36:25,310 INFO waiting for superwatcher, uwsgi to die
2024-07-14 18:36:28,314 INFO waiting for superwatcher, uwsgi to die
2024-07-14 18:36:28,314 INFO waiting for superwatcher, uwsgi to die
2024-07-14 18:36:29,316 WARN killing 'uwsgi' (16) with SIGKILL
2024-07-14 18:36:29,316 WARN killing 'uwsgi' (16) with SIGKILL
2024-07-14 18:36:29,332 WARN stopped: uwsgi (terminated by SIGKILL)
2024-07-14 18:36:29,332 WARN stopped: uwsgi (terminated by SIGKILL)
2024-07-14 18:36:29,332 WARN stopped: superwatcher (terminated by SIGTERM)
2024-07-14 18:36:29,332 WARN stopped: superwatcher (terminated by SIGTERM)

[linuxcube]

I got to move a little further, now I am getting a different error upon deployment:

sudo ./setup.sh 
Using Inventory File: ./inventory
[WARNING]: Collection azure.azcollection does not support Ansible version
2.14.14
[WARNING]: Collection kubernetes.core does not support Ansible version 2.14.14
[WARNING]: Collection amazon.aws does not support Ansible version 2.14.14
[WARNING]: Collection azure.azcollection does not support Ansible version
2.14.14
[WARNING]: Collection kubernetes.core does not support Ansible version 2.14.14
[WARNING]: Collection amazon.aws does not support Ansible version 2.14.14
[WARNING]: Collection azure.azcollection does not support Ansible version
2.14.14
[WARNING]: Collection kubernetes.core does not support Ansible version 2.14.14
[WARNING]: Collection amazon.aws does not support Ansible version 2.14.14
[WARNING]: Collection azure.azcollection does not support Ansible version
2.14.14
[WARNING]: Collection kubernetes.core does not support Ansible version 2.14.14
[WARNING]: Collection amazon.aws does not support Ansible version 2.14.14
[WARNING]: Found both group and host with same name: localhost
[WARNING]: Collection kubernetes.core does not support Ansible version 2.14.14

PLAY [localhost] *********************************************************************************************************************************************************************************

TASK [Verify that LEDGER_HOSTNAME and ASCENDER_HOSTNAME are different] ***************************************************************************************************************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

PLAY [ascender] **********************************************************************************************************************************************************************************

TASK [Gathering Facts] ***************************************************************************************************************************************************************************
ok: [ascender_host]

TASK [Verify x86_64 architecture] ****************************************************************************************************************************************************************
ok: [ascender_host] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [Verify minimum processor count when Ledger is not being installed] *************************************************************************************************************************
skipping: [ascender_host]

TASK [Verify minimum processor count when Ledger is being installed] *****************************************************************************************************************************
ok: [ascender_host] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [Verify Enterprise Linux OS Family architecture] ********************************************************************************************************************************************
ok: [ascender_host] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [Verify RedHat OS Family architecture] ******************************************************************************************************************************************************
ok: [ascender_host] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [Get mount location of /var directory] ******************************************************************************************************************************************************
changed: [ascender_host]

TASK [Get available disk of /dev/mapper/rhel_rhel9-var] ******************************************************************************************************************************************
ok: [ascender_host] => {
    "ansible_mounts": [
        {
            "block_available": 4395786,
            "block_size": 4096,
            "block_total": 5226496,
            "block_used": 830710,
            "device": "/dev/mapper/rhel_rhel9-root",
            "fstype": "xfs",
            "inode_available": 10364424,
            "inode_total": 10485760,
            "inode_used": 121336,
            "mount": "/",
            "options": "rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota",
            "size_available": 18005139456,
            "size_total": 21407727616,
            "uuid": "79617b77-57c2-4e84-a245-6a71b10fd84c"
        },
        {
            "block_available": 173590,
            "block_size": 4096,
            "block_total": 245760,
            "block_used": 72170,
            "device": "/dev/sda2",
            "fstype": "xfs",
            "inode_available": 523923,
            "inode_total": 524288,
            "inode_used": 365,
            "mount": "/boot",
            "options": "rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota",
            "size_available": 711024640,
            "size_total": 1006632960,
            "uuid": "925e2f9a-19e9-4259-8026-b0fb792f1575"
        },
        {
            "block_available": 462527,
            "block_size": 4096,
            "block_total": 2605056,
            "block_used": 2142529,
            "device": "/dev/mapper/rhel_rhel9-var",
            "fstype": "xfs",
            "inode_available": 3702495,
            "inode_total": 3960696,
            "inode_used": 258201,
            "mount": "/var",
            "options": "rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota",
            "size_available": 1894510592,
            "size_total": 10670309376,
            "uuid": "afa92ff1-d8e9-4ce8-90c6-c55e83d9c63f"
        },
        {
            "block_available": 2561475,
            "block_size": 4096,
            "block_total": 2605056,
            "block_used": 43581,
            "device": "/dev/mapper/rhel_rhel9-home",
            "fstype": "xfs",
            "inode_available": 5232399,
            "inode_total": 5242880,
            "inode_used": 10481,
            "mount": "/home",
            "options": "rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota",
            "size_available": 10491801600,
            "size_total": 10670309376,
            "uuid": "8eee13d2-1954-44ff-bcdc-2f15d07bac1c"
        },
        {
            "block_available": 2578523,
            "block_size": 4096,
            "block_total": 2605056,
            "block_used": 26533,
            "device": "/dev/mapper/rhel_rhel9-tmp",
            "fstype": "xfs",
            "inode_available": 5242863,
            "inode_total": 5242880,
            "inode_used": 17,
            "mount": "/tmp",
            "options": "rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota",
            "size_available": 10561630208,
            "size_total": 10670309376,
            "uuid": "e6cea9ca-76e5-4289-8308-3ab8290cf86a"
        },
        {
            "block_available": 2575064,
            "block_size": 4096,
            "block_total": 2605056,
            "block_used": 29992,
            "device": "/dev/mapper/rhel_rhel9-opt",
            "fstype": "xfs",
            "inode_available": 5242488,
            "inode_total": 5242880,
            "inode_used": 392,
            "mount": "/opt",
            "options": "rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota",
            "size_available": 10547462144,
            "size_total": 10670309376,
            "uuid": "296b79d4-64f0-4989-b52f-9b21f722124b"
        }
    ]
}

TASK [Ensure that free space on /dev/mapper/rhel_rhel9-var is greater than 30GB for k3s offline install] *****************************************************************************************
skipping: [ascender_host] => (item={'mount': '/', 'device': '/dev/mapper/rhel_rhel9-root', 'fstype': 'xfs', 'options': 'rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota', 'size_total': 21407727616, 'size_available': 18005139456, 'block_size': 4096, 'block_total': 5226496, 'block_available': 4395786, 'block_used': 830710, 'inode_total': 10485760, 'inode_available': 10364424, 'inode_used': 121336, 'uuid': '79617b77-57c2-4e84-a245-6a71b10fd84c'}) 
skipping: [ascender_host] => (item={'mount': '/boot', 'device': '/dev/sda2', 'fstype': 'xfs', 'options': 'rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota', 'size_total': 1006632960, 'size_available': 711024640, 'block_size': 4096, 'block_total': 245760, 'block_available': 173590, 'block_used': 72170, 'inode_total': 524288, 'inode_available': 523923, 'inode_used': 365, 'uuid': '925e2f9a-19e9-4259-8026-b0fb792f1575'}) 
skipping: [ascender_host] => (item={'mount': '/var', 'device': '/dev/mapper/rhel_rhel9-var', 'fstype': 'xfs', 'options': 'rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota', 'size_total': 10670309376, 'size_available': 1894510592, 'block_size': 4096, 'block_total': 2605056, 'block_available': 462527, 'block_used': 2142529, 'inode_total': 3960696, 'inode_available': 3702495, 'inode_used': 258201, 'uuid': 'afa92ff1-d8e9-4ce8-90c6-c55e83d9c63f'}) 
skipping: [ascender_host] => (item={'mount': '/home', 'device': '/dev/mapper/rhel_rhel9-home', 'fstype': 'xfs', 'options': 'rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota', 'size_total': 10670309376, 'size_available': 10491801600, 'block_size': 4096, 'block_total': 2605056, 'block_available': 2561475, 'block_used': 43581, 'inode_total': 5242880, 'inode_available': 5232399, 'inode_used': 10481, 'uuid': '8eee13d2-1954-44ff-bcdc-2f15d07bac1c'}) 
skipping: [ascender_host] => (item={'mount': '/tmp', 'device': '/dev/mapper/rhel_rhel9-tmp', 'fstype': 'xfs', 'options': 'rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota', 'size_total': 10670309376, 'size_available': 10561630208, 'block_size': 4096, 'block_total': 2605056, 'block_available': 2578523, 'block_used': 26533, 'inode_total': 5242880, 'inode_available': 5242863, 'inode_used': 17, 'uuid': 'e6cea9ca-76e5-4289-8308-3ab8290cf86a'}) 
skipping: [ascender_host] => (item={'mount': '/opt', 'device': '/dev/mapper/rhel_rhel9-opt', 'fstype': 'xfs', 'options': 'rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota', 'size_total': 10670309376, 'size_available': 10547462144, 'block_size': 4096, 'block_total': 2605056, 'block_available': 2575064, 'block_used': 29992, 'inode_total': 5242880, 'inode_available': 5242488, 'inode_used': 392, 'uuid': '296b79d4-64f0-4989-b52f-9b21f722124b'}) 
skipping: [ascender_host]

TASK [Ensure that free space on /dev/mapper/rhel_rhel9-var is greater than 20GB] *****************************************************************************************************************
skipping: [ascender_host] => (item={'mount': '/', 'device': '/dev/mapper/rhel_rhel9-root', 'fstype': 'xfs', 'options': 'rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota', 'size_total': 21407727616, 'size_available': 18005139456, 'block_size': 4096, 'block_total': 5226496, 'block_available': 4395786, 'block_used': 830710, 'inode_total': 10485760, 'inode_available': 10364424, 'inode_used': 121336, 'uuid': '79617b77-57c2-4e84-a245-6a71b10fd84c'}) 
skipping: [ascender_host] => (item={'mount': '/boot', 'device': '/dev/sda2', 'fstype': 'xfs', 'options': 'rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota', 'size_total': 1006632960, 'size_available': 711024640, 'block_size': 4096, 'block_total': 245760, 'block_available': 173590, 'block_used': 72170, 'inode_total': 524288, 'inode_available': 523923, 'inode_used': 365, 'uuid': '925e2f9a-19e9-4259-8026-b0fb792f1575'}) 
skipping: [ascender_host] => (item={'mount': '/var', 'device': '/dev/mapper/rhel_rhel9-var', 'fstype': 'xfs', 'options': 'rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota', 'size_total': 10670309376, 'size_available': 1894510592, 'block_size': 4096, 'block_total': 2605056, 'block_available': 462527, 'block_used': 2142529, 'inode_total': 3960696, 'inode_available': 3702495, 'inode_used': 258201, 'uuid': 'afa92ff1-d8e9-4ce8-90c6-c55e83d9c63f'}) 
skipping: [ascender_host] => (item={'mount': '/home', 'device': '/dev/mapper/rhel_rhel9-home', 'fstype': 'xfs', 'options': 'rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota', 'size_total': 10670309376, 'size_available': 10491801600, 'block_size': 4096, 'block_total': 2605056, 'block_available': 2561475, 'block_used': 43581, 'inode_total': 5242880, 'inode_available': 5232399, 'inode_used': 10481, 'uuid': '8eee13d2-1954-44ff-bcdc-2f15d07bac1c'}) 
skipping: [ascender_host] => (item={'mount': '/tmp', 'device': '/dev/mapper/rhel_rhel9-tmp', 'fstype': 'xfs', 'options': 'rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota', 'size_total': 10670309376, 'size_available': 10561630208, 'block_size': 4096, 'block_total': 2605056, 'block_available': 2578523, 'block_used': 26533, 'inode_total': 5242880, 'inode_available': 5242863, 'inode_used': 17, 'uuid': 'e6cea9ca-76e5-4289-8308-3ab8290cf86a'}) 
skipping: [ascender_host] => (item={'mount': '/opt', 'device': '/dev/mapper/rhel_rhel9-opt', 'fstype': 'xfs', 'options': 'rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota', 'size_total': 10670309376, 'size_available': 10547462144, 'block_size': 4096, 'block_total': 2605056, 'block_available': 2575064, 'block_used': 29992, 'inode_total': 5242880, 'inode_available': 5242488, 'inode_used': 392, 'uuid': '296b79d4-64f0-4989-b52f-9b21f722124b'}) 
skipping: [ascender_host]

PLAY [localhost] *********************************************************************************************************************************************************************************

TASK [Install prerequisite packages on localhost] ************************************************************************************************************************************************

TASK [common : Retrieve the Enterprise Linux details] ********************************************************************************************************************************************
ok: [localhost]

TASK [common : Ensure /root/ascender-install/playbooks/../ascender_install_artifacts exists] *****************************************************************************************************
ok: [localhost]

TASK [common : Enable EPEL Repo (Online)] ********************************************************************************************************************************************************
ok: [localhost]

TASK [common : Add the Kubernetes yum repository] ************************************************************************************************************************************************
ok: [localhost]

TASK [common : Install Necessary RPMs (if offline)] **********************************************************************************************************************************************
skipping: [localhost]

TASK [common : Install necessary packages for Enterprise Linux] **********************************************************************************************************************************
ok: [localhost]

TASK [common : Install AKS Packages] *************************************************************************************************************************************************************
skipping: [localhost]

TASK [common : Install GKE Packages] *************************************************************************************************************************************************************
skipping: [localhost]

TASK [common : Install EKS Packages] *************************************************************************************************************************************************************
skipping: [localhost]

PLAY [ascender] **********************************************************************************************************************************************************************************

TASK [Run k8s_setup role for k3s] ****************************************************************************************************************************************************************

TASK [k8s_setup : Wait 600 seconds for target connection to become reachable/usable] *************************************************************************************************************
[WARNING]: Reset is not implemented for this connection
ok: [ascender_host]

TASK [k8s_setup : Populate service facts] ********************************************************************************************************************************************************
ok: [ascender_host]

TASK [k8s_setup : Stop firewall service] *********************************************************************************************************************************************************
ok: [ascender_host]

TASK [k8s_setup : Install K3s Online (this may take up to 5 minutes)] ****************************************************************************************************************************
skipping: [ascender_host]

TASK [k8s_setup : Copy k3s Binary] ***************************************************************************************************************************************************************
skipping: [ascender_host]

TASK [k8s_setup : Create images directory] *******************************************************************************************************************************************************
skipping: [ascender_host]

TASK [k8s_setup : Copy images] *******************************************************************************************************************************************************************
[WARNING]: Unable to find '/root/ascender-install/playbooks/../offline/images' in expected paths (use -vvvvv to see paths)
skipping: [ascender_host]

TASK [k8s_setup : Make Installer executable] *****************************************************************************************************************************************************
skipping: [ascender_host]

TASK [k8s_setup : Install K3s Offline] ***********************************************************************************************************************************************************
skipping: [ascender_host]

TASK [k8s_setup : Ensure that ~/.kube directory exists] ******************************************************************************************************************************************
ok: [ascender_host -> localhost]

TASK [k8s_setup : Copy kubeconfig file from default location to the ~/.kube directory"] **********************************************************************************************************
ok: [ascender_host]

TASK [k8s_setup : Replace the kubeconfig cluster kube-api server IP with the public IP address, if the cluster is remote] ************************************************************************
skipping: [ascender_host]

TASK [k8s_setup : Get a list of all nodes] *******************************************************************************************************************************************************
ok: [ascender_host -> localhost]

TASK [k8s_setup : Ensure a local DNS entry for ascender-dev.local exists] ************************************************************************************************************************
changed: [ascender_host -> localhost]

TASK [k8s_setup : Ensure a local DNS entry for ledger-dev.local exists] **************************************************************************************************************************
changed: [ascender_host -> localhost]

TASK [k8s_setup : Ensure a local DNS entry for react.example.com exists] *************************************************************************************************************************
skipping: [ascender_host]

PLAY [localhost] *********************************************************************************************************************************************************************************

TASK [Run ascender_install role for k3s] *********************************************************************************************************************************************************

TASK [ascender_install : Retrieve the current time in order to timestamp files] ******************************************************************************************************************
ok: [localhost]

TASK [ascender_install : Create Namespace] *******************************************************************************************************************************************************
ok: [localhost]

TASK [ascender_install : Generate manifest to install AWX Operator] ******************************************************************************************************************************
changed: [localhost]

TASK [ascender_install : Generate manifest to install AWX Operator with timestamp attached, for purposes of operator deletion later] *************************************************************
changed: [localhost]

TASK [ascender_install : Copy Operator Source] ***************************************************************************************************************************************************
skipping: [localhost]

TASK [ascender_install : Import Operator Images] *************************************************************************************************************************************************
[WARNING]: Unable to find '/root/ascender-install/playbooks/../offline/images' in expected paths (use -vvvvv to see paths)
skipping: [localhost]

TASK [ascender_install : Install AWX Operator with Kustomize] ************************************************************************************************************************************
changed: [localhost]

TASK [ascender_install : delete Kustomization file] **********************************************************************************************************************************************
changed: [localhost]

TASK [ascender_install : Import Ascender Images] *************************************************************************************************************************************************
[WARNING]: Unable to find '/root/ascender-install/playbooks/../offline/images' in expected paths (use -vvvvv to see paths)
skipping: [localhost]

TASK [ascender_install : Wait for Operator deployment to be ready] *******************************************************************************************************************************
ok: [localhost]

TASK [ascender_install : Generate manifest to install AWX ascender-app k8s object with timestamp attached, for purposes of deletion later] *******************************************************
changed: [localhost]

TASK [ascender_install : Apply AWX ascender-app manifest for k3s] ********************************************************************************************************************************
ok: [localhost]

TASK [ascender_install : Wait for ascender-app-web Deployment to complete setting up (this may take up to 10 minutes)] ***************************************************************************
ok: [localhost]

TASK [ascender_install : Set the Ascender URL] ***************************************************************************************************************************************************
ok: [localhost]

TASK [ascender_install : ansible.builtin.debug] **************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "The Ascender API endpoint is http://ascender-dev.local:80/api/v2/ping/"
}

TASK [ascender_install : Wait until Ascender API is Up (This may take between 10-20 mins)] *******************************************************************************************************
ok: [localhost]

TASK [ascender_install : ansible.builtin.debug] **************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "Ascender API is up"
}

TASK [ansible.builtin.debug] *********************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "Ascender install complete."
}

PLAY [localhost] *********************************************************************************************************************************************************************************

TASK [Run ascender_react role for k3s] ***********************************************************************************************************************************************************
skipping: [localhost]

TASK [ansible.builtin.debug] *********************************************************************************************************************************************************************
skipping: [localhost]

PLAY [localhost] *********************************************************************************************************************************************************************************

TASK [Create Namespace] **************************************************************************************************************************************************************************
ok: [localhost]

TASK [Run ascender_install role for k3s] *********************************************************************************************************************************************************

TASK [ledger_install : Retrieve the current time in order to timestamp files] ********************************************************************************************************************
ok: [localhost]

TASK [ledger_install : Import images] ************************************************************************************************************************************************************
[WARNING]: Unable to find '/root/ascender-install/playbooks/../offline/images' in expected paths (use -vvvvv to see paths)
skipping: [localhost]

TASK [ledger_install : Generate manifest to install Ledger k8s object with timestamp attached, for purposes of deletion later] *******************************************************************
changed: [localhost]

TASK [ledger_install : Apply ledger manifest for k3s] ********************************************************************************************************************************************
changed: [localhost]

TASK [ledger_install : Wait for Ledger web Deployment to complete setting up] ********************************************************************************************************************
ok: [localhost]

TASK [ledger_install : Wait for Ledger parser Service to be up] **********************************************************************************************************************************
ok: [localhost]

TASK [ledger_install : ansible.builtin.debug] ****************************************************************************************************************************************************
ok: [localhost] => {
    "parser_svc.resources[0].spec.ports[0].port": "5000"
}

TASK [ledger_install : Set the Ledger URL] *******************************************************************************************************************************************************
ok: [localhost]

TASK [ledger_install : ansible.builtin.debug] ****************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "The Ledger endpoint is http://ledger-dev.local:80"
}

TASK [ledger_install : Wait until Ledger endpoint is Up] *****************************************************************************************************************************************
ok: [localhost]

TASK [ledger_install : ansible.builtin.debug] ****************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "Ledger webpage is up"
}

TASK [ledger_install : Set the Ascender URL and Credentials for HTTPS] ***************************************************************************************************************************
skipping: [localhost]

TASK [ledger_install : Set the Ascender URL and Credentials for HTTP] ****************************************************************************************************************************
ok: [localhost]

TASK [ledger_install : ansible.builtin.debug] ****************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "The Ascender API endpoint is http://ascender-dev.local:80/api/v2/ping/"
}

TASK [ledger_install : Wait until Ascender API is Up] ********************************************************************************************************************************************
ok: [localhost]

TASK [ledger_install : ansible.builtin.debug] ****************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "Ascender API is up"
}

TASK [ledger_install : Set all the logging parameters] *******************************************************************************************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed to get token: HTTP Error 401: Unauthorized", "response": "{\"detail\":\"Authentication credentials were not provided. To establish a login session, visit /api/login/.\"}"}

PLAY RECAP ***************************************************************************************************************************************************************************************
ascender_host              : ok=15   changed=3    unreachable=0    failed=0    skipped=11   rescued=0    ignored=0   
localhost                  : ok=36   changed=7    unreachable=0    failed=1    skipped=11   rescued=0    ignored=0   

ERROR OCCURRED DURING SETUP

[linuxcube]

Team, I was able to sort this out.

Solution: Ensure you have ipv6 enable, its a hard requriement!