search

rhpds / bookbag-aro-mobb

Bookbag for the MOBB ARO Workshop
Apache License 2.0
4 stars 15 forks source link

Observability (Azure Files integration) lab has issues #12

Open augustrh opened 8 months ago

augustrh commented 8 months ago

After following all the steps to prep the logs for export to Azure files the steps to view them in Grafana do not show any results. Instead the Grafana dashboards are empty (even after 15-20 mins) and display "no data" and a red warning box reappears a few times stating: "Templating ... Template variable service failed 404 page not found" (see attachment to this issue for visual).

Additionally, step 4 using the explore icon asks you to select "Loki (Application)" from the drop down but it's not present. (see attachment) Screenshot 2024-01-10 at 14 06 26 Screenshot 2024-01-10 at 14 08 12

gmidha1 commented 8 months ago

After 2 to 3 mins, dashboard gets populated with data.

Screenshot 2024-01-12 at 14 32 24
augustrh commented 7 months ago

Took a bit longer for me but yes, it worked! All good.

augustrh commented 7 months ago

Except the Loki bit doesn't work. So reopening.

augustrh commented 7 months ago

Ok, Loki not installed correctly.

[azure@bastion-gbzf5 ~]$ oc get pods -n openshift-logging | grep -i pending logging-loki-gateway-8694445477-lbwcv 0/2 Pending 0 28m logging-loki-index-gateway-1 0/1 Pending 0 27m logging-loki-ingester-1 0/1 Pending 0 27m logging-loki-query-frontend-57dd89dcf5-9f8s8 0/1 Pending 0 28m logging-loki-query-frontend-57dd89dcf5-bfjjj 0/1 Pending 0 28m

Events: Type Reason Age From Message

Warning FailedScheduling 28m default-scheduler 0/6 nodes are available: 1 Insufficient memory, 3 Insufficient cpu, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/6 nodes are available: 3 No preemption victims found for incoming pod, 3 Preemption is not helpful for scheduling.. Warning FailedScheduling 12m (x7 over 24m) default-scheduler 0/6 nodes are available: 1 Insufficient memory, 3 Insufficient cpu, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/6 nodes are available: 3 No preemption victims found for incoming pod, 3 Preemption is not helpful for scheduling..

looks like we are using Standard_D4s_v3 - which is https://azureprice.net/vm/Standard_D4s_v3

16GB RAM and 4 VCPU

augustrh commented 7 months ago

The demo listing in the RH docs suggests any size will do, but there is clearly a taint associated with it keeping it off certain sized nodes. I'll try and see if I can work out what.

augustrh commented 7 months ago

And masters are Standard_D8s_v3 https://azureprice.net/vm/Standard_D8s_v3 32GB and 8vCPU

augustrh commented 7 months ago

Adding 2x Standard_D4s_v3 resolved the issue. Total of 32GB RAM and 8 VCPU.