The Deployment "v4m-grafana" is invalid: spec.template.spec.containers[0].imagePullPolicy: Unsupported value : "__IMAGE_PULL_POLICY__"

[zabderra]

Hello,

We're encountering an issue while deploying the monitoring solution for a customer using Viya 2024.04. This pertains to a host-based deployment Version 1.2.25 (14MAY2024).

It's worth noting that the customer has previously run this project with a different Viya cadence and an older monitoring setup.

The initial symptom was the unavailability of the monitoring URL, accompanied by the error message "Application is not available". Upon inspecting dependencies, we discovered that the route to monitoring wasn't created. By enabling HELM_DEBUG=true and executing bash -x ./monitoring/bin/deploy_monitoring_openshift.sh, we obtained the following series of logs at the end of the script execution:

The Deployment "v4m-grafana" is invalid: spec.template.spec.containers[0].imagePullPolicy: Unsupported value: "__IMAGE_PULL_POLICY__": supported values: "Always", "IfNotPresent", "Never"
++ errexit_msg
++ '[' -o errexit ']'
+++ basename ./monitoring/bin/deploy_monitoring_openshift.sh
++ log_error 'Exiting script [deploy_monitoring_openshift.sh] due to an error executing the command [kubectl patch deployment -n $MON_NS v4m-grafana --patch "$(cat $grafanaProxyPatchYAML)"].'
++ '[' true = true ']'
++ level='ERROR '
++ '[' true = true ']'
++ echo -e '\033[1;37m\033[41mERROR Exiting script [deploy_monitoring_openshift.sh] due to an error executing the command [kubectl patch deployment -n $MON_NS v4m-grafana --patch "$(cat $grafanaProxyPatchYAML)"].\033[0m'
ERROR Exiting script [deploy_monitoring_openshift.sh] due to an error executing the command [kubectl patch deployment -n $MON_NS v4m-grafana --patch "$(cat $grafanaProxyPatchYAML)"].
+ cleanup
+ KEEP_TMP_DIR=false
+ '[' false '!=' true ']'
+ rm -rf /tmp/sas.mon.HYHigSok
+ log_debug 'Deleted temporary directory: [/tmp/sas.mon.HYHigSok]'
+ '[' false = true ']'

Any insights into the root cause of this issue would be greatly appreciated.

Kind regards

[gsmith-sas]

Unfortunately, those messages don't provide enough information to understand exactly what went wrong. The mention of the value "IMAGE_PULL_POLICY" is interesting. This is a placeholder value in a file that should be replaced earlier in the processing. There is a defined default value, so it is very odd that the placeholder value persisted. That suggests the problem occurred earlier in the processing. There may have been additional messages generated that might have been helpful.

I have done some additional testing today to confirm that the deploy_monitoring_openshift.sh script (version 1.2.25) runs successfully in our test OpenShift environments with default values. You mentioned that there was an earlier version of SAS Viya Monitoring deployed onto this cluster earlier. Was that version removed prior to deploying the 1.2.25 version? Or, was this an upgrade-in-place? Please let us know what earlier version was in place if this was an upgrade. Were you providing any customizations or configuration tweaks using files in a USER_DIR directory? If so, please share more details about those customizations or environment variable settings. (Please note that we are only interested in the customizations and environment variable settings related to the SAS Viya Monitoring deployment and not those related to the SAS Viya deployment.)

If you set the environment variable LOG_DEBUG_ENABLE to 'true', additional DEBUG-level log messages will be emitted as the script executes. Those could be very helpful in determining why things are failing. Please share the entire set of log messages generated by the deploy_monitoring_openshift.sh script if the problem persists. If you are a SAS employee, feel free to share these with me via internal e-mail.

Regards, Greg Smith