uselagoon / lagoon

Lagoon, the developer-focused application delivery platform
https://docs.lagoon.sh/
Apache License 2.0
567 stars 148 forks source link

Pre-rollout tasks fail when a CLI deployment cannot be scaled #2695

Closed seanhamlin closed 2 years ago

seanhamlin commented 3 years ago

Describe the bug

This was noticed on a site that was attempting to do an Openshift 3.11 deployment, but the images for the CLI deployment could not be pulled (for some reason).

+++ cat .lagoon.yml
--
  | +++ shyaml get-value tasks.pre-rollout.0.run.shell sh
  | ++ SHELL=bash
  | ++ . /oc-build-deploy/scripts/exec-pre-tasks-run.sh
  | ++++ oc -n blah-master get deploymentconfigs --no-headers=true -o name -l service=cli
  | ++++ wc -l
  | +++ [[ 1 -gt 0 ]]
  | ++++ oc -n blah-master get deploymentconfigs -l service=cli -o name
  | +++ DEPLOYMENTCONFIG=deploymentconfig.apps.openshift.io/cli
  | ++++ oc -n blah-master get deploymentconfig.apps.openshift.io/cli -o go-template '--template={{.status.readyReplicas}}'
  | +++ [[ <no value> = \<\n\o\ \v\a\l\u\e\> ]]
  | +++ oc -n blah-master scale --replicas=1 deploymentconfig.apps.openshift.io/cli
  | deploymentconfig.apps.openshift.io/cli scaled
  | +++ sleep 3
  | ++++ oc -n blah-master get deploymentconfig.apps.openshift.io/cli -o go-template '--template={{.status.readyReplicas}}'
  | +++ [[ <no value> = \<\n\o\ \v\a\l\u\e\> ]]
  | +++ oc -n blah-master scale --replicas=1 deploymentconfig.apps.openshift.io/cli
  | deploymentconfig.apps.openshift.io/cli scaled
  | +++ sleep 3

This results in an endless loop.

To Reproduce

Steps to reproduce the behavior:

  1. Deploy a site, add a pre-rollout task that depends on the CLI deployment
  2. Remove the CLI images from the image registry
  3. Deploy again
  4. See endless loop

Expected behavior

Lagoon should deal with this edge case more elegantly. Either, skip the pre-rollout task, or at least have a set number of loops before dying.

seanhamlin commented 2 years ago

This is no longer relevant in 2022. The build deploy controller now has a finite number of iterations before it gives up and fails the build.