search

bcgov / cas-pipeline

A collection of make functions used to compose pipelines
Apache License 2.0
0 stars 0 forks source link

oc start-build needs '--wait' to prevent timeout if build is pending #6

Closed matthieu-foucault closed 5 years ago

matthieu-foucault commented 5 years ago

If another build is running, or the new pod takes a long time to start for whatever reason, oc start-build --follow will timeout, and output the following:

Failed to stream the build logs - to view the logs, run oc logs build/your-build-instance
Error: unable to stream the build logs; caused by: unable to wait for build your-build-instance to run: timed out waiting for the condition

Adding the --wait option fixes that issue. This was tested ad-hoc with a new build that had to wait several minutes before starting.

matthieu-foucault commented 5 years ago

Somehow --wait does wait and streams all the logs, but the error is printed et the end anyway, so I'm not sure that is the right solution...

matthieu-foucault commented 5 years ago

As per the discussion in https://chat.pathfinder.gov.bc.ca/channel/devops-how-to?msg=99XhYDxRs5uM8hXfR, either using a parallel build policy (which I see you're using in cas-postgres) or creating a build config per branch and using the SerialLatestOnly policy are two workarounds for this

wenzowski commented 5 years ago

Yes, I leaned towards setting parallel to overcome this, as Circle will attempt a build parallelism of 4 by default. Another option if this proves problematic is to build on our OKD 3.11 cluster on CI and then push to the pathfinder OCP 3.11 cluster

matthieu-foucault commented 5 years ago

When using the parallel strategy, isn't there a risk of having the wrong version being deployed? Assuming that two commits are pushed in quick succession, and the second build is quicker than the first one, then the older commit would end up being deployed last wouldn't it?

wenzowski commented 5 years ago

This is the nice thing about splitting off the CI process from the CD process. Nothing gets deployed by the Continuous Integration flow so parallel builds on CI are safe.

The Continuous Deployment system is responsible for monitoring the commit activity on a single git branch and deploying that branch to a single namespace, so either

  1. only the more recent of the two passing commits would end up being deployed, even though both images (& sha1 tags) would have been pushed to the imagestream, or
  2. both commits would be deployed and the 2nd deploy would be blocked until the first deploy was complete
matthieu-foucault commented 5 years ago

Got it! I believe this PR can be closed then.