search

orange-cloudfoundry / cf-ops-automation

a collaboration framework for operating cloudfoundry and services at scale
22 stars 6 forks source link

Regression: race condition during deployment delete triggers recreation of delete deployment #146

Open gberche-orange opened 6 years ago

gberche-orange commented 6 years ago

Expected behavior

A commit on paas-secrets is supposed to go through the update-pipeline and remove the associated deleted resources and jobs.

Observed behavior

In pipeline https://elpaaso-concourse-micro.redacted.domain/teams/main/pipelines/coab-depls-generated/jobs/deploy-c_3b693ce6-0cd7-4598-a542-f8d90616f4ca/builds/2

We see a delete deployment git commit which instead triggers a bosh deploy without first going to the update-pipeline

image

This is actually worse than #117, as in this case, the delete deployment git commit is ignored by concourse tasks which recreates the deleted bosh deployment.

Affected release

Reproduced on head (likely 2f10ba87a5ece92e223feceb068f4de1eced7150) as latest released version is version 2.0.0

Traces and logs

o-orand commented 6 years ago

Are you sure the update-pipline hasn't been triggered? Because it's triggered on each change secrets-full without restrictions (https://github.com/orange-cloudfoundry/cf-ops-automation/blob/2f10ba87a5ece92e223feceb068f4de1eced7150/concourse/pipelines/template/depls-pipeline.yml.erb#L197-L202). We probably need to introduce an intermediate secrets branch updated by update-pipeline like we do with paas-templates repo and pipeline-current-((paas-templates-branch)).

gberche-orange commented 6 years ago

Assigning as P3 since the impact is low: a parasite new deployment will be triggered, followed by a new bosh delete