search

pipe-cd / pipecd

The One CD for All {applications, platforms, operations}
https://pipecd.dev
Apache License 2.0
1.07k stars 150 forks source link

Piped starts running a queued deployment while another one of the same application is rolling back #3086

Open nghialv opened 2 years ago

nghialv commented 2 years ago

What happened:

What you expected to happen:

How to reproduce it:

Environment:

nghialv commented 2 years ago

Instead of checking the planners and schedulers list to ensure that only one deployment of the sample application is executed at one time, the controller should have a list of executing deployments to check.

https://github.com/pipe-cd/pipecd/blob/master/pkg/app/piped/controller/controller.go#L349 https://github.com/pipe-cd/pipecd/blob/master/pkg/app/piped/controller/controller.go#L574

Hosshii commented 2 years ago

/assign @Hosshii

Hosshii commented 2 years ago

@nghialv Piped deployment is marked as completed when scheduler.Run() function is returned. Because scheduler error is ignored here, If piped failed to send deployment status to control plain, the consistency between control plain and piped is likely to be lost. To fix this, I think it should mark piped completed only when request to control plain succeeded, and retry request to control plain if it failed.