dmitryserbin / azdev-release-orchestrator

Azure DevOps extension to manage and orchestrate release pipelines
MIT License
23 stars 13 forks source link

ReleaseOrchestrator stucked in "in progress" state on MAIN Release #9

Closed kboro closed 5 years ago

kboro commented 5 years ago

Hello,

Frequently ReleaseOrchestrator stucked in "in progress" state on MAIN Release (the Release where this task is added to).

In Release report console I can see that triggered Release has been finished successfully :

2019-03-20T08:06:36.9903054Z Release automatically started as stages deployment conditions are met 2019-03-20T08:07:17.4058074Z Stage (15677) deployment completed with status 2019-03-20T08:07:17.4059090Z Phase completed with status

but "MAIN" Release which triggers this external Release stucked at:

image

and when Timeout occurs MAIN Release just fails.

dmitryserbin commented 5 years ago

Please advise VSTS/AzDev version you're running (assuming it's on-prem from the screenshot)? The task might not be fully comparable with your version as API interfaces are different in old versions.

Please provide some more details, if possible? Maybe try enabling debug mode and re-running the pipeline.

kboro commented 5 years ago

Its TFS 2018 Update.3

Please note this issue happens randomly - sometimes when triggered Release is finished then Release Orchestrator task is being finished immediately.

Sometimes tho it waits for even 10 minutes after triggered Release is finished and then switch to "Successfull"

walczak00 commented 5 years ago

We're using azure devops hosted version and running into the same issue.

dmitryserbin commented 5 years ago

What task version are you running guys?

kboro commented 5 years ago

1.2.0

dmitryserbin commented 5 years ago

Could you please provide logs from the failed run?

kboro commented 5 years ago

There are no failures, its more related with delay between sending results.

In Release (A) we have “Release Orchestrator” task which triggers Release (B). When Release (B) is finished we expect that “Release Orchestrator” task on Release (A) is immediately finished as well.

Currently (it happens randomly) Release (A) “ReleasePrchestrator” task is finished successfully but several minutes after Release (B) is finished what generates delays in Release (A) pipeline

dmitryserbin commented 5 years ago

I will add debug mode support to the task to help troubleshooting this kind of issues. Also, some logs from successful run (A) will help me to understand the problem better.

dmitryserbin commented 5 years ago

Please give it another go using v1.2.1 with enabled debug mode and share the logs. To enable debug, edit release and add variable called DEBUG with value release-orchestrator:* (see readme for reference).

kboro commented 5 years ago

I have a debug log. Just to clarify I have added my current Task and Pipeline configuration:

RELEASE (A) Orchestrator Task configuration:

image


Release (B) Pipeline:

image

According to LOG, ReleaseOrchestrator task is waiting for APPROVAL to Deploy to STAGE environment but as you see on ReleasePipeline above, STAGE environment should be triggered manually. Since it never get Stage Deployment approval then ReleaseORchestrator is waiting until reach Task timeout.

It should create RELEASE-B, Deploy to QA env (per pipeline configuration) and finished.

Debug-Log:

Release-log.txt

dmitryserbin commented 5 years ago

Thanks for the update.

According to the screenshots provided STAGE will never be triggered as it doesn't have any automated triggers configured in the pipeline. In order to deploy this manual stage you need to either enable Filter Definition Stages parameter and specific all target stage names (i.e. QA,STAGE) OR configure automated triggers in the pipeline (after release or after stage).

Please let me know if that works.

dmitryserbin commented 5 years ago

Also, after looking into the logs, it feels like there's a few things I can potentially improve for 'new release' scenarios with manual stages.

I will keep you posted.

kboro commented 5 years ago

enable "Filter Definition Stages" is working great ! Issue has been solved,

Thank you

dmitryserbin commented 5 years ago

Great! I will make some user output improvements in the task and update docs to make it easier to understand.