search

rfennell / AzurePipelines

A single Repo to contain the source for ALL my Azure DevOps Build/Release Extensions. The packages can be found in the Azure DevOps Marketplace
https://marketplace.visualstudio.com/search?term=fenn&target=VSTS&sortBy=Relevance
MIT License
346 stars 433 forks source link

Wrap More in Retry Logic #1544

Closed nnnvvvcv closed 1 year ago

nnnvvvcv commented 1 year ago

Azure DevOps Extensions

WIKI Updater

Platform

Azure DevOps Services

Azure DevOps Server (TFS) Version

No response

Extension Version

No response

Describe the bug

We have quite a concurrent pipeline with a single build/test followed by a large set of deployment stages running in parallel. Part of the deployment stages we use the Release Notes Generator task followed by the WikiUpdater task.

Because these deployments are happening in parallel occasionally we hit the scenario where two WikiUpdater tasks are running around the same time on the same wiki repo.

One of the tasks succeeds but the other fails with:

Cloned dev.azure.com/<org>/<project>/_git/release_notes to /home/vsts/work/1/s\repo
Set GIT values in /home/vsts/work/1/s\repo
Pull in case of post clone updates from other users
Working file name is __latest.md
Created the '__latest.md' in '/home/vsts/work/1/s\repo/SIT/SFI' - fixing line-endings
Added SIT/SFI/__latest.md to repo /home/vsts/work/1/s\repo
Committed file "/home/vsts/work/1/s\repo" with message "SFI SIT Release notes from build [60243]" as SHA 81f8e9f
Attempt 1 - Push to dev.azure.com/<org>/<project>/_git/release_notes
Push failed, probably due to target being updated completed, will retry up to 32 times
Pull to get updates from other users
##[error]Error: Need to specify how to reconcile divergent branches.
##[error]Error: Need to specify how to reconcile divergent branches.

Looking at the code - the retry is around only the git push command - but because the remote repo has been updated by a parallel task it is the pull which is failing. I think this could be fixed by using a rebase rather than pull when iterating through the retries.

The line in question is:

https://github.com/rfennell/AzurePipelines/blob/70073f554776d99a3f24921a7c3fe6724253172e/Extensions/WikiUpdater/WikiUpdaterTask/src/GitWikiFuntions.ts#L320

Repo Steps

Run tasks in parallel against the same wiki repo

Expected Behavior

All parallel tasks complete (albeit with possibly retries)

Logging Information

Cloned dev.azure.com///_git/release_notes to /home/vsts/work/1/s\repo Set GIT values in /home/vsts/work/1/s\repo Pull in case of post clone updates from other users Working file name is latest.md Created the 'latest.md' in '/home/vsts/work/1/s\repo/SIT/SFI' - fixing line-endings Added SIT/SFI/__latest.md to repo /home/vsts/work/1/s\repo Committed file "/home/vsts/work/1/s\repo" with message "SFI SIT Release notes from build [60243]" as SHA 81f8e9f Attempt 1 - Push to dev.azure.com///_git/release_notes Push failed, probably due to target being updated completed, will retry up to 32 times Pull to get updates from other users

[error]Error: Need to specify how to reconcile divergent branches.

[error]Error: Need to specify how to reconcile divergent branches.

rfennell commented 1 year ago

I think the best option is to add a parameter so you can pick a pull or a rebase, leaving pull as the default so it is not a breaking change for other uses.

I will try to get that out this week

nnnvvvcv commented 1 year ago

Works like a dream. Thanks for the quick turnaround!

rfennell commented 1 year ago

Wow, that's great news. I had expected more would be needed. A nice surprise for Monday