Triggering task hangs or gets information about the completion with very long delay

[1Siger1]

We are using extensively this task to trigger nightly builds on Azure DevOps. Lately there is problem that the tasks do not complete even when the triggered build completed with success. There are also long delays between completion of the child build and getting information by the triggerer. Refresh time for all triggered builds is set to 60 seconds. Build time for all triggered builds is shorter than 25 minutes. Here is example image:

1. Normaly behaving triggered build - completed without problems
2. To long wait time even when there is 60 seconds refresh time set. Build which was triggered ended in about 18 minutes, still the task waited almost 40 minutes.
3. Hanged trigger. Called build completed in 14 minutes and 39 seconds. This information was never propagated to parent task.

Point 2 is something with which we can live with. Point 3 should never happen. I know that we can set timeout, but in the end still the build will fail.

@huserben do you see any way how this can be fixed?

[huserben]

Hi @1Siger1

this is indeed strange behavior. When you say lately the problem started happening, is it possible to narrow it down to a certain time? Whether it might have been introduced with some update?

I do not remember having done any impactful changes of late so I'm wondering whether it's the task itself or something in the infrastructure/azure devops?

From your image there seems to be an error when sending the request. That feels like the backend somehow did not respond properly. Is there any kind of error or meaningful log to be seen in build 2 or your image?

[Iridias]

I encountered the same problem yesterday (using TriggerBuild version 3.1.7). The triggered build finished after 2 Minutes, but 20 Minutes later, the task was still waiting! (Refreshtime was set to 10 seconds) The last message in the log is:

Waiting for builds to finish - This might take a while...

I canceled the build at this time, to not block the build-agent any longer.

[huserben]

Hi @Iridias

did this just happen "out of the blue" or did you do some update (e.g. of the task, of your Azure DevOps Server in case you are using it or anything else). Or was it ever working for you?

In the last days/weeks I have observer sometimes some issues with Azure DevOps which might be related to high traffic in relation to the current situation - it might be that something is wrong there.

I tested it quickly before and it was still working ok - however it's kind of hard to have a proper test with for example a very long running build...

[1Siger1]

Hi @huserben,

I am monitoring my configurations and I can say that this problems are totally random. My builds are most of the time started little after midnight so there shouldn't be big strain on Azure DevOps resources. I turned on system.debug parameter to get more information when the problem reappears again. I will get back when I get more logs. Thanks for your time.

[huserben]

Hi @1Siger1 @Iridias

as it's been a few months since the last message, did this ever happen again?

[Iridias]

To be honest, I replaced it with a shell-script, that uses azure-cli (its available by default in the bash-build-task). And to make sure, that those calls won't hang as well, I prepended them with "timeout": https://www.gnu.org/software/coreutils/timeout Like so:

cto="10s" # default command time-out
timeout ${cto} az pipelines build queue -p "${project}" --definition-name "${pipelineName}" -o tsv

Of course with more checks, wait-loop and so on. (And of course you have to login in the script first via az login -u ... - you may want to specify the credentials for it via azure-key-vault and corresponding task)

[huserben]

Ok, thanks for sharing your solution here.

I think with the Azure CLI a lot of things got quite easy to automate without the task :)

I'll close the issue then for the time being.