vcsjones / AzureSignTool

SignTool Library and Azure Key Vault Support
MIT License
272 stars 87 forks source link

AzureSignTool hanging during build on Azure #107

Closed cwindle closed 3 years ago

cwindle commented 4 years ago

I have seen this issue happen twice now. Our build usually completes in about 8-10 minutes and signs around 25 files. However, both times that I have seen the problem, the build was still going after 30-40 minutes, and eventually Azure cancelled it due to lack of activity. Looking at the logs it happened on trying to sign the very first file. Here is the logs leading up to and after the problem. Everything happened in the first 3-4 minutes up to the "CMD=AzureSignTool" line, which is exactly what we are sending as the command-line. The Attempting to cancel the build.... happened about 50+ minutes later. This is a rare occurrence, as this is only the second time it happened. I didn't think much about it the first time, which was maybe a week or two ago, but when it happened again in exactly the same place in exactly the same way, I thought it was worth reporting. I have replaced some of the information with "*" for security reasons.

Azure Log Excerpt:

CopyFilesToOutputDirectory: Copying file from "obj\x86\Release\ExpressCheck.Core.dll" to "D:/a/1/s/bin/x86/Release/ExpressCheck.Core.dll". ExpressCheck.Core -> D:\a\1\s\bin\x86\Release\ExpressCheck.Core.dll Copying file from "obj\x86\Release\ExpressCheck.Core.pdb" to "D:/a/1/s/bin/x86/Release/ExpressCheck.Core.pdb". PostBuildEvent: call D:\a\1\s\signtool\signcode.bat "D:/a/1/s/bin/x86/Release/ExpressCheck.Core.dll" Signing File: "D:/a/1/s/bin/x86/Release/ExpressCheck.Core.dll" "CMD=AzureSignTool sign -fd sha256 -td sha256 -kvu https://**************.vault.azure.net/ -kvi ****----**** -kvs * -kvc ** -tr http://timestamp.comodoca.com -v "D:/a/1/s/bin/x86/Release/ExpressCheck.Core.dll" Attempting to cancel the build... Done Building Project "D:\a\1\s\ExpressCheck.sln" (Rebuild target(s)) -- FAILED. C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\15.0\Bin\Microsoft.Common.CurrentVersion.targets(5165,5): warning MSB5021: Terminating the task executable "cmd" and its child processes because the build was canceled. [D:\a\1\s\source\ExpressCheck.Core\ExpressCheck.Core.csproj] Done Building Project "D:\a\1\s\source\ExpressCheck.Core\ExpressCheck.Core.csproj" (Rebuild target(s)) -- FAILED.

Build FAILED.

"D:\a\1\s\ExpressCheck.sln" (Rebuild target) (1) -> "D:\a\1\s\source\ExpressCheck.Core\ExpressCheck.Core.csproj" (Rebuild target) (2) -> (PostBuildEvent target) -> C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\15.0\Bin\Microsoft.Common.CurrentVersion.targets(5165,5): warning MSB5021: Terminating the task executable "cmd" and its child processes because the build was canceled. [D:\a\1\s\source\ExpressCheck.Core\ExpressCheck.Core.csproj]

1 Warning(s)
0 Error(s)

Time Elapsed 00:56:34.91

[error]The operation was canceled.

Finishing: Running Build Script

cwindle commented 4 years ago

Just wanted to give an update on this issue. After only seeing it twice over the last maybe two months, today I was doing a build and the same thing has happened 3 times in a row, where the first executable that is built in our build script is trying to sign it with AzureSignTool and it is just sitting there with no error or success until I either manually cancel the build or Azure eventually times out from inactivity. So, I am going to go try it a 4th time, and then going to have to revert back to building on non-azure system to get this build done.

cwindle commented 4 years ago

Update - I tried the same exact build script we have running on Azure on a standalone PC and it is working properly, so it could be something to do with the current VMs that we are using Azure with. The version of AzureSigntool we are using is 2.0.17.

cwindle commented 3 years ago

So, I think I finally figured out the problem is due to the Azure build systems. Sometimes, they have the proper .NET Core SDK installed already and sometimes they don't. When they don't, it installs the SDK, but indicates you have to restart a new shell. So, I think I have it worked out now by pre-installing the SDK everytime.

vcsjones commented 3 years ago

So, I think I have it worked out now by pre-installing the SDK everytime.

Sounds like everything is working then. If not, please feel free to re-open the issue.