search

microsoft / azure-pipelines-tasks

Tasks for Azure Pipelines
https://aka.ms/tfbuild
MIT License
3.47k stars 2.6k forks source link

PublishSymbols fails after Windows Update #13491

Closed lg2de closed 2 years ago

lg2de commented 4 years ago

Required Information

Question, Bug, or Feature?: Bug Enter Task Name: PublishSymbols

Environment

Issue Description

The builds using PublishSymbols task were running fine until installation Windows updates. Now the step fails with Encountered Win32 error '203' from method 'SymEnumSourceFiles'.

Similar issues exist but already closed without helpful information:

Please note, the waning on dbghelp.dll is very confusing because expected and actual path are identical!

Task logs

1_Initialize Job.txt 14_Publish symbols.txt

Error logs

##[warning]Library dbghelp.dll is already loaded from an unexpected path. Expected 'C:\build\_work\_tasks\PublishSymbols_0675668a-7bba-4ccb-901d-5ad6554ca653\2.0.16\dbghelp.dll'. Actual 'C:\build\_work\_tasks\PublishSymbols_0675668a-7bba-4ccb-901d-5ad6554ca653\2.0.16\dbghelp.dll'.
##[error]Encountered Win32 error '203' from method 'SymEnumSourceFiles'.
johnterickson commented 3 years ago

@lg2de Are you running the 32-bit version of the agent or the 64-bit version of the agent? (For comparison to #9139)

johnterickson commented 3 years ago

@lg2de Also can you confirm that if you uninstall the KB, then this error goes away?

isnits commented 3 years ago

Hello, The same issue here. OS: Windows Server 2019 Standard Build 1809 Build agent: x64 version 2.153.1 installed on x64 OS Tried with V1 and V2 Publish Symbols tasks Mentioned updates are not installed in the system Error: [warning]Library dbghelp.dll is already loaded from an unexpected path. Expected 'C:\Builds_tasks\PublishSymbols_0675668a-7bba-4ccb-901d-5ad6554ca653\2.0.16\dbghelp.dll'. Actual 'C:\Builds_tasks\PublishSymbols_0675668a-7bba-4ccb-901d-5ad6554ca653\2.0.16\dbghelp.dll'.

[error]Encountered Win32 error '203' from method 'SymEnumSourceFiles'.

lg2de commented 3 years ago

@johnterickson , initially I already noted that x64 is running. I did not try to deinstall any windows update (KB?). This is the only change we noticed in the time range.

M463 commented 3 years ago

I spend some serious time on this issue as well. After failing to uninstall any of the mentioned KB (KB4570505, KB4565349, KB4566424) updates via wusa and experimenting with several versions of the task, I was eventually able to fix the problem by removing McAfee antivirus from the affected server. It has since been replaced with a competitors product but unfortunately I'm not allowed to share the name of the product here due to company policies.

Zanadar commented 3 years ago

I am also experiencing this issue... can anyone confirm that removing a particular windows update resolves the issue? Doesn't seem like anything else changed on the system in question.

gwill1010 commented 3 years ago

I have the exact same issue too. Please respond.

github-actions[bot] commented 3 years ago

This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days

lg2de commented 3 years ago

Disabling McAfee does not solve the problem.

von-seydlitz commented 3 years ago

Error still exists on Windows 2019 systems. Any Updates on this?

dan-griffin commented 3 years ago

We also have the same issue. Server 2019, x64 agent. Error on previous agent versions and latest 2.189.0. Last success publish was Jan 27. We didnt do another build until last week. Thats when the error started for us. Tried restarting service, restarting server, upgrading agent, etc.

patrickchongty commented 3 years ago

We also have the same issue. Server 2019, x64 agent. After reviewing @M463 solution above (as we have Mcafee installed on our server as well), we have tested disabling McAfee AV on the server and have identified McAfee as the culprit. However we cannot replace it with any other products so anyone knows what AV exclusions needs to be in place?

gbutler202 commented 3 years ago

We've been suffering with this issue as well. Server 2019, x64 agent, McAfee AV. We liaised with the AV vendor and with their help were able to identify in our case the Antimalware Scan Interface (AMSI) feature under On-Access Scan was the culprit. We tried OAS folder exclusions (to which we added the agent's base folder path) but that wasn't sufficient as the process touches bits of Powershell, .Net and system libs. Disabling AMSI or uninstalling AV protection entirely is what resolved it for us.

nick-gallimore commented 2 years ago

I came across this issue recently when experimenting with this task and spent some time digging into the problem.

Environment: running a self-hosted build agent on a Windows 10 1809 VM (it’s an agent I use for testing stuff, stubbornly refuses to upgrade to a newer version of Windows 10, when I get time I’m going to blow it away and start again) agent version is 2.181.2 with Publish Symbols task version 2.172.0, connecting to an on-prem Azure DevOps Server 2020.1.1. All machines mentioned are running x64 OS.

What I found was there are 3 problems:

  1. The warning about “Library dbghelp.dll is already loaded from an unexpected path” is formatted incorrectly. I stared at the PowerShell script for ages, checked comparisons weren’t case-sensitive (they’re not, and the two paths are the same case anyway, but still), etc, until I thought to check the localized string value, which is: "Library dbghelp.dll is already loaded from an unexpected path. Expected '{0}'. Actual '{0}'." Note the double use of format argument 0, and the absence of format argument 1. The actual path is (for me) “C:\WINDOWS\SYSTEM32\dbghelp.dll”.
  2. The Win32 error code being reported (203 - ERROR_ENVVAR_NOT_FOUND) is incorrect. The PowerShell script calls SymEnumSourceFilesWrapper, which is a short method in an inline C# class in the script. This in turn P/Invokes out to SymEnumSourceFiles and returns the Boolean result. If that returns false, it throws a message formatted with the New-Win32ErrorMessage scripting function. This function calls the scripting function Get-LastWin32Error, which calls System.Runtime.InteropServices.Marshal.GetLastWin32Error to finally get the Win32 error, and it returns the 203 error code. However, this is error code is unrelated to the call to SymEnumSourceFiles, and the usage of GetLastWin32Error is contrary to the correct usage of GetLastError. As the doco states: “You should call the GetLastError function immediately when a function's return value indicates that such a call will return useful data.” If a call is made to GetLastWin32Error immediately after the failure in SymEnumSourceFilesWrapper, it returns an error code of 5 (good old ERROR_ACCESS_DENIED). PowerShell must be making at least one (probably quite a few) calls to Win32 functions between the return from SymEnumSourceFilesWrapper and the eventual call to GetLastWin32Error.
  3. Trying to get something that could be debugged more easily, I converted the PowerShell script to C#. Of course, it ran fine on my dev box (Win10 21H1). Then I tried running it directly on the build agent and it failed with what I can only assume is the same access violation that happens when the task runs. The call stack is shown below. Curious about the warning message that mentioned dbghelp.dll, I tried copying the version of that DLL that ships with the task next to my test EXE, and low and behold the EXE worked and printed out all of the source files referenced in the PDB. So, it seems that certain versions of dbghelp.dll are problematic when it comes to enumerating the source files in a PDB file. I tried with multiple PDB files and got consistent results (didn’t work with the OS version of the DLL, worked with the task version of the DLL). Call stack of access violation: 
    dbghelp.dll!EnumRemappedSourceFiles(struct _PROCESS_ENTRY *,struct _MODULE_ENTRY *,unsigned short const *,struct _CALLBACKDATA *)   Unknown
dbghelp.dll!modEnumSourceFiles(struct _PROCESS_ENTRY *,struct _MODULE_ENTRY *,unsigned long,unsigned long,unsigned short const *,unsigned __int64,struct _CALLBACKDATA *,unsigned long) Unknown
dbghelp.dll!ModLoop(struct _PROCESS_ENTRY *,unsigned __int64,unsigned short const *,unsigned short const *,unsigned long,unsigned __int64,unsigned long,struct _CALLBACKDATA *,unsigned long,int (*)(struct _PROCESS_ENTRY *,struct _MODULE_ENTRY *,unsigned long,unsigned long,unsigned short const *,unsigned __int64,struct _CALLBACKDATA *,unsigned long))  Unknown
dbghelp.dll!EnumSourceFiles(void *,unsigned __int64,unsigned short const *,struct _CALLBACKDATA *)  Unknown
dbghelp.dll!SymEnumSourceFilesExW(void *,unsigned __int64,unsigned short const *,struct _IMAGEHLP_SOURCEREMAPPINGTOOLPREFS *,int (*)(struct _SOURCEFILEW *,void *),void *)  Unknown
dbghelp.dll!SymEnumSourceFilesW ()  Unknown
[Managed to Native Transition]  
>   PdbReader.exe!PdbReader.NativeMethods.SymEnumSourceFilesWrapper(System.IntPtr hProcess, ulong ModeBase, System.Collections.Generic.List<string> referencedSourceFiles) Line 39  C#

After all of that, I tried running the task on one of our production build agents and it worked fine. It’s running Windows 10 2004. It still gets the warning about the DLL being loaded from the unexpected path, but it’s a different version of the DLL (confirmed, checked the DLL in the OS on both the original VM that fails and this machine).

Hope this helps someone.

github-actions[bot] commented 2 years ago

This issue is stale because it has been open for 180 days with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days