KERNELBASE.dll exception with missing method exception VssEnvironment.EnumerateVssPaths()

erwinkramer commented 7 years ago

Have you try trouble shooting doc?

Yes.

Agent version and platform

v2.114.0
v2.115.0
v2.116.1

Platform:

Windows Server 2016
VS2015 Professional (Update 3)

VSTS type and version

VisualStudio.com

What's not working?

tf.exe in the vstsom folder from the installed local agent.

See following when i try to run tf.exe manually:

Event log entry: Faulting application name: TF.exe, version: 15.108.25929.0, time stamp: 0x583e1c41 Faulting module name: KERNELBASE.dll, version: 10.0.14393.1066, time stamp: 0x58d9f07f Exception code: 0xe0434352 Fault offset: 0x000da932 Faulting process id: 0x19f0 Faulting application start time: 0x01d2c7384050cee8 Faulting application path: C:\Users\Administrator\agent\externals\vstsom\TF.exe Faulting module path: C:\Windows\System32\KERNELBASE.dll Report Id: 6095e440-f263-4213-8901-0bbc0b167ef5 Faulting package full name: Faulting package-relative application ID:

What have i tried?

Checked for corruption on server level.
Repaired VS2015.
Redownloaded and configured the agent.
Checked permissions.
Tried it as a different Windows user.
Checked 'Turn on DEP for essential Windows programs and services only'

erwinkramer commented 7 years ago

I now realize that there is no official support for Windows Server 2016... I guess that's what's up?

ericsciple commented 7 years ago

@erwinkramer no i think it should work. i can setup a server 2016 machine tomorrow and try.

do you have .net 4.6 installed? i think that was related last time we hit a missing-method-exception.

how are you running tf.exe? is this during the get-sources step? or are you running it from one of your scripts?

ericsciple commented 7 years ago

does it work when the agent runs the get sources step against at TF Version Control repository? iirc, the agent validates that .NET 4.6.

but in the screenshot you pasted, it looks like you are not running the agent at all.

erwinkramer commented 7 years ago

@ericsciple

do you have .net 4.6 installed?

Yes.

but in the screenshot you pasted, it looks like you are not running the agent at all.

That's correct, i tried it manually to see if it created the same missing method exception, and it did. I originally got the message in my build definition output on the Get Sources step. See attached logfile. logs_10.zip

ericsciple commented 7 years ago

oh ok, gotcha

ericsciple commented 7 years ago

I'm unable to repro. Look under the reg key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\NET Framework Setup\NDP\v4\Full

What is the decimal value for Release? Mine is 394802 which corresponds to 4.6.2 ...which is a bit stange. Since it's not Win 10 I would have expected to see 394806. I ran winver.exe and it reports Windows Server 2016 Version 1607 (OS Build 14393.0)

erwinkramer commented 7 years ago

@ericsciple ,i have the same 394802.

erwinkramer commented 7 years ago

My OS Build is 14393.1066

erwinkramer commented 7 years ago

I uninstalled KB4015217 so now my OS Build is 14393.447, still no luck. Unfortuanly i can't uninstall the KB to go back to 14393.0. Could you update to my OS Build to rule out security updates?

ericsciple commented 7 years ago

Yes I'll update my VM. In the meantime I'm going to loop in someone from the version control team to see if they have heard of this issue. I was hoping to get a repro.

erwinkramer commented 7 years ago

@ericsciple , thanks for the support.

Just to be clear, this machine has BizTalk 2016 enterprise, MS SQL Server 2016 enterprise and VS2015 installed as well.

ericsciple commented 7 years ago

Even on Windows 14393.1066 I still can't repro. I've sent a mail to the appropriate Version Control folks and log a bug internally for them to investigate (bug 992928).

whoisj commented 7 years ago

@erwinkramer thanks for bringing this to our attention.

Give me some time to sync with @ericsciple and get a similar VM setup so I can attempt to reproduce the issue.

One question: does your Server 2016 have VS2015 and/or VS2017 installed on it? If so, into which path(s)? and which build(s)?

Thanks!

ericsciple commented 7 years ago

@erwinkramer also can you dump the externals\vstsom dir and send me the output. Something like:

dir D:\vsts-agent\externals\vstsom -r | foreach-object { "$($_.length) $($_.fullname)" } > ~\Desktop\externals.txt

I want to diff against a fresh download and rule out a corrupted layout,

erwinkramer commented 7 years ago

@ericsciple externals.txt

@whoisj Only VS2015. See: Path: C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE

Other VS stuff:

Also worth noting, SQL info:

ericsciple commented 7 years ago

i diff'd my file against yours and the layout is exactly the same

whoisj commented 7 years ago

I'm completely out of ideas. Without a local repro this is going to very difficult to resolve.

Tomorrow, I'll perform a codewalk and see if there's any hints as to why VssEnvironment.EnumeratePaths could result in a MissingMethodException while still successfully loading the strong name signed DLL.

whoisj commented 7 years ago

... Of course, as soon as I claim to be out of ideas publicly, I have an idea. 🐑

@erwinkramer can you, from a Developer Command Prompt, run the following command?

gacutil /l | findstr /I Microsoft.VisualStudio.*

... and share the output. I'm wondering if the GAC isn't out enemy here. Perhaps there's a GAC version of the assembly that's being loaded that's actually older than the version shipped with the Agent. It's worth a try!

TingluoHuang commented 7 years ago

@erwinkramer as @whoisj mentioned, check the GAC, another way to check it is when tf.exe crash, attach VS Debugger, check where is Microsoft.VisualStudio.Service.Common.dll loading from. you can see it under VS->Debug->Windows->Modules

erwinkramer commented 7 years ago

@whoisj output.txt

TingluoHuang commented 7 years ago

it's in the GAC. :D

  Microsoft.VisualStudio.Services.Client, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL
  Microsoft.VisualStudio.Services.Common, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL
  Microsoft.VisualStudio.Services.ReleaseManagement.WebApi, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL
  Microsoft.VisualStudio.Services.WebApi, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a, processorArchitecture=MSIL

erwinkramer commented 7 years ago

@TingluoHuang : Oh well so all i have to do is unGAC it? How did it end up there anyway? Output from all that i could get when attaching the debugger below. I guess this hints to GAC'd stuff too.

'TF.exe' (CLR v4.0.30319: TF.exe): Loaded 'C:\Windows\Microsoft.Net\assembly\GAC_32\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll'. Symbols loaded. 'TF.exe' (CLR v4.0.30319: TF.exe): Loaded 'C:\Users\Administrator\agent\agent\externals\vstsom\TF.exe'. Cannot find or open the PDB file. 'TF.exe' (CLR v4.0.30319: TF.exe): Loaded 'C:\Users\Administrator\agent\agent\externals\vstsom\Microsoft.TeamFoundation.Client.dll'. Cannot find or open the PDB file. 'TF.exe' (CLR v4.0.30319: TF.exe): Loaded 'C:\Windows\Microsoft.Net\assembly\GAC_MSIL\System.Drawing\v4.0_4.0.0.0__b03f5f7f11d50a3a\System.Drawing.dll'. Symbols loaded. 'TF.exe' (CLR v4.0.30319: TF.exe): Loaded 'C:\Windows\Microsoft.Net\assembly\GAC_MSIL\System\v4.0_4.0.0.0__b77a5c561934e089\System.dll'. Symbols loaded. 'TF.exe' (CLR v4.0.30319: TF.exe): Loaded 'C:\Users\Administrator\agent\agent\externals\vstsom\Microsoft.TeamFoundation.Common.dll'. Cannot find or open the PDB file. 'TF.exe' (CLR v4.0.30319: TF.exe): Loaded 'C:\Users\Administrator\agent\agent\externals\vstsom\Microsoft.TeamFoundation.VersionControl.Client.dll'. Cannot find or open the PDB file. 'TF.exe' (CLR v4.0.30319: TF.exe): Loaded 'C:\Windows\Microsoft.Net\assembly\GAC_MSIL\System.Core\v4.0_4.0.0.0__b77a5c561934e089\System.Core.dll'. Symbols loaded. 'TF.exe' (CLR v4.0.30319: TF.exe): Loaded 'C:\Windows\Microsoft.Net\assembly\GAC_MSIL\Microsoft.VisualStudio.Services.Common\v4.0_15.0.0.0__b03f5f7f11d50a3a\Microsoft.VisualStudio.Services.Common.dll'. Cannot find or open the PDB file. The program '[8240] TF.exe' has exited with code 0 (0x0).

erwinkramer commented 7 years ago

Removing Microsoft.VisualStudio.Services.Commonfrom the GAC did the trick. Pfew, thanks a lot for your assistance guys.

Still curious what installer put that file in the GAC. It cerainly wasn't me. This machine is brand new and i did nothing with the GAC yet.

TingluoHuang commented 7 years ago

@erwinkramer can you find the dll under C:\Windows\Microsoft.Net\assembly\GAC_MSIL, check file properties->details->fileversion?

erwinkramer commented 7 years ago

@TingluoHuang fileversion 15.98.25225.0

erwinkramer commented 7 years ago

More info:

TingluoHuang commented 7 years ago

@erwinkramer can you share a full list of installed software on your box?

TingluoHuang commented 7 years ago

@erwinkramer is that dll has ditail signatures? we never ship that version of assembly into TFS product. :( do you have ReleaseManagement installed on your box?

ericsciple commented 7 years ago

*digital signature

erwinkramer commented 7 years ago

TingluoHuang commented 7 years ago

@erwinkramer are you using a physical PC or VM? can you easily re-setup the machine, install all software you had one by one, then check GAC after every software installed?

TingluoHuang commented 7 years ago

@erwinkramer have you or anyone manually put those dll into GAC?

erwinkramer commented 7 years ago

@TingluoHuang It was a brand new VM, nothing done with the GAC. I could re-setup the machine, however i didn't save any snapshots and i don't have much time atm. I do however suspect BizTalk Server 2016 Feature Pack 1 because it adds some VSTS integration features and it's the main reason i'm setting this VM up.

I can check on another machine if i install the feature pack, if it gac's stuff.

TingluoHuang commented 7 years ago

@erwinkramer Great, thanks for the inportant information, i will try install BizTalk myself. :D

erwinkramer commented 7 years ago

@TingluoHuang Have fun ;)

TingluoHuang commented 7 years ago

@erwinkramer I confirmed BizTalk feature pack 1 is the evil. :D it will put those dll into GAC. uninstall will remove them from GAC. I will follow up with their product team to solve this problem.

erwinkramer commented 7 years ago

@TingluoHuang Great! Glad we identified the program.

@ericsciple Shouldn't the .NET versioning info on How to: Determine Which .NET Framework Versions Are Installed still be addressed because it corresponds with Win 10 like you mentioned?

ericsciple commented 7 years ago

@erwinkramer true. i'll see if i can find an appropriate internal distribution list to email regarding the issue.

ADammy74 commented 7 years ago

@TingluoHuang Do you have any news to this error. I am also setting up BizTalk Feature Pack 1 ALM and I am hanging at the same Point. It would be great to get a solution for this. Thanks in advance!

TingluoHuang commented 7 years ago

BizTalk Feature feature pack 1 CU1 should release soon and has the fix.

usmanshaheen commented 7 years ago

I managed to resolve it by:

have a separate build server.
install BTS FP1
create a copy of BizTalk Server 2016\Developers Tools
uninstall BTS FP1
replace Developers Tools folder with the copy created at step 3
copy missing assembly Microsoft.BizTalk.ALM.JsonSchema from BTS server and GAC it on build server
got a successful build
create a new queue for Release and host its build agent on BTS FP1 server

bryanmacfarlane commented 7 years ago

Also note that we worked with the biz talk team to fix this for their next update.

madev4cloud commented 7 years ago

I am stuck with this, we are migrating from BizTalk 2013 to 2016 I need to set up build & releases. Help how to fix this!

TingluoHuang commented 7 years ago

@manisko what version of biztalk you installed, have you try 2016 CU3? have you check your GAC on that box as the issue mentioned above?

madev4cloud commented 7 years ago

I have in my build server BizTalk 2016 FP1, haven't tried CU3 - can I install CU3 with FP1 ?

TingluoHuang commented 7 years ago

@manisko i got confirmed from BizTalk team, the following release should contains the fix. https://www.microsoft.com/en-us/download/details.aspx?id=55100

TingluoHuang commented 7 years ago

@manisko i think you can. :)

madev4cloud commented 7 years ago

Just sec ago installed CU1/CU2 (BTS2016-KB4014788-ENU.exe) and it works for me! Anyway big thanks for help.

TingluoHuang commented 7 years ago

@manisko cool, glad you unblocked.

shyshkov commented 4 years ago

I had the same problem with the project. @erwinkramer was close to the solution.

The problem lies in the wrong .NET Framework runtime version.

One has to install the correct .NET Framework runtime version!

i.e. I had 4.6.2 on the Windows Server 2016, but an application requires 4.7.2, this is why I got a method not found in Unity object exception and KERNELBASE.dll exception also in Events and an application didn't work. After installing .NET Framework 4.7.2 the problem gone.

https://docs.microsoft.com/en-us/dotnet/framework/migration-guide/how-to-determine-which-versions-are-installed?redirectedfrom=MSDN

microsoft / azure-pipelines-agent