vswhere hangs - Githubissues

ks-jeppe commented 7 years ago

Sometimes when I call vswhere with '-latest -legacy' multiple times it will hang indefinitely.

This happens both with the release and debug version, but a colleague couldn't reproduce the issue with a fairly similar setup.

My machine has VS2015 Pro, and VS2017 Pro. It also has remnants of uninstalled versions going back to VS2008 but these are correctly not being detected when vswhere works correctly.

Attaching and pausing the process running a debug build shows two threads with the following stack traces:

    ntdll.dll!_NtWaitForSingleObject@12()  Unknown
    ntdll.dll!_RtlpWaitOnCriticalSection@8()   Unknown
    ntdll.dll!_RtlEnterCriticalSection@4() Unknown
    ntdll.dll!_LdrGetProcedureAddressEx@20()   Unknown
    ntdll.dll!_LdrGetProcedureAddress@16() Unknown
    KernelBase.dll!_GetProcAddress@8() Unknown
    Microsoft.VisualStudio.Setup.Configuration.Native.dll!__acrt_lock_and_call<class <lambda_a463b7b0560cfcaf9b17f27c6ef46564> >(enum __acrt_lock_id,class <lambda_a463b7b0560cfcaf9b17f27c6ef46564> &&)    Unknown
    Microsoft.VisualStudio.Setup.Configuration.Native.dll!___acrt_IsValidLocaleName@4()    Unknown
    Microsoft.VisualStudio.Setup.Configuration.Native.dll!__expandlocale() Unknown
    Microsoft.VisualStudio.Setup.Configuration.Native.dll!__wsetlocale()   Unknown
    Microsoft.VisualStudio.Setup.Configuration.Native.dll!__acrt_lock_and_call<class <lambda_e378711a6f6581bf7f0efd7cdf97f5d9> >(enum __acrt_lock_id,class <lambda_e378711a6f6581bf7f0efd7cdf97f5d9> &&)    Unknown
    Microsoft.VisualStudio.Setup.Configuration.Native.dll!__crt_seh_guarded_call<void>::operator()<class <lambda_c76fdea48760d5f9368b465f31df4405>,class <lambda_e378711a6f6581bf7f0efd7cdf97f5d9> &,class <lambda_e927a58b2a85c081d733e8c6192ae2d2> >(class <lambda_c76fdea48760d5f9368b465f31df4405> &&,class <lambda_e378711a6f6581bf7f0efd7cdf97f5d9> &,class <lambda_e927a58b2a85c081d733e8c6192ae2d2> &&) Unknown
    Microsoft.VisualStudio.Setup.Configuration.Native.dll!__acrt_lock_and_call<class <lambda_e378711a6f6581bf7f0efd7cdf97f5d9> >(enum __acrt_lock_id,class <lambda_e378711a6f6581bf7f0efd7cdf97f5d9> &&)    Unknown
    Microsoft.VisualStudio.Setup.Configuration.Native.dll!__acrt_lock_and_call<class <lambda_e378711a6f6581bf7f0efd7cdf97f5d9> >(enum __acrt_lock_id,class <lambda_e378711a6f6581bf7f0efd7cdf97f5d9> &&)    Unknown
    Microsoft.VisualStudio.Setup.Configuration.Native.dll!__crt_seh_guarded_call<void>::operator()<class <lambda_70818de7b02deff9841e8b0962a60ed9>,class <lambda_2af78c5f5901b1372d98f9ab3177dfa6> &,class <lambda_f51fe5fd7c79a33db34fc9310f277369> &>(class <lambda_70818de7b02deff9841e8b0962a60ed9> &&,class <lambda_2af78c5f5901b1372d98f9ab3177dfa6> &,class <lambda_f51fe5fd7c79a33db34fc9310f277369> &) Unknown
    Microsoft.VisualStudio.Setup.Configuration.Native.dll!__wsetlocale()   Unknown
    Microsoft.VisualStudio.Setup.Configuration.Native.dll!_wcscpy_s()  Unknown
    Microsoft.VisualStudio.Setup.Configuration.Native.dll!_setlocale() Unknown
    Microsoft.VisualStudio.Setup.Configuration.Native.dll!std::_Locinfo::_Locinfo_ctor(class std::_Locinfo *,char const *)  Unknown
    Microsoft.VisualStudio.Setup.Configuration.Native.dll!std::_Locinfo::_Locinfo(char const *) Unknown
    Microsoft.VisualStudio.Setup.Configuration.Native.dll!std::ctype<wchar_t>::_Getcat(class std::locale::facet const * *,class std::locale const *)    Unknown
    Microsoft.VisualStudio.Setup.Configuration.Native.dll!std::use_facet<class std::ctype<wchar_t> >(class std::locale const &) Unknown
    Microsoft.VisualStudio.Setup.Configuration.Native.dll!SetupHelper::Trim(wchar_t const * const,wchar_t * *)  Unknown
    Microsoft.VisualStudio.Setup.Configuration.Native.dll!SetupHelper::ParseVersion(wchar_t const *,unsigned __int64 *) Unknown
>   vswhere.exe!InstanceSelector::Less(const _com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > & a={...}, const _com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > & b={...}) Line 49  C++
    vswhere.exe!InstanceSelector::Select::__l26::<lambda>(const _com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > & a={...}, const _com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > & b={...}) Line 150  C++
    vswhere.exe!std::_Debug_lt_pred<bool <lambda>(const _com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > &, const _com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > &) &,_com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > &,_com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > &>(InstanceSelector::Select::__l26::bool <lambda>(const _com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > &, const _com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > &) & _Pred=bool <lambda>(const _com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > & a, const _com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > & b){...}, _com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > & _Left={...}, _com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > & _Right={...}, const wchar_t * _File=0x00bac6a8, unsigned int _Line=2766) Line 948  C++
    vswhere.exe!std::_Insertion_sort_unchecked<_com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > *,bool <lambda>(const _com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > &, const _com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > &) >(_com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > * _First=0x00606448, _com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > * _Last=0x00606450, InstanceSelector::Select::__l26::bool <lambda>(const _com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > &, const _com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > &) & _Pred=bool <lambda>(const _com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > & a, const _com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > & b){...}) Line 2766 C++
    vswhere.exe!std::_Sort_unchecked1<_com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > *,int,bool <lambda>(const _com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > &, const _com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > &) >(_com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > * _First=0x00606448, _com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > * _Last=0x00606450, int _Ideal=2, InstanceSelector::Select::__l26::bool <lambda>(const _com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > &, const _com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > &) & _Pred=bool <lambda>(const _com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > & a, const _com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > & b){...}) Line 2901    C++
    vswhere.exe!std::_Sort_unchecked<_com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > *,bool <lambda>(const _com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > &, const _com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > &) >(_com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > * _First=0x00606448, _com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > * _Last=0x00606450, InstanceSelector::Select::__l26::bool <lambda>(const _com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > &, const _com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > &) & _Pred=bool <lambda>(const _com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > & a, const _com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > & b){...}) Line 2908   C++
    vswhere.exe!std::sort<std::_Vector_iterator<std::_Vector_val<std::_Simple_types<_com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > > > >,bool <lambda>(const _com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > &, const _com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > &) >(std::_Vector_iterator<std::_Vector_val<std::_Simple_types<_com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > > > > _First={...}, std::_Vector_iterator<std::_Vector_val<std::_Simple_types<_com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > > > > _Last={...}, InstanceSelector::Select::__l26::bool <lambda>(const _com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > &, const _com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > &) _Pred=bool <lambda>(const _com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > & a, const _com_ptr_t<_com_IIID<ISetupInstance,&_GUID_b41463c3_8866_43b5_bc33_2b0676f7f42e> > & b){...}) Line 2916    C++
    vswhere.exe!InstanceSelector::Select(IEnumSetupInstances * pEnum=0x00606310) Line 147   C++
    vswhere.exe!wmain(int argc=3, const wchar_t * * argv=0x00603738) Line 49    C++
    vswhere.exe!invoke_main() Line 79   C++
    vswhere.exe!__scrt_common_main_seh() Line 259   C++
    vswhere.exe!__scrt_common_main() Line 302   C++
    vswhere.exe!wmainCRTStartup() Line 17   C++
    kernel32.dll!@BaseThreadInitThunk@12() Unknown
    ntdll.dll!___RtlUserThreadStart@8()    Unknown
    ntdll.dll!__RtlUserThreadStart@8() Unknown

>   ntdll.dll!_NtWaitForSingleObject@12()  Unknown
    ntdll.dll!_RtlpWaitOnCriticalSection@8()   Unknown
    ntdll.dll!_RtlEnterCriticalSection@4() Unknown
    Microsoft.VisualStudio.Setup.Configuration.Native.dll!___acrt_lock()   Unknown
    Microsoft.VisualStudio.Setup.Configuration.Native.dll!__crt_seh_guarded_call<void>::operator()<class <lambda_3518db117f0e7cdb002338c5d3c47b6c>,class <lambda_b2ea41f6bbb362cd97d94c6828d90b61> &,class <lambda_abdedf541bb04549bc734292b4a045d4> >(class <lambda_3518db117f0e7cdb002338c5d3c47b6c> &&,class <lambda_b2ea41f6bbb362cd97d94c6828d90b61> &,class <lambda_abdedf541bb04549bc734292b4a045d4> &&) Unknown
    Microsoft.VisualStudio.Setup.Configuration.Native.dll!__acrt_lock_and_call<class <lambda_b2ea41f6bbb362cd97d94c6828d90b61> >(enum __acrt_lock_id,class <lambda_b2ea41f6bbb362cd97d94c6828d90b61> &&)    Unknown
    Microsoft.VisualStudio.Setup.Configuration.Native.dll!__acrt_lock_and_call<class <lambda_e69574bed617af4e071282c136b37893> >(enum __acrt_lock_id,class <lambda_e69574bed617af4e071282c136b37893> &&)    Unknown
    Microsoft.VisualStudio.Setup.Configuration.Native.dll!___acrt_getptd_noexit()  Unknown
    Microsoft.VisualStudio.Setup.Configuration.Native.dll!___acrt_thread_attach()  Unknown
    Microsoft.VisualStudio.Setup.Configuration.Native.dll!___scrt_dllmain_crt_thread_attach()  Unknown
    Microsoft.VisualStudio.Setup.Configuration.Native.dll!operator new(unsigned int,struct std::nothrow_t const &)  Unknown
    Microsoft.VisualStudio.Setup.Configuration.Native.dll!operator new(unsigned int,struct std::nothrow_t const &)  Unknown
    Microsoft.VisualStudio.Setup.Configuration.Native.dll!__DllMainCRTStartup@12() Unknown
    ntdll.dll!_LdrpCallInitRoutine@16()    Unknown
    ntdll.dll!_LdrpInitializeThread@4()    Unknown
    ntdll.dll!__LdrpInitialize@8() Unknown
    ntdll.dll!_LdrInitializeThunk@8()  Unknown

heaths commented 7 years ago

Thank you for the detailed report. I am unable to repro this even in a fast loop. Also, on what version of vswhere and the query API (newer versions will print both in the header for the default formatter) are you running?

Pausing the process could yield any possible stack trace. Are you able to generate a dump when the process is hung?

ks-jeppe commented 7 years ago

Version header says: Visual Studio Locator version 1.0.66-g24f19ddd29 [query version 1.10.80.60812]

The backtraces from the report are when the process is hung. They always look the same.

Do you mean a .DMP file? I could, but I lack a place to host it as it's about 14 megabytes zipped.

heaths commented 7 years ago

Those seem to be privately built bits. Can you repro with official builds? Are you building with a Preview release of VS? If not, what version of VS?

If you've signed into VS, your Microsoft Account is already hooked up to OneDrive or signing up for other services like Dropbox could provide space for a .dmp.

ks-jeppe commented 7 years ago

I can't seem to reproduce it with official builds.

The one I built was built with VS 2017, version 15.2 (26430.6) Release

heaths commented 7 years ago

Release or Debug configuration? This might be an issue to take up with the Visual C++ team. From the stack it seems there could be a deadlock scenario but the query API and vswhere don't directly create or use any locks.

ks-jeppe commented 7 years ago

It happens all the time in the Debug configuration. I'm not seeing it right now in release configuration, but I think it might have happened once.

I found a way to reproduce it on multiple computers with the debug build where it has 2017 and at least also 2015 installed.

Start with no VS instances open.
Start running for /l %i in (1,1,10000) do vswhere.exe -legacy -latest
Launch Visual Studio 2017 and load a solution
About the time it is done loading the solution vswhere.exe will be hanging in a state where it cannot be killed with Ctrl+C

It definitely sounds like a Visual C++ runtime bug with a DLL loader deadlock

refack commented 7 years ago

About the time it is done loading the solution vswhere.exe will be hanging in a state where it cannot be killed with Ctrl+C

Maybe related: happens also when I hit Ctrl-C during VsDevCmd.bat, process hangs.

heaths commented 7 years ago

Seems this is a known issue that was fixed in a newer, currently non-default UCRT. To work around this issue in your own code, you can add the following to the vcxproj files:

<TargetUniversalCRTVersion>10.0.14393.0</TargetUniversalCRTVersion>

I'll not be making this change to the vswhere code base since the default is for the LTS release of Windows 10. When the default changes, we'll get it automatically. Until then, since it does not repro in release builds and with a work around available, I'm resolving the bug. If this becomes a larger problem we can reconsider making the change for all builds.

microsoft / vswhere

vswhere hangs #72