Win32OLE fails with error 0x80070057 "The parameter is incorrect" for some .NET COM libraries

[krolchatina]

What problems are you experiencing?

When using win32ole from ruby delivered with ruby installer COM classes exported from (some?) .NET libraries cannot be created with error 0x80070057 "The parameter is incorrect".

The error occurs for .NET COM assemblies with "mscoree.dll" (i.e. relative path) in the InprocServer32 registry key. If InprocServer32 contains full path (for example, C:\Windows\system32\mscoree.dll) the error does not occur.

If I manually extract ruby from "rubyinstaller2-packages", the error does not occur at all. I did a bit of research and found that if I delete file lib\ruby\3.0.0\rubygems\defaults\operating_system.rb the error stops to occur. Apparently it has something to do with modified load pathes that somehow affect mscoree.dll

I made a small .net com library that demonstrates the problem. The source of this library is attached to the issue and it can be built with "dotnet build". ClassLibraryForRuby.zip

Steps to reproduce

1. Install ruby installer
2. Register a .net com library, for example C:\Windows\Microsoft.NET\Framework64\v4.0.30319\RegAsm.exe /register ClassLibraryForRuby.dll /codebase /tlb /verbose
3. Run IRB and use win32ole to create object:

C:\Ruby30-x64_ri\bin>irb.cmd                                                                                                                                                                                       irb(main):001:0> require 'win32ole'
=> true
irb(main):002:0> WIN32OLE.new('ClassLibraryForRuby.RubyComLibrary')
(irb):2:in `initialize': failed to create WIN32OLE object from `ClassLibraryForRuby.RubyComLibrary' (WIN32OLERuntimeError)
    HRESULT error code:0x80070057
      The parameter is incorrect.
        from (irb):2:in `new'
        from (irb):2:in `<main>'
        from C:/Ruby30-x64_ri/lib/ruby/gems/3.0.0/gems/irb-1.3.5/exe/irb:11:in `<top (required)>'
        from C:/Ruby30-x64_ri/bin/irb.cmd:31:in `load'
        from C:/Ruby30-x64_ri/bin/irb.cmd:31:in `<main>'
irb(main):003:0>

What's the output from ridk version?

---                                                                                                                                                                                                                ruby:                                                                                                                                                                                                                
  path: C:/Ruby30-x64_ri                                                                                                                                                                                             
  version: 3.0.2                                                                                                                                                                                                     
  platform: x64-mingw32                                                                                                                                                                                              
  cc: gcc.exe (Rev5, Built by MSYS2 project) 10.3.0                                                                                                                                                                ruby_installer:                                                                                                                                                                                                      
  package_version: 3.0.2-1                                                                                                                                                                                           
  git_commit: 80c9ca9                                                                                                                                                                                              
  os: Microsoft Windows [Version 10.0.19043.1165]  

The problem seems to be reproducible at least with all 3.0.* versions of ruby with rubyinstaller. There are also reports on 2.6, for example: https://bugs.ruby-lang.org/issues/16116

[larskanis]

Thank you for providing a self contained example! I already noticed your post on bugs.ruby-lang.org but didn't had the time to dig deeper into it.

RubyInstaller is using executable and library search path separation, which was introduced by Windows-7. The classic way is that the PATH environment variable controls both, but RubyInstaller disables library lookups through PATH in favor of dedicated paths set by RubyInstaller::Runtime.add_dll_directory. That separation is necessary for coupling Ruby and MSYS2 together. With deleting the operating_system.rb you changed to classic library searches. It is described here: https://github.com/oneclick/rubyinstaller2/wiki/For-gem-developers#user-content-dll-loading

Thanks to your simple reproducible example and all the great work you've already done, I'm certain that we can finally fix this issue!

[develowear]

161 was solved by deleting operating_system.rb.

Thanks a lot.

[larskanis]

Sorry it took so long to to try your example! But now I was finally able to compile your DLL and try it out.

I compared the Windows DLL loader debug trace between the standard search mode and the SetDefaultDllDirectories search mode used by RubyInstaller-2.4+ . I made use of gdb as described in https://github.com/oneclick/rubyinstaller2/wiki/For-gem-developers#-debug-dll-loading :

Standard mode (as used if operating_system.rb is missing):

warning: 03f4:058c @ 19635218 - LdrLoadDll - ENTER: DLL name: mscoree.dll
warning: 03f4:058c @ 19635218 - LdrpLoadDllInternal - ENTER: DLL name: mscoree.dll
warning: 03f4:058c @ 19635234 - LdrpFindKnownDll - ENTER: DLL name: mscoree.dll
warning: 03f4:058c @ 19635234 - LdrpFindKnownDll - RETURN: Status: 0xc0000135
warning: 03f4:058c @ 19635250 - LdrpSearchPath - ENTER: DLL name: mscoree.dll
warning: 03f4:058c @ 19635250 - LdrpComputeLazyDllPath - INFO: DLL search path computed: C:\rubyxxx\bin;C:\Windows\SYSTEM32;C:\Windows\system;C:\Windows;.;C:\rubyxxx\bin;C:\msys64\ucrt64\bin;C:\msys64\usr\bin;C:\rubyxxx\bin;C:\Ruby30\bin;C:\Ruby27\bin;C:\Ruby31\bin;C:\Ruby30-x64\bin;C:\Ruby31-x64\bin;C:\Ruby27-x64\bin;C:\Ruby25\bin;C:\Ruby25-x64\bin;C:\Ruby24-x64\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;c:\Program Files (x86)\Inno Setup 6;c:\Program Fi
warning: 03f4:058c @ 19635296 - LdrpResolveDllName - ENTER: DLL name: C:\rubyxxx\bin\mscoree.dll
warning: 03f4:058c @ 19635312 - LdrpResolveDllName - RETURN: Status: 0xc0000135
warning: 03f4:058c @ 19635312 - LdrpResolveDllName - ENTER: DLL name: C:\Windows\SYSTEM32\mscoree.dll
warning: 03f4:058c @ 19635328 - LdrpResolveDllName - RETURN: Status: 0x00000000
warning: 03f4:058c @ 19635343 - LdrpSearchPath - RETURN: Status: 0x00000000
warning: 03f4:058c @ 19635343 - LdrpMapViewOfSection - ENTER: DLL name: C:\Windows\SYSTEM32\mscoree.dll
warning: 03f4:058c @ 19635375 - LdrpMapViewOfSection - RETURN: Status: 0x00000000
warning: 03f4:058c @ 19635375 - LdrpFindDllActivationContext - INFO: Probing for the manifest of DLL "C:\Windows\SYSTEM32\mscoree.dll" failed with status 0xc000008a
warning: 03f4:058c @ 19635406 - LdrpGetProcedureAddress - INFO: Locating procedure "RtlEnterCriticalSection" by name
warning: 03f4:058c @ 19635421 - LdrpGetProcedureAddress - INFO: Locating procedure "RtlLeaveCriticalSection" by name
warning: 03f4:058c @ 19635437 - LdrpGetProcedureAddress - INFO: Locating procedure "RtlInitializeCriticalSection" by name
warning: 03f4:058c @ 19635453 - LdrpGetProcedureAddress - INFO: Locating procedure "RtlDeleteCriticalSection" by name
warning: 03f4:058c @ 19635468 - LdrpGetProcedureAddress - INFO: Locating procedure "RtlAllocateHeap" by name
warning: 03f4:058c @ 19635468 - LdrpGetProcedureAddress - INFO: Locating procedure "RtlEncodePointer" by name
warning: 03f4:058c @ 19635484 - LdrpGetProcedureAddress - INFO: Locating procedure "RtlDecodePointer" by name
warning: 03f4:058c @ 19635500 - LdrpGetProcedureAddress - INFO: Locating procedure "RtlReAllocateHeap" by name
warning: 03f4:058c @ 19635515 - LdrpGetProcedureAddress - INFO: Locating procedure "RtlSizeHeap" by name
warning: 03f4:058c @ 19635515 - LdrpGetProcedureAddress - INFO: Locating procedure "RtlReleaseSRWLockExclusive" by name
warning: 03f4:058c @ 19635531 - LdrpGetProcedureAddress - INFO: Locating procedure "RtlAcquireSRWLockExclusive" by name
warning: 03f4:058c @ 19635546 - LdrpInitializeNode - INFO: Calling init routine 00007FFFEBA94AF0 for DLL "C:\Windows\SYSTEM32\mscoree.dll"
warning: 03f4:058c @ 19635562 - LdrpLoadDllInternal - RETURN: Status: 0x00000000
warning: 03f4:058c @ 19635562 - LdrLoadDll - RETURN: Status: 0x00000000

and with operating_system.rb:

warning: 02e0:0c68 @ 18503546 - LdrLoadDll - ENTER: DLL name: mscoree.dll
warning: 02e0:0c68 @ 18503546 - LdrpLoadDllInternal - ENTER: DLL name: mscoree.dll
warning: 02e0:0c68 @ 18503546 - LdrpFindKnownDll - ENTER: DLL name: mscoree.dll
warning: 02e0:0c68 @ 18503562 - LdrpFindKnownDll - RETURN: Status: 0xc0000135
warning: 02e0:0c68 @ 18503562 - LdrpSearchPath - ENTER: DLL name: mscoree.dll
warning: 02e0:0c68 @ 18503578 - LdrpComputeLazyDllPath - ERROR: Lazy DLL search path computation failed with status: 0xc000000d.
warning: 02e0:0c68 @ 18503578 - LdrpSearchPath - RETURN: Status: 0xc000000d
warning: 02e0:0c68 @ 18503593 - LdrpProcessWork - ERROR: Unable to load DLL: "mscoree.dll", Parent Module: "(null)", Status: 0xc000000d
warning: 02e0:0c68 @ 18503593 - LdrpLoadDllInternal - RETURN: Status: 0xc000000d
warning: 02e0:0c68 @ 18503609 - LdrLoadDll - RETURN: Status: 0xc000000d
test.rb:4:in `initialize': failed to create WIN32OLE object from `ClassLibraryForRuby.RubyComLibrary' (WIN32OLERuntimeError)
    HRESULT error code:0x80070057
      The parameter is incorrect.
        from test.rb:4:in `new'
        from test.rb:4:in `<main>'

For some reason Windows doesn't search the mscoree.dll within c:/Windows/system32 in the second case. Although it should, since SetDefaultDllDirectories() is called with argument LOAD_LIBRARY_SEARCH_DEFAULT_DIRS = 0x00001000. This should include c:/Windows/system32 according to the specs here. Even adding the search path manually per RubyInstaller::Runtime.add_dll_directory("C:/windows/system32") doesn't help.

So I tried to load the mscoree.dll in advance and that works. So for me it looks like a Windows bug when resolving dotnet dependencies.

Nevertheless adding this in front of your test.rb fixes the issue for me:

require "fiddle"
Fiddle.dlopen("mscoree")

[larskanis]

I think the fix above is simple enough to close this issue.