search

azurefx / DotNET.jl

Julia ❤️ .NET
MIT License
91 stars 4 forks source link

Trouble using external assemblies #20

Open nwamsley1 opened 1 year ago

nwamsley1 commented 1 year ago

I have been having trouble using external assemblies. Following the readme I have tried the following:

using DotNET
reader = T"System.Reflection.Assembly".LoadFrom(raw"/Users/n.t.wamsley/Projects/Julia_Testing/ThermoFisher.CommonCore.RawFileReader.dll")

Which returns 

System.Reflection.RuntimeAssembly("ThermoFisher.CommonCore.RawFileReader, Version=5.0.0.88, Culture=neutral, PublicKeyToken=1aef06afb5abd953")

How can I use methods and types from this library? For example in the RawFileReader library, there is an attribute 'RawFileReaderAdapter' with a method 'FileFactory'. I would like to do something like 

rawFile = RawFileReaderAdapter.FileFactory(filename)

rawFile then has methods I could call to return data from the file.

The problem is that if I enter the following

rawfilereaderadapter = reader.GetType("ThermoFisher.CommonCore.RawFileReader.RawFileReaderAdapter", true, true)
filefactory = rawfilereaderadapter.GetMethod("FileFactory")
filefactory.ReturnType
filefactory.ReturnTypeCustomAttributes

I get 

ThermoFisher.CommonCore.RawFileReader.RawFileReaderAdapter
System.Reflection.RuntimeMethodInfo("ThermoFisher.CommonCore.Data.Interfaces.IRawDataExtended FileFactory(System.String)")
ThermoFisher.CommonCore.Data.Interfaces.IRawDataExtended
System.Reflection.RuntimeParameterInfo("ThermoFisher.CommonCore.Data.Interfaces.IRawDataExtended")

respectively. Now I need to invoke the FileFactory method. 

julia> filepath = convert(CLRObject, "/Users/n.t.wamsley/Projects/SAGE_TESTING/MA4358_FFPE_HPVpos_01_071522.raw")
julia> rawfile = filefactory.Invoke("", [filepath])
ThermoFisher.CommonCore.RawFileReader.RawFileAccess("ThermoFisher.CommonCore.RawFileReader.RawFileAccess")
julia> rawfile.IsOpen
false

rawfile.IsOpen should be returning true. I have verified that the file path is correct. So I suspect I am passing methods incorrectly. I wasn't sure if you had any ideas about what I could be doing wrong?

Here is a snippet from the .xml file describing the types and methods defined in the dll

 <member name="T:ThermoFisher.CommonCore.RawFileReader.RawFileReaderAdapter">
            <summary>
            This static class contains factories to open raw files
            </summary>
        </member>
        <member name="M:ThermoFisher.CommonCore.RawFileReader.RawFileReaderAdapter.FileFactory(System.String)">
            <summary>
            Create an IRawDataExtended interface to read data from a raw file
            </summary>
            <param name="fileName">File to open</param>
            <returns>Interface to read data from file</returns>
        </member>
nwamsley1 commented 1 year ago

There is a clearer explanation of the problem with a minimal example and context on stack overflow. https://stackoverflow.com/questions/75325784/using-a-net-assembly-in-with-dotnet-jl-julia

azurefx commented 1 year ago

Hi nwamsley1,

You can invoke instance methods or static methods directly without explicit use of reflection. If the invocation raises an exception, it will be thrown in Julia as a CLRException.

julia> using DotNET

julia> T"System.Reflection.Assembly".LoadFrom("ThermoFisher.CommonCore.RawFileReader.dll")
System.Reflection.RuntimeAssembly("ThermoFisher.CommonCore.RawFileReader, Version=5.0.0.88, Culture=neutral, PublicKeyToken=1aef06afb5abd953")

julia> RawFileAdapter = T"ThermoFisher.CommonCore.RawFileReader.RawFileReaderAdapter, ThermoFisher.CommonCore.RawFileReader"
ThermoFisher.CommonCore.RawFileReader.RawFileReaderAdapter

julia> rawFile = RawFileAdapter.FileFactory("empty")
ThermoFisher.CommonCore.RawFileReader.RawFileAccess("ThermoFisher.CommonCore.RawFileReader.RawFileAccess")

julia> rawFile.IsOpen
false

julia> rawFile.IsError
true

There is no exception, indicating that the RawFileAccess object has been successfully created.

I'm not familiar with this library, and I don't have sample data that can be read by this FileFactory. You can check the documentation of this library to see if there is an error code or other way to check for errors.

azurefx commented 1 year ago

I checked the DLL and found there is actually an exception:

julia> fileError = rawFile.FileError
ThermoFisher.CommonCore.RawFileReader.Facade.RawFileLoader("ThermoFisher.CommonCore.RawFileReader.Facade.RawFileLoader")

julia> fileError.ErrorCode
-1

julia> fileError.WarningMessage
"Information: Creating mutex for: empty.raw\r\n"

julia> fileError.ErrorMessage
"Encountered problems while trying to read '' as a Raw File!\r\n\r\nException type: System.MissingMethodException\r\nMessage       : Method not found: 'Void System.Threading.Mutex..ctor(Boolean, System.String, Boolean ByRef, System.Security.AccessControl.MutexSecurity)'.\r\nStacktrace:\r\n   at ThermoFisher.CommonCore.RawFileReader.Utilities.CreateNamedMutexAndWait(String fileName, Boolean useGlobalNamespace, DeviceErrors errors)\r\n   at ThermoFisher.CommonCore.RawFileReader.Facade.RawFileLoader..ctor(String fileName)\r\n"

The reason is that RawFileReader calls the Mutex constructor which has a MutexSecurity parameter that is only supported by .NET Framework, indicating that this library may only run on .NET Framework.

DotNET currently only supports cross-platform .NET (Core) environment (#19) but I may add .NET Framework 4.X support on Windows in the future.

nwamsley1 commented 1 year ago

Thanks, this is very helpful. It turns out that Thermo has a .NET standard 2.0 build of these libraries (I think this would work?). I requested it, so I think I will be able to get this to work after all. I can replicate what you are showing on my machine. Thanks for you help!

nwamsley1 commented 1 year ago

With the .NET Core 2.0 build, I encounter an error where a space seems to be added to the front of the filepath. 

julia> filepath = convert(CLRObject,"/Users/n.t.wamsley/Projects/Julia_Testing/MA4358_FFPE_HPVpos_01_071522.raw")
julia> rawFile = RawFileAdapter.FileFactory(filepath)
julia> fileError = rawFile.FileError
julia> fileError.ErrorMessage
" /Users/n.t.wamsley/Projects/Julia_Testing/MA4358_FFPE_HPVpos_01_071522.raw"
julia> fileError.WarningMessage
"Information: Creating mutex for: /Users/n.t.wamsley/Projects/Julia_Testing/MA4358_FFPE_HPVpos_01_071522.raw\nInformation: Created mutex for: /Users/n.t.wamsley/Projects/Julia_Testing/MA4358_FFPE_HPVpos_01_071522.raw\nInformation: Release mutex for: /Users/n.t.wamsley/Projects/Julia_Testing/MA4358_FFPE_HPVpos_01_071522.raw\nInformation: Close mutex for: /Users/n.t.wamsley/Projects/Julia_Testing/MA4358_FFPE_HPVpos_01_071522.raw\n"
azurefx commented 1 year ago

This error message doesn't seem informative enough.. Are there any error codes?

By the way, there should be no spaces at the beginning of the string. This can be easily verified:

julia> juliastr = "/Users/n.t.wamsley/Projects/Julia_Testing/MA4358_FFPE_HPVpos_01_071522.raw"
"/Users/n.t.wamsley/Projects/Julia_Testing/MA4358_FFPE_HPVpos_01_071522.raw"

julia> length(juliastr)
74

julia> clrstr = convert(CLRObject, juliastr)
System.String("/Users/n.t.wamsley/Projects/Julia_Testing/MA4358_FFPE_HPVpos_01_071522.raw")

julia> clrstr.Length
74

# no need to `convert` a Julia string to `CLRObject` since most primitive types in Julia are automatically converted back and forth
julia> clrstr.StartsWith("/Users")
true

I suspect that the space is caused by string interpolation, and the leading error message somehow disappeared. If you can load the .NET Core library in Python, can you try to see if you get the same error?

nwamsley1 commented 1 year ago

The error code is 2. I verified the string length, just strange that the error message seems to be adding a space in what it prints out. From the xml

 <member name="P:ThermoFisher.CommonCore.RawFileReader.Writers.DeviceErrors.ErrorCode">
            <summary>
            Gets the error code number.
            Typically this is a windows system error number.
            If no number is encoded:
            Returns "1" if the error message is null
            Returns "2" is there is an error message.
            A return of "0" will occur if there has been no error message
            </summary>
        </member>

From some googling, it seems like the windows error code 2 has to do with a "can't find the file specified" type error.

azurefx commented 1 year ago

0x2 means ERROR_FILE_NOT_FOUND: The system cannot find the file specified.

azurefx commented 1 year ago

I don't know how to debug system calls under *nix, but if you are running Windows, you can try the procmon tool from Sysinternals tookit, which records disk activities of a specific process. Then you can check for failed CreateFile(A|W) or other Win32 API calls related to the file system.

Can you share the .NET Core version of the library and a valid raw file? I can help you debug it on a Windows computer.

nwamsley1 commented 1 year ago

That would be awesome.

https://figshare.com/articles/dataset/Raw_and_DLL/22022552

I think this link will work.

azurefx commented 1 year ago

It works fine on my computer.

julia> using DotNET

julia> T"System.Reflection.Assembly".LoadFrom("ThermoFisher.CommonCore.RawFileReader.dll")
System.Reflection.RuntimeAssembly("ThermoFisher.CommonCore.RawFileReader, Version=5.0.0.88, Culture=neutral, PublicKeyToken=1aef06afb5abd953")

julia> RawFileAdapter = T"ThermoFisher.CommonCore.RawFileReader.RawFileReaderAdapter, ThermoFisher.CommonCore.RawFileReader"
ThermoFisher.CommonCore.RawFileReader.RawFileReaderAdapter

julia> rawFile = RawFileAdapter.FileFactory("MA4358_FFPE_HPVpos_01_071522.raw")
ThermoFisher.CommonCore.RawFileReader.RawFileAccess("ThermoFisher.CommonCore.RawFileReader.RawFileAccess")

julia> rawFile.IsOpen
true

julia> rawFile.IsError
false

julia> rawFile.ComputerName
"THERMO-2KV4HQ2"

julia> rawFile.InstrumentCount
4

julia> versioninfo()
Julia Version 1.7.2
Commit bf53498635 (2022-02-06 15:21 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: AMD Ryzen 7 3700X 8-Core Processor
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-12.0.1 (ORCJIT, znver2)
Environment:
  JULIA_NUM_THREADS = 4

julia> DotNET.detect_runtime(DotNET.CoreCLR.CoreCLRHost)
6-element Vector{Any}:
 (type = "Microsoft.NETCore.App", version = v"6.0.13", path = "C:\\Program Files\\dotnet\\shared\\Microsoft.NETCore.App\\6.0.13\\coreclr.dll")
 (type = "Microsoft.NETCore.App", version = v"3.1.32", path = "C:\\Program Files\\dotnet\\shared\\Microsoft.NETCore.App\\3.1.32\\coreclr.dll")
 (type = "Microsoft.NETCore.App", version = v"3.0.1", path = "C:\\Program Files\\dotnet\\shared\\Microsoft.NETCore.App\\3.0.1\\coreclr.dll")
 (type = "Microsoft.NETCore.App", version = v"2.0.9", path = "C:\\Program Files\\dotnet\\shared\\Microsoft.NETCore.App\\2.0.9\\coreclr.dll")
 (type = "Microsoft.NETCore.App", version = v"2.0.7", path = "C:\\Program Files\\dotnet\\shared\\Microsoft.NETCore.App\\2.0.7\\coreclr.dll")
 (type = "Microsoft.NETCore.App", version = v"2.0.6", path = "C:\\Program Files\\dotnet\\shared\\Microsoft.NETCore.App\\2.0.6\\coreclr.dll")

DotNET will choose the first runtime returned by detect_runtime. What version of .NET are you using?

nwamsley1 commented 1 year ago 
julia> versioninfo()
Julia Version 1.8.5
Commit 17cfb8e65ea (2023-01-08 06:45 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin21.4.0)
  CPU: 16 × Intel(R) Core(TM) i7-10700K CPU @ 3.80GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, skylake)
  Threads: 1 on 16 virtual cores

julia> DotNET.detect_runtime(DotNET.CoreCLR.CoreCLRHost)
9-element Vector{Any}:
 (type = "Microsoft.NETCore.App", version = v"7.0.2", path = "/usr/local/share/dotnet/shared/Microsoft.NETCore.App/7.0.2/libcoreclr.dylib")
 (type = "Microsoft.NETCore.App", version = v"5.0.17", path = "/usr/local/share/dotnet/shared/Microsoft.NETCore.App/5.0.17/libcoreclr.dylib")
 (type = "Microsoft.NETCore.App", version = v"5.0.13", path = "/usr/local/share/dotnet/shared/Microsoft.NETCore.App/5.0.13/libcoreclr.dylib")
 (type = "Microsoft.NETCore.App", version = v"5.0.12", path = "/usr/local/share/dotnet/shared/Microsoft.NETCore.App/5.0.12/libcoreclr.dylib")
 (type = "Microsoft.NETCore.App", version = v"5.0.11", path = "/usr/local/share/dotnet/shared/Microsoft.NETCore.App/5.0.11/libcoreclr.dylib")
 (type = "Microsoft.NETCore.App", version = v"3.1.26", path = "/usr/local/share/dotnet/shared/Microsoft.NETCore.App/3.1.26/libcoreclr.dylib")
 (type = "Microsoft.NETCore.App", version = v"3.1.22", path = "/usr/local/share/dotnet/shared/Microsoft.NETCore.App/3.1.22/libcoreclr.dylib")
 (type = "Microsoft.NETCore.App", version = v"3.1.21", path = "/usr/local/share/dotnet/shared/Microsoft.NETCore.App/3.1.21/libcoreclr.dylib")
 (type = "Microsoft.NETCore.App", version = v"3.1.20", path = "/usr/local/share/dotnet/shared/Microsoft.NETCore.App/3.1.20/libcoreclr.dylib")

I am running a different version than you are. I tried removing 7.0.2 so that it would use the earlier version. That did not work. So this could be something MacOS specific.

nwamsley1 commented 1 year ago

From Thermo's documentation 

// This code has been compiled and tested using Visual Studio 2013 Update 5, Visual Studio 
// 2015 Update 3, Visual Studio 2017 Update 2, Visual Studio 2019, and Visual Studio 2022, Visual
// Studio MAC, and MonoDevelop.  Additional tools used include Resharper 2017.1.  This application
// has been tested with .Net Framework 4.5.1, 4.5.2, 4.6, 4.6.1, 4.6.2, 4.7, and 4.8.
nwamsley1 commented 1 year ago

I think it is strange that in all the file error messages I get, a space is appended to the front of the file paths. Even though the string I provide clearly does not have the space. Could be a red herring, but if not I could try and figure out at what step that space gets added.

azurefx commented 1 year ago

Just tested it on my Mac. As I guessed, that space is part of the error message's string.

shell> touch empty

julia> rawFile = RawFileAdapter.FileFactory("empty")
ThermoFisher.CommonCore.RawFileReader.RawFileAccess("ThermoFisher.CommonCore.RawFileReader.RawFileAccess")

julia> rawFile.IsOpen
false

julia> rawFile.IsError
true

julia> rawFile.FileError
ThermoFisher.CommonCore.RawFileReader.Facade.RawFileLoader("ThermoFisher.CommonCore.RawFileReader.Facade.RawFileLoader")

julia> rawFile.FileError.WarningMessage
"Information: Creating mutex for: empty.raw\nInformation: Created mutex for: empty.raw\nInformation: Release mutex for: empty.raw\nInformation: Close mutex for: empty.raw\n"

julia> rawFile.FileError.ErrorMessage
"Named maps are not supported. empty.raw"

julia> rawFile.FileError.ErrorCode
2

julia> versioninfo()
Julia Version 1.5.0
Commit 96786e22cc (2020-08-01 23:44 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin18.7.0)
  CPU: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-9.0.1 (ORCJIT, skylake)

julia> DotNET.detect_runtime(DotNET.CoreCLR.CoreCLRHost)
1-element Array{Any,1}:
 (type = "Microsoft.NETCore.App", version = v"3.1.7", path = "/usr/local/share/dotnet/shared/Microsoft.NETCore.App/3.1.7/libcoreclr.dylib")

This library creates named memory-mapped files in system memory when reading files, and names are a Windows-only feature.

PlatformNotSupportedException .NET Core and .NET 5+ only: Calls to the CreateNew method with a named memory mapped file (that is, a non-null mapName) are supported on Windows operating systems only.

Here is the minimal code to reproduce the problem:

julia> MemoryMappedFile = T"System.IO.MemoryMappedFiles.MemoryMappedFile, System.IO.MemoryMappedFiles"
System.IO.MemoryMappedFiles.MemoryMappedFile

julia> MemoryMappedFile.CreateNew(null, 1024) # success
System.IO.MemoryMappedFiles.MemoryMappedFile("System.IO.MemoryMappedFiles.MemoryMappedFile")

julia> MemoryMappedFile.CreateNew("mmap with a name", 1024) # failure
ERROR: CLRException: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
 ---> System.PlatformNotSupportedException: Named maps are not supported.
   at System.IO.MemoryMappedFiles.MemoryMappedFile.CreateCore(FileStream fileStream, String mapName, HandleInheritability inheritability, MemoryMappedFileAccess access, MemoryMappedFileOptions options, Int64 capacity)
   at System.IO.MemoryMappedFiles.MemoryMappedFile.CreateNew(String mapName, Int64 capacity, MemoryMappedFileAccess access, MemoryMappedFileOptions options, HandleInheritability inheritability)
   at System.IO.MemoryMappedFiles.MemoryMappedFile.CreateNew(String mapName, Int64 capacity)
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.RuntimeType.InvokeMember(String name, BindingFlags bindingFlags, Binder binder, Object target, Object[] providedArgs, ParameterModifier[] modifiers, CultureInfo culture, String[] namedParams)
   at CLRBridge.Meta.InvokeMember(IntPtr type, String name, BindingFlags bindingFlags, IntPtr binder, IntPtr target, IntPtr[] providedArgs, UInt32 argCount, IntPtr& exception)
Stacktrace:
 [1] track_and_throw(::UInt64) at /Users/azure/.julia/packages/DotNET/Rx490/src/CLRBridge.jl:46
(omitted)

Unfortunately this library doesn't seem to be cross-platform ready, you may need a Windows computer.

nwamsley1 commented 1 year ago

Ah OK. I can get that same error on my Mac, so it is probably platform specific. I let the developers of the RawFileReader package know and pointed them to this thread. I think in the mean time I have figured out an alternative solution to using the files. I can convert to a Parquet file format using C# or Python and then use the .parquet files in Julia. Thanks for your replies and help!

nwamsley1 commented 1 year ago

I have used these library in python using the pythnonnet package. It works when I use the mono runtime, but I just discovered that when I use the coreclr runtime, I get the exact same error we are getting here.