NativeAOT Compilation

[mobiusklein]

Has the library been compiled successfully as either a static or shared library with .NET 8? While I was able to get a shared library to build on Windows, it would fail on attempting to open the file with the following FileError value:

Error Encountered ThermoFisher.CommonCore.RawFileReader.Facade.MemoryMappingRawFileLoader

There were several warnings about reflection being used to compute sizes during marshaling where a compile time alternative is available as well.

This feature would let the C# library be wrapped and used as though it were an opaque C library. Using it as-is is still possible with nethost or some other native hosting layer as long as the machine already has a suitable .NET runtime version already installed.

[jshofstahl]

We do have some internal builds of RawFileReader against .Net 8 but I need to check to see if the code is ready for external release.

[mobiusklein]

Hello, I wanted to follow up to see if there's any news here.

[jshofstahl]

Our Product Evaluation (OPE) group is looking at the update. I am waiting for their final documentation and sign off before I update the files.

[mobiusklein]

Thank you.

[mobiusklein]

I was able to compile the new Net8 build as a NativeAOT static library, but I see a barrage of warnings:

ILC : AOT analysis warning IL3050: ThermoFisher.CommonCore.RawFileReader.Readers.ReaderExtensionMethods.ReadPreviousRevisionAndConvertExt<T,TPrev>(IMemoryReader,Int64&): Using member 'System.Runtime.InteropServices.Marshal.SizeOf(Type)' which has 'Requir
esDynamicCodeAttribute' can break functionality when AOT compiling. Marshalling code for the object might not be available. Use the SizeOf<T> overload instead.      
ILC : AOT analysis warning IL3050: ThermoFisher.CommonCore.RawFileReader.Readers.MemoryArrayAccessor.WriteStruct<T>(Int64,!!0): Using member 'System.Runtime.InteropServices.Marshal.SizeOf(Type)' which has 'RequiresDynamicCodeAttribute' can break function
ality when AOT compiling. Marshalling code for the object might not be available. Use the SizeOf<T> overload instead. 
ILC : AOT analysis warning IL3050: ThermoFisher.CommonCore.RawFileReader.Readers.MemoryArrayReader.ReadPreviousRevisionAndConvert<T>(Int64,Int32): Using member 'System.Runtime.InteropServices.Marshal.SizeOf(Type)' which has 'RequiresDynamicCodeAttribute' 
 can break functionality when AOT compiling. Marshalling code for the object might not be available. Use the SizeOf<T> overload instead. 
ILC : AOT analysis warning IL3050: ThermoFisher.CommonCore.RawFileReader.Readers.MemoryArrayReader.ReadStructure<T>(Int64,Int64&): Using member 'System.Runtime.InteropServices.Marshal.SizeOf(Type)' which has 'RequiresDynamicCodeAttribute' can break funct 
ionality when AOT compiling. Marshalling code for the object might not be available. Use the SizeOf<T> overload instead. 
ILC : AOT analysis warning IL3050: ThermoFisher.CommonCore.RawFileReader.Readers.MemMapAccessor.WriteStruct<T>(Int64,!!0): Using member 'System.Runtime.InteropServices.Marshal.SizeOf(Type)' which has 'RequiresDynamicCodeAttribute' can break functionality
 when AOT compiling. Marshalling code for the object might not be available. Use the SizeOf<T> overload instead. 
ILC : AOT analysis warning IL3050: ThermoFisher.CommonCore.RawFileReader.Readers.MemMapReader.ReadPreviousRevisionAndConvert<T>(Int64,Int32): Using member 'System.Runtime.InteropServices.Marshal.SizeOf(Type)' which has 'RequiresDynamicCodeAttribute' can  
break functionality when AOT compiling. Marshalling code for the object might not be available. Use the SizeOf<T> overload instead. 
ILC : AOT analysis warning IL3050: ThermoFisher.CommonCore.RawFileReader.Readers.MemMapReader.ReadStructure<T>(Int64,Int64&): Using member 'System.Runtime.InteropServices.Marshal.SizeOf(Type)' which has 'RequiresDynamicCodeAttribute' can break functional 
ity when AOT compiling. Marshalling code for the object might not be available. Use the SizeOf<T> overload instead. 
ILC : Trim analysis warning IL2075: ThermoFisher.CommonCore.Data.CommonCoreDataObject.PerformEquals(Object,Boolean): 'this' argument does not satisfy 'DynamicallyAccessedMemberTypes.NonPublicFields' in call to 'System.Type.GetFields(BindingFlags)'. The r
eturn value of method 'System.Object.GetType()' does not have matching annotations. The source value must declare at least the same requirements as those declared on the target location it is assigned to. 
ILC : AOT analysis warning IL3050: ThermoFisher.CommonCore.RawFileReader.Readers.MemoryArrayReader.ConvertBytesToStructure<T>(IEnumerable): Using member 'System.Runtime.InteropServices.Marshal.PtrToStructure(IntPtr,Type)' which has 'RequiresDynamicCodeAt 
tribute' can break functionality when AOT compiling. Marshalling code for the object might not be available. 
ILC : AOT analysis warning IL3050: ThermoFisher.CommonCore.RawFileReader.Utilities.ReadStructure<T>(IMemoryReader,Int64&,Int32,Int32[,]): Using member 'System.Runtime.InteropServices.Marshal.PtrToStructure(IntPtr,Type)' which has 'RequiresDynamicCodeAttr
ibute' can break functionality when AOT compiling. Marshalling code for the object might not be available. 
ILC : AOT analysis warning IL3050: ThermoFisher.CommonCore.RawFileReader.Readers.MemMapReader.ConvertBytesToStructure<T>(IEnumerable): Using member 'System.Runtime.InteropServices.Marshal.PtrToStructure(IntPtr,Type)' which has 'RequiresDynamicCodeAttribu
te' can break functionality when AOT compiling. Marshalling code for the object might not be available. 
ILC : AOT analysis warning IL3050: ThermoFisher.CommonCore.RawFileReader.Utilities.GetEnumValues<T>(Type,String[]&,String): Using member 'System.Enum.GetValues(Type)' which has 'RequiresDynamicCodeAttribute' can break functionality when AOT compiling. It
 might not be possible to create an array of the enum type at runtime. Use the GetValues<TEnum> overload or the GetValuesAsUnderlyingType method instead. 
ILC : Trim analysis warning IL2075: ThermoFisher.CommonCore.RawFileReader.Utilities.GetEnumValues<T>(Type,String[]&,String): 'this' argument does not satisfy 'DynamicallyAccessedMemberTypes.PublicFields' in call to 'System.Type.GetField(String)'. The ret 
urn value of method 'System.Object.GetType()' does not have matching annotations. The source value must declare at least the same requirements as those declared on the target location it is assigned to. 
ILC : AOT analysis warning IL3050: ThermoFisher.CommonCore.RawFileReader.StructWrappers.Scan.FilterStringParser.FilterParserDictionary.<.cctor>g__AddTokenSet|3_1<T>(FilterStringParser.TokenCategory,String[],Boolean): Using member 'System.Enum.GetValues(T
ype)' which has 'RequiresDynamicCodeAttribute' can break functionality when AOT compiling. It might not be possible to create an array of the enum type at runtime. Use the GetValues<TEnum> overload or the GetValuesAsUnderlyingType method instead. 

I compiled the same .NET code that I have previously used with dynamically linking with and bootstrapping a .NET runtime via FFI, but opening a RAW file fails:

Error: Encountered problems while trying to read '../tests/data/small.RAW' as a Raw File!

Exception type: System.NotSupportedException
Message       : 'ThermoFisher.CommonCore.RawFileReader.FileIoStructs.RawFileInfoStruct3' is missing structure marshalling data. This can happen for code that is not compatible with AOT. Inspect and fix AOT related warnings that were generated when the app was published. For more information see https://aka.ms/nativeaot-compatibility
Stacktrace:
   at Internal.Runtime.CompilerHelpers.RuntimeInteropData.GetStructUnsafeStructSize(RuntimeTypeHandle) + 0xf0
   at ThermoFisher.CommonCore.RawFileReader.Readers.ReaderExtensionMethods.ReadPreviousRevisionAndConvertExt[T,TPrev](IMemoryReader, Int64&) + 0x43
   at ThermoFisher.CommonCore.RawFileReader.StructWrappers.RawFileInfo.Load(IMemoryReader, Int64, Int32) + 0xa7
   at ThermoFisher.CommonCore.RawFileReader.Readers.ReaderExtensionMethods.LoadRawFileObjectExt[T](IMemoryReader, Func`1, Int32, Int64&) + 0x34
   at ThermoFisher.CommonCore.RawFileReader.Facade.MemoryMappingRawFileLoader.ReadMetaData(IMemoryReader, Int64) + 0x9d
   at ThermoFisher.CommonCore.RawFileReader.Facade.MemoryMappingRawFileLoader.LoadRawFile(IDisposableReader, Boolean) + 0x156
   at ThermoFisher.CommonCore.RawFileReader.Facade.MemoryMappingRawFileLoader..ctor(String) + 0x22b

, Information: Creating mutex for: ../tests/data/small.RAW
Information: Created mutex for: ../tests/data/small.RAW
Information: Release mutex for: ../tests/data/small.RAW
Information: Close mutex for: ../tests/data/small.RAW

Based upon the compilation warnings, this runtime warning seems to follow from using reflection or the like in a way that isn't compatible with AOT compilation. So right now we're still going to need to ensure the .NET runtime is available separately on the system to use the RawFileReader library.

Is making the library compatible with native AOT static compilation something that Thermo is interested in doing? If not, it's good to know to avoid pouring more time into this.

[jshofstahl]

Can you provide me some more details about how you are building your application (Windows, Linux, or MacOS)? Also are you using Visual Studio or Visual Studio Code? Part of this information is so that I can try building it myself.

[mobiusklein]

All work is being done on Windows right now.

The .NET codebase is located at https://github.com/mobiusklein/thermorawfilereader.rs/tree/feature/native-aot/librawfilereader. The Rust codebase using the static library FFI is https://github.com/mobiusklein/thermorawfilereader.rs/tree/feature/native-aot/ffilib

The .NET build process is running explicitly with the dotnet CLI tool, but development is happening in Visual Studio Code, though I do have

dotnet publish librawfilereader -c Release --use-current-runtime

The statically compiled .NET code itself is able to start up, safely do basic arithmetic, call across language boundaries to pass strings, etcetera. The error message I showed is produced at https://github.com/mobiusklein/thermorawfilereader.rs/blob/feature/native-aot/librawfilereader/Lib.cs#L1028

The associated Rust program is automatically locating the AOT compilation SDK and compiling the .NET library in the build script, but it can be built independently to be linked with something else like C++, but I doubt the behavior would change.

[mobiusklein]

To verify it's not the Rust-static library linking bit causing the problem, I added a pure .NET AOT compiled executable in https://github.com/mobiusklein/thermorawfilereader.rs/blob/feature/native-aot/librawfilereader/test_rawfilereader.csproj.

This runs the same file-opening logic and checks whether there was an error set https://github.com/mobiusklein/thermorawfilereader.rs/blob/feature/native-aot/librawfilereader/Lib.cs#L1358-L1379

dotnet publish -c Release --use-current-runtime librawfilereader/test_rawfilereader.csproj
librawfilereader/bin/Release/net8.0/win-x64/publish/test_rawfilereader.exe tests\data\small.RAW

This builds the test CLI application and runs the native AOT executable, producing the following output:

1
Error: Encountered problems while trying to read 'tests\data\small.RAW' as a Raw File!

Exception type: System.NotSupportedException
Message       : 'ThermoFisher.CommonCore.RawFileReader.FileIoStructs.RawFileInfoStruct3' is missing structure marshalling data. This can happen for code that is not compatible with AOT. Inspect and fix AOT related warnings that were generated when the app was published. For more information see https://aka.ms/nativeaot-compatibility
Stacktrace:
   at Internal.Runtime.CompilerHelpers.RuntimeInteropData.GetStructUnsafeStructSize(RuntimeTypeHandle) + 0xf0
   at ThermoFisher.CommonCore.RawFileReader.Readers.ReaderExtensionMethods.ReadPreviousRevisionAndConvertExt[T,TPrev](IMemoryReader, Int64&) + 0x43
   at ThermoFisher.CommonCore.RawFileReader.StructWrappers.RawFileInfo.Load(IMemoryReader, Int64, Int32) + 0xa7
   at ThermoFisher.CommonCore.RawFileReader.Readers.ReaderExtensionMethods.LoadRawFileObjectExt[T](IMemoryReader, Func`1, Int32, Int64&) + 0x34
   at ThermoFisher.CommonCore.RawFileReader.Facade.MemoryMappingRawFileLoader.ReadMetaData(IMemoryReader, Int64) + 0x9d
   at ThermoFisher.CommonCore.RawFileReader.Facade.MemoryMappingRawFileLoader.LoadRawFile(IDisposableReader, Boolean) + 0x156
   at ThermoFisher.CommonCore.RawFileReader.Facade.MemoryMappingRawFileLoader..ctor(String) + 0x22b

, Information: Creating mutex for: tests\data\small.RAW
Information: Created mutex for: tests\data\small.RAW
Information: Release mutex for: tests\data\small.RAW
Information: Close mutex for: tests\data\small.RAW

InvalidFormat
Encountered problems while trying to read 'tests\data\small.RAW' as a Raw File!

Exception type: System.NotSupportedException
Message       : 'ThermoFisher.CommonCore.RawFileReader.FileIoStructs.RawFileInfoStruct3' is missing structure marshalling data. This can happen for code that is not compatible with AOT. Inspect and fix AOT related warnings that were generated when the app was published. For more information see https://aka.ms/nativeaot-compatibility
Stacktrace:
   at Internal.Runtime.CompilerHelpers.RuntimeInteropData.GetStructUnsafeStructSize(RuntimeTypeHandle) + 0xf0
   at ThermoFisher.CommonCore.RawFileReader.Readers.ReaderExtensionMethods.ReadPreviousRevisionAndConvertExt[T,TPrev](IMemoryReader, Int64&) + 0x43
   at ThermoFisher.CommonCore.RawFileReader.StructWrappers.RawFileInfo.Load(IMemoryReader, Int64, Int32) + 0xa7
   at ThermoFisher.CommonCore.RawFileReader.Readers.ReaderExtensionMethods.LoadRawFileObjectExt[T](IMemoryReader, Func`1, Int32, Int64&) + 0x34
   at ThermoFisher.CommonCore.RawFileReader.Facade.MemoryMappingRawFileLoader.ReadMetaData(IMemoryReader, Int64) + 0x9d
   at ThermoFisher.CommonCore.RawFileReader.Facade.MemoryMappingRawFileLoader.LoadRawFile(IDisposableReader, Boolean) + 0x156
   at ThermoFisher.CommonCore.RawFileReader.Facade.MemoryMappingRawFileLoader..ctor(String) + 0x22b

I get the same native AOT warnings about relying on reflection and runtime type information, which have the associated generic-based static dispatch alternatives called out in the earlier message