Android on-device tests that fail under LLVM

[jonathanpeppers]

Description

We have a set of on-device tests that only fail when building with -p:EnableLLVM=true.

Error message
Expected: True
But was:  False
Stack trace
   at Java.InteropTests.JniValueMarshalerContractTests`1[[System.Nullable`1[[System.Boolean, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].CreateValue()
   at System.Reflection.MethodInvoker.InterpretedInvoke(Object , IntPtr* )
   at System.Reflection.MethodInvoker.Invoke(Object , IntPtr* , BindingFlags )

Example:

• https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=8098222&view=ms.vss-test-web.build-test-results-tab&runId=80171849&resultId=100662&paneView=debug
• https://github.com/xamarin/java.interop/blob/151b03ee1eefa62e3d29b034b2aaf05ea73d99ed/tests/Java.Interop-Tests/Java.Interop/JniValueMarshalerContractTests.cs#L401-L419

They only fail on bool, sbyte, char, and short. The tests pass if -p:EnableLLVM is false (which is the default for customers) and they pass for the other data types.

Reproduction Steps

Build and run the xamarin/xamarin-android device test project with -c Release -p:EnableLLVM=true:

• https://github.com/xamarin/xamarin-android/blob/main/tests/Mono.Android-Tests/Java.Interop-Tests/Java.Interop-Tests.NET.csproj
• https://github.com/xamarin/xamarin-android/blob/main/Documentation/workflow/UnitTests.md#on-device-unit-tests

Expected behavior

Tests pass when -p:EnableLLVM=true.

Actual behavior

Tests fail when -p:EnableLLVM=true.

Regression?

Yes, this worked in .NET 6 & .NET 7 and past .NET 8 previews.

Known Workarounds

Use -p:EnableLLVM=false instead.

Configuration

Started happening: https://github.com/xamarin/xamarin-android/pull/8176

.NET SDK: 8.0.100-preview.7.23364.32

Other information

No response

[akoeplinger]

/cc @vargaz

[vargaz]

Would it be possible to create a smaller testcase that doesn't depend on xamarin-android ? Or the functionality is android specific ?

[jonathanpeppers]

Maybe @jonpryor would know how to do this when he's back.

I'm guessing this would be something like:

• Console app running on macOS
• Mono runtime
• Bring in some amount of code from java.interop

But then you'd have to setup the <MonoAOTCompiler/> task w/ LLVM?

I don't see the required AOT pack to make this kind of project:

https://dev.azure.com/dnceng/public/_artifacts/feed/dotnet8/NuGet/Microsoft.NETCore.App.Runtime.AOT.osx-arm64.Cross.maccatalyst-arm64/overview/8.0.0-preview.7.23369.13

I think we would need Microsoft.NETCore.App.Runtime.AOT.osx-arm64.Cross.osx-arm64 to do this on an M1, which doesn't exist?

Maybe the only way to do this is on Android.

[SamMonoRT]

cc @fanyang-mono

[vargaz]

Tried extracting a testcase out of the test suite into a standalone app, but can't reproduce the problem.

[SamMonoRT]

This seems like a test case failure. Moving to 9.0.0, but based on ongoing investigations, if needed will backport fix to 8.0\rc1