[NAOT][Apple][mobile] Undefined symbols for architecture x86_64: do_vxsort_avx2 causing AppleAppBuilderTask to fail

matouskozak commented 5 months ago

Build Information

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=629964 Build error leg or test failing:

tvossimulator-x64 Release AllSubsets_NativeAOT_RuntimeTests
iossimulator-x64 Release AllSubsets_NativeAOT_RuntimeTests
Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "/Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018: Undefined symbols for architecture x86_64: [/Users/runner/work/1/s/src/tests/build.proj]",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Error log:

normal/x86_64/nativeaot_SmokeTests_UnitTests_dependency_info.dat -o /Users/runner/work/1/s/artifacts/tests/coreclr/obj/iossimulator.x64.Release/Managed/build/iOSApps/nativeaot_SmokeTests_UnitTests/bin-nativeaot_SmokeTests_UnitTests-x64/nativeaot_SmokeTests_UnitTests/Debug-iphonesimulator/nativeaot_SmokeTests_UnitTests.app/nativeaot_SmokeTests_UnitTests [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3624320Z /Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018: Undefined symbols for architecture x86_64: [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3624960Z /Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018:   "do_vxsort_avx2(unsigned char**, unsigned char**, unsigned char*, unsigned char*)", referenced from: [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3625650Z /Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018:       WKS::gc_heap::plan_phase(int) in libRuntime.WorkstationGC.a(gcwks.cpp.o) [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3626330Z /Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018:   "do_vxsort_avx512(unsigned char**, unsigned char**, unsigned char*, unsigned char*)", referenced from: [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3627020Z /Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018:       WKS::gc_heap::plan_phase(int) in libRuntime.WorkstationGC.a(gcwks.cpp.o) [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3627650Z /Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018:   "IsSupportedInstructionSet(InstructionSet)", referenced from: [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3628360Z /Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018:       WKS::gc_heap::grow_mark_list() in libRuntime.WorkstationGC.a(gcwks.cpp.o) [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3629010Z /Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018:       WKS::gc_heap::plan_phase(int) in libRuntime.WorkstationGC.a(gcwks.cpp.o) [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3629730Z /Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018:       WKS::gc_heap::do_post_gc() in libRuntime.WorkstationGC.a(gcwks.cpp.o) [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3630350Z /Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018:   "InitSupportedInstructionSet(int)", referenced from: [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3631020Z /Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018:       WKS::gc_heap::initialize_gc(unsigned long, unsigned long, unsigned long) in libRuntime.WorkstationGC.a(gcwks.cpp.o) [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3631690Z /Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018: ld: symbol(s) not found for architecture x86_64 [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3632460Z /Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018: clang: error: linker command failed with exit code 1 (use -v to see invocation) [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3633540Z /Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018: note: Run script build phase 'Generate CMakeFiles/ALL_BUILD' will be run during every build because the option to run the script phase "Based on dependency analysis" is unchecked. (in target 'ALL_BUILD' from project 'nativeaot_SmokeTests_UnitTests') [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3634320Z /Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018: ** BUILD FAILED ** [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3634840Z /Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018: The following build commands failed: [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3636140Z /Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018:  Ld /Users/runner/work/1/s/artifacts/tests/coreclr/obj/iossimulator.x64.Release/Managed/build/iOSApps/nativeaot_SmokeTests_UnitTests/bin-nativeaot_SmokeTests_UnitTests-x64/nativeaot_SmokeTests_UnitTests/Debug-iphonesimulator/nativeaot_SmokeTests_UnitTests.app/nativeaot_SmokeTests_UnitTests normal (in target 'nativeaot_SmokeTests_UnitTests' from project 'nativeaot_SmokeTests_UnitTests') [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3637100Z /Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018: (1 failure) [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3637870Z /Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018:    at Utils.RunProcess(TaskLoggingHelper logger, String path, String args, IDictionary`2 envVars, String workingDir, Boolean ignoreErrors, Boolean silent, MessageImportance debugMessageImportance) in /_/src/tasks/Common/Utils.cs:line 123 [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3638790Z /Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018:    at Xcode.BuildAppBundle(String xcodePrjPath, Boolean optimized, String devTeamProvisioning) in /_/src/tasks/AppleAppBuilder/Xcode.cs:line 644 [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3639540Z /Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018:    at AppleAppBuilderTask.Execute() in /_/src/tasks/AppleAppBuilder/AppleAppBuilder.cs:line 334 [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3640320Z /Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute() [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3641250Z /Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask) [/Users/runner/work/1/s/src/tests/build.proj]
2024-04-03T21:32:33.3641830Z     0 Warning(s)
2024-04-03T21:32:33.3641990Z     6 Error(s)

Report

Build	Definition	Step Name	Console log	Pull Request
632885	dotnet/runtime	Build Tests	Log	dotnet/runtime#100389
631365	dotnet/runtime	Build Tests	Log	dotnet/runtime#100389
630062	dotnet/runtime	Build Tests	Log	dotnet/runtime#100389
629964	dotnet/runtime	Build Tests	Log

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	4	4

Known issue validation

Build: :mag_right: https://dev.azure.com/dnceng-public/public/_build/results?buildId=629964 Error message validated: [/Users/runner/work/1/s/src/mono/msbuild/apple/build/AppleBuild.targets(296,5): error MSB4018: Undefined symbols for architecture x86_64: [/Users/runner/work/1/s/src/tests/build.proj]] Result validation: :white_check_mark: Known issue matched with the provided build. Validation performed at: 4/4/2024 7:53:18 PM UTC

dotnet-policy-service[bot] commented 5 months ago

Tagging subscribers to this area: @agocke, @MichalStrehovsky, @jkotas See info in area-owners.md if you want to be subscribed.

dotnet-policy-service[bot] commented 5 months ago

Tagging subscribers to 'os-ios': @steveisok, @akoeplinger, @kotlarmilos See info in area-owners.md if you want to be subscribed.

dotnet-policy-service[bot] commented 5 months ago

Tagging subscribers to 'os-tvos': @steveisok, @akoeplinger See info in area-owners.md if you want to be subscribed.

matouskozak commented 5 months ago

Could it be caused by https://github.com/dotnet/runtime/pull/98712 @cshung @janvorli?

cshung commented 5 months ago

Yes, it is my problem. For NativeAOT compile on x64 platform, we need to link one more library. Who would be a good person to talk to about AppleAppBuilderTask?

am11 commented 5 months ago

Unlike other smoke tests, seems like mono library-mode smoke tests are using aotsdk from package while the aot runtime build took place with live bits, hence the conflict. It should be using live build for both. @ivanpovazan any idea how to bring library-mode tests to the same plan as rest of the platforms (use live build of nativeaot)?

ivanpovazan commented 5 months ago

Unlike other smoke tests, seems like mono library-mode smoke tests are using aotsdk from package while the aot runtime build took place with live bits, hence the conflict. It should be using live build for both. @ivanpovazan any idea how to bring library-mode tests to the same plan as rest of the platforms (use live build of nativeaot)?

I think that all NativeAOT runtime tests (tests under: https://github.com/dotnet/runtime/tree/main/src/tests) are using aotsdk based on: https://github.com/dotnet/runtime/blob/ffb257818db89672d84d538964cec1b54fb2a094/src/tests/Directory.Build.targets#L563-L563 so I don't think that is the problem.

The problem seems to be specific to inclusion of libRuntime.Vxsort*.a libraries and looking at the binlogs from the failed test build, it seems that when building runtime tests for iossimulator (most probably all ios platforms) _targetArchitecture is not being set, which in turn prevents the required library to be included in NativeLibrary item group and further passed to the native linker when bundling iOS app bundle.

@matouskozak @kotlarmilos do you remember why we had to explicitly set these properties here: https://github.com/dotnet/runtime/blob/e61064949214b9eebdbf9faad58b71afd240ff15/src/tests/Directory.Build.props#L213-L216 I would assume we were hitting similar problems for _targetOS, as we are now having with _targetArchitecture not being resolved at the adequate time during build.

kotlarmilos commented 5 months ago

@matouskozak @kotlarmilos do you remember why we had to explicitly set these properties here:

They are used in https://github.com/dotnet/runtime/blob/main/src/coreclr/nativeaot/BuildIntegration/Microsoft.NETCore.Native.Unix.targets.

ivanpovazan commented 5 months ago

Yes, but _targetOS (and _targetArchitecture) should be resolved in https://github.com/dotnet/runtime/blob/main/src/coreclr/nativeaot/BuildIntegration/Microsoft.DotNet.ILCompiler.SingleEntry.targets so I was asking if you remember why did we have to set it manually

kotlarmilos commented 5 months ago

Can you check from the binlog if the condition in Microsoft.DotNet.ILCompiler.SingleEntry.targets is true? <PropertyGroup Condition="'$(RuntimeIdentifier)' != '' and '$(NETCoreSdkPortableRuntimeIdentifier)' != ''">

am11 commented 5 months ago

I think that all NativeAOT runtime tests (tests under: main/src/tests) are using aotsdk based on:

Yes, that is resolved to the live build of runtime. I was suspecting that iOS platforms end up using BuildIntegration targets from the SDK (which is still at preview 1 per global.json), rather than the clr artifacts directory (live build) which causes the missing libvxsort in the logs and lead to the missing symbol during the link. With current situation, any time we will add a new dependency or remove one, it will run into the same error.

Thanks for looking into it everyone. 👍

cshung commented 5 months ago

Ideally, the test should use the target files in the repo, if we could do that, that would be the best.

ivanpovazan commented 5 months ago

@cshung the problem is most probably related to how we configure runtime tests for NativeAOT on iOS, and not with the build integration targets file.

@kotlarmilos NETCoreSdkPortableRuntimeIdentifier is not set.

As a short-term solution, we could probably just add <_targetArchitecture>$(TargetArchitecture)</_targetArchitecture> in the place I linked above.
As a long-term solution, we might want to investigate how we can configure NativeAOT runtime tests to include Microsoft.NET.Sdk.props and with it Microsoft.NETCoreSdk.BundledVersions.props which defines NETCoreSdkPortableRuntimeIdentifier. This way we would avoid any similar future problems with changes introduced to NativeAOT build (or its dependencies).

dotnet / runtime