search

dotnet / sdk

Core functionality needed to create .NET Core projects, that is shared between Visual Studio and CLI
https://dot.net/core
MIT License
2.71k stars 1.06k forks source link

Updating from 7.0.100 to 7.0.101 crashes msbuild on mac #30271

Open mattleibow opened 1 year ago

mattleibow commented 1 year ago

Describe the bug

We were using 7.0.100-rtm.22519.39 to build maui, but when we updated to 7.0.101-servicing.22571.1 the mac builds crashed.

To Reproduce

  1. Use your mac
  2. Clone the PR: https://github.com/dotnet/maui/pull/12152
  3. Restore dotnet tools:
    dotnet tool restore
  4. Install a local dotnet dotnet cake --target=dotnet
  5. Use the local dotnet and build the msbuild tasks:
    bin/dotnet/dotnet build Microsoft.Maui.BuildTasks.slnf /target:Build /property:configuration=Debug /maxcpucount:1 /binarylogger:Microsoft.Maui.BuildTasks-mac-Debug-Build-dotnet.binlog
  6. Use the local dotnet and build the solution:
    bin/dotnet/dotnet build Microsoft.Maui-mac.slnf /target:Build /property:configuration=Debug /maxcpucount:1 /binarylogger:Microsoft.Maui-mac-Debug-Build-dotnet.binlog

I have also branches and devops builds:

There are other failures in the CI that are because I was trying to get a smaller repro without all the windows builds, but this always happens on mac in the "Build .NET MAUI" stage.

Exceptions (if any)

Further technical details

dsplaisted commented 1 year ago

@agocke @jtschuster @LakshanF @MichalStrehovsky @tlakollo @vitek-karas @VSadov I'm not sure exactly where to route this, could someone look at this MacOS native crash during a build that seems to have been introduced in 7.0.101?

agocke commented 1 year ago

MAUI goes to Mono. @steveisok

steveisok commented 1 year ago

MAUI goes to Mono. @steveisok

During build time I believe it's still using desktop .NET w/ CoreCLR. The mono runtime becomes active when running the app.

agocke commented 1 year ago

Ah, I misunderstood. I didn't realize that this was dotnet build itself crashing. @dsplaisted what is it that we should be looking at? The binary log here appears to be corrupted (I can't open it).

MichalStrehovsky commented 1 year ago

The crash dump is some sort of JSON, the interesting bit:

  "exception" : {"codes":"0x0000000000000001, 0x0000000000000000","rawCodes":[1,0],"type":"EXC_CRASH","signal":"SIGKILL (Code Signature Invalid)"},
  "termination" : {"namespace":"CODESIGNING","flags":0,"code":2},

Looks like we were already running some code when macOS decided it's not going to let us finish. I don't know who's the unlucky person on the team who deals with this Apple stuff. Cc @mangod9 @janvorli?

janvorli commented 1 year ago

I'll take a look.

janvorli commented 1 year ago

I was trying to repro it on my mac, but the build seems to require Android SDK. Since I can only access the mac via SSH console, is there a way to install it from console?

VSadov commented 1 year ago

I see an msbuild crash om musl-arm as well

That is in https://github.com/dotnet/runtime/pull/79894 , not sure if related. https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/118290/logs/995

/__w/1/s/.dotnet/sdk/7.0.100/MSBuild.dll /nologo -maxcpucount /m -verbosity:m /v:minimal /bl:/__w/1/s/artifacts/log/Debug/ToolsetRestore.binlog /clp:Summary /clp:ErrorsOnly;NoSummary /nr:false /p:TreatWarningsAsErrors=true /p:ContinuousIntegrationBuild=true /p:__ToolsetLocationOutputFile=/__w/1/s/artifacts/toolset/8.0.0-beta.22606.1.txt /t:__WriteToolsetLocation /warnaserror /__w/1/s/artifacts/toolset/restore.proj
/__w/1/s/eng/common/tools.sh: line 467:   637 Segmentation fault      (core dumped) "$_InitializeBuildTool" "$@"
MichalStrehovsky commented 1 year ago

I see an msbuild crash om musl-arm as well

That is in dotnet/runtime#79894 , not sure if related. https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/118290/logs/995

/__w/1/s/.dotnet/sdk/7.0.100/MSBuild.dll /nologo -maxcpucount /m -verbosity:m /v:minimal /bl:/__w/1/s/artifacts/log/Debug/ToolsetRestore.binlog /clp:Summary /clp:ErrorsOnly;NoSummary /nr:false /p:TreatWarningsAsErrors=true /p:ContinuousIntegrationBuild=true /p:__ToolsetLocationOutputFile=/__w/1/s/artifacts/toolset/8.0.0-beta.22606.1.txt /t:__WriteToolsetLocation /warnaserror /__w/1/s/artifacts/toolset/restore.proj
/__w/1/s/eng/common/tools.sh: line 467:   637 Segmentation fault      (core dumped) "$_InitializeBuildTool" "$@"

We track that in https://github.com/dotnet/runtime/issues/76759. On Linux, it cannot be code signing related (if we're to trust the dump JSON).

rmarinho commented 1 year ago

Any updates on this issue? I see https://github.com/dotnet/runtime/pull/79894 was merged, does that help us?

steveisok commented 1 year ago

@janvorli Please let me know if you need help getting set up with anything mobile. This is a P0 issue for the maui team

janvorli commented 1 year ago

@steveisok I am stuck on the build failing due to requiring Android SDK. If you can share details on how to install it on macOS from the command line, it would be awesome. I don't have my macs connected to screens, I just SSH to them.

steveisok commented 1 year ago

@Redth @mattleibow Jan should install the maui workload here, right? If so, can you provide him with the right command?

mattleibow commented 1 year ago

Ah no. We are trying to build maui.

Step 4 will install the sdk and required workloads I to the Maui directory. A separate install of the sdk.

steveisok commented 1 year ago

@janvorli It may be easier to follow the issue steps since that'll bring down the android sdk.

janvorli commented 1 year ago

I've followed the steps and it didn't. It failed with a message that the Android SDK is missing.

janvorli commented 1 year ago

Let me run it again to be able to share the exact error message, I don't have the console where I've tried that last time open anymore.

mattleibow commented 1 year ago

It won't bring down the android SDK. Do you have VS installed? That might be easier as you will need the Android SDK, OpenJDK and whatever else comes with VS.

janvorli commented 1 year ago

I don't have VS installed, as I don't have access to the GUI, only the command line. Or actually, I could wire it up, but it would be a bit of a hassle, so I was hoping I could avoid it.

The error I am getting is that the step 5 is failing with: /Users/janvorli/git/maui/bin/dotnet/packs/Microsoft.Android.Sdk.Darwin/33.0.4/tools/Xamarin.Android.Tooling.targets(70,5): error XA5300: The Android SDK directory could not be found. Check that the Android SDK Manager in Visual Studio shows a valid installation. To use a custom SDK path for a command line build, set the 'AndroidSdkDirectory' MSBuild property to the custom path. [/Users/janvorli/git/maui/src/Core/src/Core.csproj::TargetFramework=net7.0-android]

Before this error, I can also see the following warning: /Users/janvorli/git/maui/bin/dotnet/packs/Microsoft.Android.Sdk.Darwin/33.0.4/tools/Xamarin.Android.Tooling.targets(70,5): warning : An exception occurred while reading the output of '/usr/libexec/java_home -X'. Exception: Root element is missing. [/Users/janvorli/git/maui/src/Core/src/Core.csproj::TargetFramework=net7.0-android]

steveisok commented 1 year ago

It won't bring down the android SDK. Do you have VS installed? That might be easier as you will need the Android SDK, OpenJDK and whatever else comes with VS.

My bad. I thought it did.

@janvorli this should help https://github.com/dotnet/runtime/blob/main/docs/workflow/testing/libraries/testing-android.md#using-a-terminal

mattleibow commented 1 year ago

Yeah, those errors are saying you don't have the Android SDK and Java SDK installed.

mattleibow commented 1 year ago

One thing you can do is try building without Android, but maybe might hide the issue. In the root of the repo, there is a file: Directory.Build.Override.props.in

Rename that to Directory.Build.Override.props and set all the <_Include*> values to true except Android and Tizen:

https://gist.github.com/mattleibow/d03066db457a7b0ec8d11354fceac43d

If that works, then it may be the Android projects or something. In that case you may have to download and extract the Android SDK from Google. @jonathanpeppers maybe there is a way to do this better? You have a script to get the Android pre-requisites?

Also, this is probably going to have issues if you do not have Xcode installed as we need that to build iOS and Mac...

janvorli commented 1 year ago

Thanks for the pointers, I'll give it a try.

janvorli commented 1 year ago

@mattleibow if I disable the Android and Tizen as you have suggested, then step 5 passes, but step 6 fails with errors like /Users/janvorli/git/maui/src/Workload/Shared/FrameworkList.targets(24,5): error MSB4018: System.IO.DirectoryNotFoundException: Could not find a part of the path '/Users/janvorli/git/maui/src/Compatibility/Core/src/bin/Debug/net7.0-android/ref/Microsoft.Maui.Controls.Compatibility.dll'. [/Users/janvorli/git/maui/src/Workload/Microsoft.Maui.Controls.Ref/Microsoft.Maui.Controls.Ref.csproj] So it seems it still tries to do something with Android.

Anyways, it seems that it is not worth the hassle trying to patch stuff and wasting your time, I'll bring my mac mini to my office tomorrow so that I can connect it to the monitor and install the VS.

janvorli commented 1 year ago

In the end, I was able to install the JDK using some instructions I've found online and Android SDK from the command line using modified instructions that @steveisok shared (I needed to modify the version to 33). The step 5 passed ok without any exclusions then.

But the step 6 failed due to an outdated xcode installed on my mac. I wanted to update it, but it needs to get the OS updated first. But for some reason, the "Software update" dialog has been sitting for almost a day in "Checking for updates" phase. I have been trying various ways to fix that, but so far without success.

I wonder though - does the issue repro on macOS arm64 too? Besides the mac mini I am trying this on, I also have M1 mini, so if arm64 had the same behavior, I could debug it there. That device is much more up to date.

janvorli commented 1 year ago

I was going to give it a try on my arm64 mac which is more up to date, but I've found that the dev/test-30000 is not present in the repo anymore. Is there some other branch that I can use instead?

mattleibow commented 1 year ago

Should be back. Just restored the branch

janvorli commented 1 year ago

@mattleibow thank you!

Redth commented 1 year ago

@janvorli any updates on this one?

janvorli commented 1 year ago

@Redth I have resolved the issues with my x64 mac and managed to install the JDK and Android SDK there. Unfortunately, I was unable to make the step 5 of the build work yet (the iOS parts are failing). With the help of @mattleibow and other folks, I have found that I have some strange kind of Apple Developer Account (Organization) that doesn't let me create / get the provisioning profiles. The tabs / menus to do that are simply missing in my view of the App Store Connect web.

I have contacted the account holder to help me figure out how to fix that, but he didn't get back to me yet.

mattleibow commented 1 year ago

Any updates? Looks like net8 builds are fine, so looks to just be an issue with net7 things.

janvorli commented 1 year ago

I am still in the same state w.r.t. my Apple account. I have asked someone from our team yesterday to help investigate the issue using his Apple account that doesn't have the issues I do. I hope to get some news from him today.

janvorli commented 1 year ago

With @mattleibow's help, I was finally able to successfully run all the steps and repro the issue. And after some struggling, with instrumented libcoreclr.dylib, I was able to attach lldb to it at the time of crash. But the crash is really weird. The crashing thread sits at the first instruction of the libSkiaSharp.dylib!sk_picture_recorder_new that was just called from managed code. And yet the error is SIGKILL (Code Signature Invalid) as mentioned above. Actually, looking at the libSkiaSharp.dylib, it doesn't seem to be signed. But if that was the problem, I would expect it to be 100% repro. In reality, it usually repros at the first clean build and then if I repeat the last step, it seems to repro only on some runs. Do I assume it correctly that the sk_picture_recorder_new function is called on every build run?

Btw, here is the call stack at the time of the crash:

(lldb) clrstack -f
OS Thread Id: 0x36cc56 (28)
        Child SP               IP Call Site
0000700003037B78 00000001BF91F640 libSkiaSharp.dylib!sk_picture_recorder_new
0000700003037B80 0000000111F3A5EA
0000700003037B88                  [InlinedCallFrame: 0000700003037b88]
0000700003037B88                  [InlinedCallFrame: 0000700003037b88]
0000700003037B80 0000000111F3A5EA Microsoft.DotNet.InternalAbstractions.dll!ILStubClass.IL_STUB_PInvoke() + 90
0000700003037C00 0000000111F27AAD SkiaSharp.dll!SkiaSharp.SkiaApi.sk_picture_recorder_new() + 173
0000700003037C30 0000000111F2795E SkiaSharp.dll!SkiaSharp.SKPictureRecorder..ctor() + 30
0000700003037C70 0000000111F276AE Svg.Skia.dll!Svg.Skia.SkiaModelExtensions.ToSKPicture(ShimSkiaSharp.SKPicture) + 158 [/_/src/Svg.Skia/SkiaModelExtensions.cs @ 930]
0000700003037CE0 0000000111EE1020 Svg.Skia.dll!Svg.Skia.SKSvg.Load(System.String) + 256 [/_/src/Svg.Skia/SKSvg.cs @ 70]
0000700003037D60 0000000111EE04AD Microsoft.Maui.Resizetizer.dll!Microsoft.Maui.Resizetizer.SkiaSharpSvgTools..ctor(System.String, System.Nullable`1<SkiaSharp.SKSize>, System.Nullable`1<SkiaSharp.SKColor>, System.Nullable`1<SkiaSharp.SKColor>, Microsoft.Maui.Resizetizer.ILogger) + 253
0000700003037E00 0000000111EDF7F8 Microsoft.Maui.Resizetizer.dll!Microsoft.Maui.Resizetizer.SkiaSharpTools.Create(Boolean, System.String, System.Nullable`1<SkiaSharp.SKSize>, System.Nullable`1<SkiaSharp.SKColor>, System.Nullable`1<SkiaSharp.SKColor>, Microsoft.Maui.Resizetizer.ILogger) + 200
0000700003037E60 0000000111EE1534 Microsoft.Maui.Resizetizer.dll!Microsoft.Maui.Resizetizer.SkiaSharpAppIconTools..ctor(Microsoft.Maui.Resizetizer.ResizeImageInfo, Microsoft.Maui.Resizetizer.ILogger) + 644
0000700003037F90 0000000111EDD7CD Microsoft.Maui.Resizetizer.dll!Microsoft.Maui.Resizetizer.ResizetizeImages.ProcessAppIcon(Microsoft.Maui.Resizetizer.ResizeImageInfo, System.Collections.Concurrent.ConcurrentBag`1<Microsoft.Maui.Resizetizer.ResizedImageInfo>) + 1325
0000700003038220 0000000111EDBA71 Microsoft.Maui.Resizetizer.dll!Microsoft.Maui.Resizetizer.ResizetizeImages+<>c__DisplayClass26_0.<ExecuteAsync>b__0(Microsoft.Maui.Resizetizer.ResizeImageInfo) + 193
00007000030382E0 0000000111EDB8F9 Microsoft.Maui.Resizetizer.dll!Microsoft.Maui.Resizetizer.AsyncTaskExtensions+<>c__DisplayClass0_0`1[[System.__Canon, System.Private.CoreLib]].<ParallelForEach>b__0(System.__Canon) + 89
0000700003038340 000000010E725364 System.Threading.Tasks.Parallel.dll!System.Threading.Tasks.Parallel+<>c__DisplayClass19_0`1[[System.__Canon, System.Private.CoreLib]].<ForWorker>b__1(System.Threading.Tasks.RangeWorker ByRef, Int32, Boolean ByRef) + 740
0000700003038400 000000010E724798 System.Threading.Tasks.Parallel.dll!System.Threading.Tasks.TaskReplicator+Replica.Execute() + 120
0000700003038450 000000010ECB4119 System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) + 121
00007000030384A0 000000010E72432C System.Private.CoreLib.dll!System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread) + 188
0000700003038520 000000010ECB5A10 System.Private.CoreLib.dll!System.Threading.Tasks.ThreadPoolTaskScheduler.TryExecuteTaskInline(System.Threading.Tasks.Task, Boolean) + 144
0000700003038560 000000010ECB3D59 System.Private.CoreLib.dll!System.Threading.Tasks.TaskScheduler.TryRunInline(System.Threading.Tasks.Task, Boolean) + 121
0000700003038590 000000010ECB5835 System.Private.CoreLib.dll!System.Threading.Tasks.Task.InternalRunSynchronously(System.Threading.Tasks.TaskScheduler, Boolean) + 165
00007000030385D0 000000010ECB5528 System.Threading.Tasks.Parallel.dll!System.Threading.Tasks.TaskReplicator.Run[[System.Threading.Tasks.RangeWorker, System.Threading.Tasks.Parallel]](ReplicatableUserAction`1<System.Threading.Tasks.RangeWorker>, System.Threading.Tasks.ParallelOptions, Boolean) + 424
0000700003038630 000000010ECB4F9D System.Threading.Tasks.Parallel.dll!System.Threading.Tasks.Parallel.ForWorker[[System.__Canon, System.Private.CoreLib]](Int32, Int32, System.Threading.Tasks.ParallelOptions, System.Action`1<Int32>, System.Action`2<Int32,System.Threading.Tasks.ParallelLoopState>, System.Func`4<Int32,System.Threading.Tasks.ParallelLoopState,System.__Canon,System.__Canon>, System.Func`1<System.__Canon>, System.Action`1<System.__Canon>) + 797
00007000030386D0 000000010ECB46D9 System.Threading.Tasks.Parallel.dll!System.Threading.Tasks.Parallel.ForEachWorker[[System.__Canon, System.Private.CoreLib],[System.__Canon, System.Private.CoreLib]](System.Collections.Generic.IEnumerable`1<System.__Canon>, System.Threading.Tasks.ParallelOptions, System.Action`1<System.__Canon>, System.Action`2<System.__Canon,System.Threading.Tasks.ParallelLoopState>, System.Action`3<System.__Canon,System.Threading.Tasks.ParallelLoopState,Int64>, System.Func`4<System.__Canon,System.Threading.Tasks.ParallelLoopState,System.__Canon,System.__Canon>, System.Func`5<System.__Canon,System.Threading.Tasks.ParallelLoopState,Int64,System.__Canon,System.__Canon>, System.Func`1<System.__Canon>, System.Action`1<System.__Canon>) + 681
0000700003038760 000000010ECB43A8 System.Threading.Tasks.Parallel.dll!System.Threading.Tasks.Parallel.ForEach[[System.__Canon, System.Private.CoreLib]](System.Collections.Generic.IEnumerable`1<System.__Canon>, System.Threading.Tasks.ParallelOptions, System.Action`1<System.__Canon>) + 152
00007000030387D0 0000000111EDB737 Microsoft.Maui.Resizetizer.dll!Microsoft.Maui.Resizetizer.AsyncTaskExtensions.ParallelForEach[[System.__Canon, System.Private.CoreLib]](Microsoft.Maui.Resizetizer.MauiAsyncTask, System.Collections.Generic.IEnumerable`1<System.__Canon>, System.Action`1<System.__Canon>) + 423
0000700003038880 0000000111EDA5DF Microsoft.Maui.Resizetizer.dll!Microsoft.Maui.Resizetizer.ResizetizeImages.ExecuteAsync() + 591
0000700003038AA0 0000000111ED9FA4 Microsoft.Maui.Resizetizer.dll!Microsoft.Maui.Resizetizer.MauiAsyncTask.<Execute>b__0_0() + 52
0000700003038AC0 000000010EEA94B7 System.Private.CoreLib.dll!System.Threading.Tasks.Task`1[[System.__Canon, System.Private.CoreLib]].InnerInvoke() + 87
0000700003038AF0 000000010E7610D3 System.Private.CoreLib.dll!System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(System.Threading.Thread, System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) + 83
0000700003038B30 000000010E72430F System.Private.CoreLib.dll!System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread) + 159
0000700003038BB0 000000010E722145 System.Private.CoreLib.dll!System.Threading.ThreadPoolWorkQueue.Dispatch() + 325
0000700003038C10 0000000112B27030 System.Private.CoreLib.dll!System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart() + 464
0000700003038CD0 0000000112A3EE21 System.Private.CoreLib.dll!System.Threading.Thread.StartCallback() + 81
0000700003038CF0 000000010757BBA9 libcoreclr.dylib!CallDescrWorkerInternal + 124 at /Users/janvorli/git/runtime/src/coreclr/vm/amd64/calldescrworkeramd64.S:101
0000700003038D10 00000001073CF8FB libcoreclr.dylib!DispatchCallSimple(unsigned long*, unsigned int, unsigned long, unsigned int) + 219 at /Users/janvorli/git/runtime/src/coreclr/pal/inc/pal.h:4855
0000700003038D10 00000001073CF8AA libcoreclr.dylib!DispatchCallSimple(unsigned long*, unsigned int, unsigned long, unsigned int) + 138 at /Users/janvorli/git/runtime/src/coreclr/vm/threads.inl:42
0000700003038DA0 00000001073E40F8 libcoreclr.dylib!ThreadNative::KickOffThread_Worker(void*) + 136 at /Users/janvorli/git/runtime/src/coreclr/vm/comsynchronizable.cpp:0
0000700003038DF0 000000010739CCAE libcoreclr.dylib!ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) + 286 at /Users/janvorli/git/runtime/src/coreclr/vm/threads.inl:42
0000700003038DF0 000000010739CCAC libcoreclr.dylib!ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) + 284 at /Users/janvorli/git/runtime/src/coreclr/vm/threads.cpp:7298
0000700003038DF0 000000010739CC5C libcoreclr.dylib!ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) + 204 at /Users/janvorli/git/runtime/src/coreclr/vm/threads.inl:37
0000700003038DF0 000000010739CC5C libcoreclr.dylib!ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) + 204 at /Users/janvorli/git/runtime/src/coreclr/vm/threads.inl:37
0000700003038DF0 000000010739CC00 libcoreclr.dylib!ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) + 112 at /Users/janvorli/git/runtime/src/coreclr/vm/threads.cpp:7498
0000700003038E98                  [DebuggerU2MCatchHandlerFrame: 0000700003038e98]
0000700003038F00 000000010739D200 libcoreclr.dylib!ManagedThreadBase::KickOff(void (*)(void*), void*) + 32 at /Users/janvorli/git/runtime/src/coreclr/vm/threads.cpp:7582
0000700003038F00 000000010739D1E8 libcoreclr.dylib!ManagedThreadBase::KickOff(void (*)(void*), void*) + 8 at /Users/janvorli/git/runtime/src/coreclr/vm/threads.cpp:7272
0000700003038F30 00000001073E41CA libcoreclr.dylib!ThreadNative::KickOffThread(void*) + 170 at /Users/janvorli/git/runtime/src/coreclr/vm/comsynchronizable.cpp:230
0000700003038F90 00000001072AAD77 libcoreclr.dylib!CorUnix::CPalThread::ThreadEntry(void*) + 407 at /Users/janvorli/git/runtime/src/coreclr/pal/src/include/pal/thread.hpp:468
0000700003038FC0 00007FF818C594E1 libsystem_pthread.dylib!_pthread_start + 125
0000700003038FE0 00007FF818C54F6B libsystem_pthread.dylib!thread_start + 15
jonathanpeppers commented 1 year ago

@mattleibow can you add a new MSBuild property to make this code path use a plain foreach (and not Parallel.ForEach())?

https://github.com/dotnet/maui/blob/eb61bb267162cb2791d4a63c3b6b56ad7f0e8f04/src/SingleProject/Resizetizer/src/ResizetizeImages.cs#L44

My memory is that MSBuild can sometimes get a different path for $CWD on background threads depending on MSBuild nodes, and I wonder if that influences the code signing behavior? I wonder if the above problem even repros at all if it was a plain foreach.

You might also want to pass in the same parallelization options we're using here (but this might not related to this bug):

https://github.com/xamarin/xamarin-android-tools/blob/099fd95f5459d9df78af0ad28f46e3016dd7ca1d/src/Microsoft.Android.Build.BaseTasks/AsyncTaskExtensions.cs#L71-L84

It looks like the existing code doesn't handle cancellation or limit the concurrency.

mattleibow commented 1 year ago

I added some properties to control parallelization in the task: https://github.com/dotnet/maui/pull/12954

Maybe the dotnet build ran out of CPU or something changed between our RTM and GA...

mattleibow commented 1 year ago

I updated the PR and also added things to remove all the paralellism, and it still fails.

I got this PR that is now using 102: https://github.com/dotnet/maui/pull/12954

Still the same thing. Also interesting that this was working before GA and is still working on net8. So something happened the week of the GA build that broke it but is either not in net8 or different in net8.

mattleibow commented 1 year ago

Managed to get numbers of SDK here, but

janvorli commented 1 year ago

I have extracted the runtime commits for the above mentioned SDKs and ran a git bisect to try to find out the one causing the trouble. I have first used the older version of the SDK that is marked as green as a basis, implanting in the files from the build of runtime from each commit the bisect picked. To my surprise, the issue didn't repro with any of those. Then I've tried to use the GA SDK that was failing, again implanting in the same set of version of runtime I've built. And this time, it failed for all of them. That makes me conclude that the issue is not in the runtime (shared/Microsoft.NETCore.App part of the SDK), but rather in some other stuff in the SDK.

mandel-macaque commented 1 year ago

@janvorli @mattleibow I have some experience with coding and macOS, let me know how to get the binaries and I can quickly verify the possible signing issue and the exact entitlement that is missing.

mattleibow commented 1 year ago

There are the few steps in the description with how to get a build. It uses a local copy of dotnet so does not affect things.

mattleibow commented 1 year ago

@jonathanpeppers had a look at all this and discovered that adding the /maxcpucount:1 parameter causes the crash: https://github.com/dotnet/sdk/issues/30271

This should now be reproducible on main.