[macOS] msbuild failed with exit code null on specific task after recent update

GalaxiaGuy commented 2 years ago

Description

When trying to build a Xamarin project using the ResizetizerNT package, the build fails - msbuild failed with exit code null.

The issue is reported on the Resizetizer repo here:

https://github.com/Redth/ResizetizerNT/issues/81

But it only started happening with the update to the VM images. Previous versions worked fine and when running it locally on machines with various configurations there are no problems.

Virtual environments affected

[ ] Ubuntu 18.04
[ ] Ubuntu 20.04
[x] macOS 10.15
[X] macOS 11
[ ] Windows Server 2016
[ ] Windows Server 2019
[ ] Windows Server 2022

Image version and build link

Environment: macos-11 Version: 20211029.3

Is it regression?

Yes. All previous versions have been fine.

Expected behavior

Build completes.

Actual behavior

Build fails with:

##[debug]Exit code null received from tool '/Library/Frameworks/Mono.framework/Versions/Current/Commands/msbuild'
##[debug]STDIO streams have closed for tool '/Library/Frameworks/Mono.framework/Versions/Current/Commands/msbuild'
##[debug]task result: Failed

Repro steps

Create a Xamarin project that uses ResizetizerNT and attempt to build.

al-cheb commented 2 years ago

Hey, @GalaxiaGuy We will take a look at it.

miketimofeev commented 2 years ago

@GalaxiaGuy could you take a look at this comment https://github.com/actions/virtual-environments/issues/4060#issuecomment-958738538? Is this related?

GalaxiaGuy commented 2 years ago

@miketimofeev I don't believe so. Nuget restore seems to be working fine (and working the same as on previous successful runs).

vsafonkin commented 2 years ago

Hi @GalaxiaGuy, could you please provide some repro in public repository? Maybe sample application project with ResizetizerNT or something like this.

GalaxiaGuy commented 2 years ago

@vsafonkin https://github.com/GalaxiaGuy/xf-bugs/actions https://github.com/GalaxiaGuy/xf-bugs/tree/resizetizer

The resizetizer builds are a branch with Resizetizer installed. The other branch is just to demonstrate it builds okay otherwise.

On Github actions the error is different Process completed with exit code 137

2021-11-10T09:55:06.6704620Z /Users/runner/work/_temp/22fafe09-22ec-4dce-8368-169171187546.sh: line 1:  1801 Killed: 9               msbuild Bug.Android/Bug.Android.csproj /t:PackageForAndroid /p:Configuration=Debug
2021-11-10T09:55:06.6716150Z ##[error]Process completed with exit code 137.

vsafonkin commented 2 years ago

@GalaxiaGuy, as I see the latest version of resizetizer is 0.3.0: https://www.nuget.org/packages/Resizetizer.NT In your repro project (great thanks for it!) resizetizer is 0.2.0: https://github.com/GalaxiaGuy/xf-bugs/blob/resizetizer/Bug/Bug.csproj

I've tried to update it to 0.3.0 and project build was successful: https://github.com/vsafonkin-test-organization/xf-bugs/actions/runs/1443981192

Probably, this problem appears from resizetizer side.

RonnyRos commented 2 years ago

If i can just chip in, i have tried everything from both 0.2.0 and 0.3.0. If i remove just the resizetizer nuget from the repo, the appcenter build goes through and on azure pipelines aswell (with of course no images working). If i add 0.3.0 to Android and iOS and the shared project it fails with the same errorcode. This is due to the fact that appcenter and AzurePipelines run on a too low version SDK it seems and there the bug somewhere lies. My local iOS version is higher (15.1+), same with Android. Hope this can help abit to solve this issue.

vsafonkin commented 2 years ago

@RonnyRos, could you try to add step to setup Xamarin versions as Xamarin.iOS 15.2 and Xamarin.Android 12.0?

- name: Set default Xamarin SDK versions
  run: |
    $VM_ASSETS/select-xamarin-sdk-v2.sh --mono=6_12_11

GalaxiaGuy commented 2 years ago

@vsafonkin That is... unfortunate. Sadly, the project I'm actually working is more complicated in various ways so I'm not sure off hand what is important.

Just for reference, the timing of the problem from our side:

We were using resizetizer 0.2.0 without a problem.
Some time last week all our builds started failing on AzDo (across all our active branches) despite building fine locally.
We tried updating resizetizer to 0.3.0, but the result was the same.

I've just confirmed the current macOS 10.15 image 20211106.1 has the same problem.

macOS 11 20211029.3 and macOS 10.15 20211002.1 worked.

I also just tried adding the Xamarin SDK selection to the build and the result was the same

GalaxiaGuy commented 2 years ago

@vsafonkin I just tried updating to 0.3.0 on my repro repo, and it failed the same way: https://github.com/GalaxiaGuy/xf-bugs/runs/4165286373?check_suite_focus=true

After checking your changes, I noticed you only updated the package in the shared project, not in the platform (Android) project.

So, I downgraded the platform package version (so now shared is on 0.3.0 and Android is on 0.2.0) and it seemed to work: https://github.com/GalaxiaGuy/xf-bugs/runs/4167246595?check_suite_focus=true

Sadly, on my actual project it did not make a difference.

I'm not really sure what conclusions to draw from this, but I shall continue investigating.

RonnyRos commented 2 years ago

@vsafonkin No luck unfortunatley

RonnyRos commented 2 years ago

More info from Azure Pipeline: ProcessSharedFonts: Creating "obj/Release/sharedfont.stamp" because "AlwaysCreate" was specified.

[debug]Exit code null received from tool '/Library/Frameworks/Mono.framework/Versions/Current/Commands/msbuild'

[debug]STDIO streams have closed for tool '/Library/Frameworks/Mono.framework/Versions/Current/Commands/msbuild'

[debug]task result: Failed

[error]Xamarin.iOS task failed with error Error: The process '/Library/Frameworks/Mono.framework/Versions/Current/Commands/msbuild' failed with exit code null. For guidance on setting up the build pipeline, see https://go.microsoft.com/fwlink/?LinkId=760847.

[debug]Processed: ##vso[task.issue type=error;]Xamarin.iOS task failed with error Error: The process '/Library/Frameworks/Mono.framework/Versions/Current/Commands/msbuild' failed with exit code null. For guidance on setting up the build pipeline, see https://go.microsoft.com/fwlink/?LinkId=760847.

[debug]Processed: ##vso[task.complete result=Failed;]Xamarin.iOS task failed with error Error: The process '/Library/Frameworks/Mono.framework/Versions/Current/Commands/msbuild' failed with exit code null. For guidance on setting up the build pipeline, see https://go.microsoft.com/fwlink/?LinkId=760847.

Finishing: XamariniOS

vsafonkin commented 2 years ago

@GalaxiaGuy, @RonnyRos , could you try to remove the current mono version and install the previous mono version on the runner in runtime? As I see, msbuild version was updated: mono 6.12.0.125 -> 6.12.0.140 msbuild 16.6.0.15801 -> 16.9.0.22901

You can do it with the helper script: https://github.com/vsafonkin-test-organization/xf-bugs/blob/resizetizer/install-mono.sh I've tried to downgrade mono in the sample repro repository and it works fine.

RonnyRos commented 2 years ago

Im able to set the SDK in the pipeline, i get mono missing on nuget restore task. Also im not sure how to set the MSbuild version? Do you want me to switch to MSbuild only task instead of Xamarin.iOS task?. Again thanks alot for the help. I currently use the following simple YAML:

trigger:

iOS

pool: vmImage: 'macOS-11'

steps:

script: sudo $AGENT_HOMEDIRECTORY/scripts/select-xamarin-sdk.sh 6_12_0_125 # ive tried 6.12.0.140 aswell without luck displayName: 'Select the Xamarin SDK version' enabled: true
task: NuGetToolInstaller@1
task: NuGetCommand@2 inputs: restoreSolution: '*/.sln'
task: XamariniOS@2 inputs: solutionFile: '*/.sln' configuration: 'Release' buildForSimulator: true packageApp: false clean: true runNugetRestore: true

vsafonkin commented 2 years ago

@RonnyRos, could you please try to add script in your repository and invoke it in your pipeline? There is no Xamarin bundle 6_12_0_125 on the runner so your effort to switch Xamarin sdk is not working.

RonnyRos commented 2 years ago

I run it as Bash, not sure if thats what you intended.

task: Bash@3 inputs: filePath: '$(Build.SourcesDirectory)/install-mono.sh'

/Users/runner/work/1/s/install-mono.sh: line 1: MONO_VERSION=6.12.0.140: command not found /tmp/visualstudio.XEDj ~/work/1/s Installing Mono ... Downloading https://dl.xamarin.com/MonoFrameworkMDK/Macx86/MonoFramework-MDK-.macos10.xamarin.universal.pkg... https://dl.xamarin.com/MonoFrameworkMDK/Macx86/MonoFramework-MDK-.macos10.xamarin.universal.pkg was downloaded successfully to ./MonoFramework-MDK-.macos10.xamarin.universal.pkg Installing MonoFramework-MDK-.macos10.xamarin.universal.pkg... installer: Error - the package path specified was invalid: '/tmp/visualstudio.XEDj/MonoFramework-MDK-.macos10.xamarin.universal.pkg'. Installing nunit3-console for Mono mv: rename /tmp/visualstudio.XEDj/nunit3-console to /Library/Frameworks/Mono.framework/Versions//Commands/nunit3-console: No such file or directory Creating short symlink '' Move to backup folder

vsafonkin commented 2 years ago

/Users/runner/work/1/s/install-mono.sh: line 1: MONO_VERSION=6.12.0.140: command not found

It's weird, version 6.12.0.125 is specified in script. Please double-check that the script is transferred correctly.

RonnyRos commented 2 years ago

Got the bash working, invalid encoding (copied the content wrong). Now it downloads and runs, still the same error. If you like i can send you the pipeline log in more private manner i dont want to add it here due to privacy. Noticed this one as well: Microsoft (R) Build Engine version 16.6.0 for Mono

vsafonkin commented 2 years ago

@RonnyRos, I've tested it on Azure DevOps: yaml file: https://github.com/vsafonkin/xf-bugs/blob/resizetizer/azure-pipelines.yaml script: https://github.com/vsafonkin/xf-bugs/blob/resizetizer/install-mono.sh

Test run: https://dev.azure.com/v-vlsafo/test/_build/results?buildId=6&view=results

RonnyRos commented 2 years ago

Yes the the script works (I might of not been clear enough in the last post about that), but still the same error occurs when trying to build the solution. And again this only occurs when resizetizer is added to the projects. I probably need to find out in the iOS project whats off from this project and making it fail as MSBuild seems different.

[debug]Exit code null received from tool '/Library/Frameworks/Mono.framework/Versions/Current/Commands/msbuild'

[debug]STDIO streams have closed for tool '/Library/Frameworks/Mono.framework/Versions/Current/Commands/msbuild'

[debug]task result: Failed

[error]Xamarin.iOS task failed with error Error: The process '/Library/Frameworks/Mono.framework/Versions/Current/Commands/msbuild' failed with exit code null. For guidance on setting up the build pipeline, see https://go.microsoft.com/fwlink/?LinkId=760847.

[debug]Processed: ##vso[task.issue type=error;]Xamarin.iOS task failed with error Error: The process '/Library/Frameworks/Mono.framework/Versions/Current/Commands/msbuild' failed with exit code null. For guidance on setting up the build pipeline, see https://go.microsoft.com/fwlink/?LinkId=760847.

[debug]Processed: ##vso[task.complete result=Failed;]Xamarin.iOS task failed with error Error: The process '/Library/Frameworks/Mono.framework/Versions/Current/Commands/msbuild' failed with exit code null. For guidance on setting up the build pipeline, see https://go.microsoft.com/fwlink/?LinkId=760847.

Finishing: XamariniOS

vsafonkin commented 2 years ago

Are your project building successfully on your local machine? If yes, could you check msbuild version on your local machine?

msbuild --version

RonnyRos commented 2 years ago

Locally it works with: MsBuild 16.9.0

Pipeline seems to run: 'Found msbuild version: 16.6.0.15801'

vsafonkin commented 2 years ago

Thanks, looks like the current msbuild version on the runner is the same as on your local machine. I will continue to investigate it.

RonnyRos commented 2 years ago

No thank you, and i would guess it would need to work with the newest mono SDK and iOS ver aswell?. The resizetizer pipeline you created is running on old the SDKs.

vsafonkin commented 2 years ago

The current default Xamarin SDKs on the runner: https://github.com/actions/virtual-environments/blob/main/images/macos/macos-11-Readme.md#xamarin-bundles

I just wanted to find out if the update of msbuild is the reason of issue.

GerbenIndeqa commented 2 years ago

We have been experiencing the same issue. Also have a Xamarin solution that builds into an iOS app. Ran our last succesfull build about 2 weeks ago. Locally we can still build the project without issue on multiple DEV devices, both Windows and a Mac. We do use the ResizetizerNT package as well, which we also recently upgraded. But the pipeline also fails now when building older commits before these changes.

We are running the pipline on macOS-10.15, but I've also tried 10.14 and 11. The only thing I have found so far is a bump in the VM image from 19H1417 to 19H1419.

I've logs available from the last succesfull run, failures including system diagnostics if that can be any of help.

vsafonkin commented 2 years ago

Hi @GerbenIndeqa, your project is public or private? It will be great if we have some public repo to reproduce this issue.

GerbenIndeqa commented 2 years ago

@vsafonkin Private unfortunately

vsafonkin commented 2 years ago

@GerbenIndeqa, I see, could you please share the failures logs here then?

eddieyanez commented 2 years ago

Hi @vsafonkin, I've been experiencing the same issue and have spent the last couple of days working with MS App Center support to identify why my build has stopped working suddenly last month.

The project build on App Center again if I remove the Nuget package. The version I had installed was 0.2.0. I have since then installed the package version 0.3.0, built the app locally but found that an image was named with uppercase characters so I had to fix this first. I've then committed the change and run the builds on App Center and the builds fail again even though they work locally. My project is not public so I cannot share this with you, but I can share the below log line items from the build logs with you from the (no further info is provided).

2021-11-16T09:13:30.2834200Z ##[error]Xamarin.iOS task failed with error Error: /Library/Frameworks/Mono.framework/Versions/6_12_11/bin/msbuild failed with return code: null. For guidance on setting up the build definition, see https://go.microsoft.com/fwlink/?LinkId=760847.
2021-11-16T09:13:30.2851110Z ##[section]Finishing: Build Xamarin.iOS solution

2021-11-16T09:13:49.6486680Z ##[error]Error: The process '/Library/Frameworks/Mono.framework/Versions/6_12_11/bin/msbuild' failed with exit code null
See https://go.microsoft.com/fwlink/?LinkId=760847.
2021-11-16T09:13:49.6499730Z ##[section]Finishing: Build bundle from Xamarin.Android project

Is there anything that you'd like me to try on my code to help investigate / try a fix?

GerbenIndeqa commented 2 years ago

@vsafonkin I was a bit too quickly early on perhaps. I can't share the full logs and definitely not here. As it also contains sensitive information. But if you are looking for something specific let me know. If you have a way to share logs privately I think I can extract the log from the build task and share that with you.

eddieyanez commented 2 years ago

I can also confirm that the solution referred to here https://github.com/actions/virtual-environments/issues/4436#issuecomment-965424675 by @GalaxiaGuy addressed the problem with the build for me. However, the app will have no images.

Specifically the solution works if using this configuration:

Shared project - Resizetizer.NT (0.3.0) Android project - Resizetizer.NT (0.2.0) iOS project - Resizetizer.NT (0.2.0)

The following two configurations however do not work:

All projects using Resizetizer.NT version 0.2.0 i.e.:

Shared project - Resizetizer.NT (0.2.0) Android project - Resizetizer.NT (0.2.0) iOS project - Resizetizer.NT (0.2.0)

All projects using Resizetizer.NT version 0.3.0 i.e.:

Shared project - Resizetizer.NT (0.3.0) Android project - Resizetizer.NT (0.3.0) iOS project - Resizetizer.NT (0.3.0)

NunoBem commented 2 years ago

I can also confirm that the solution referred to here #4436 (comment) by @GalaxiaGuy addressed the problem for me.

Specifically the solution works if using this configuration:

Shared project - Resizetizer.NT (0.3.0) Android project - Resizetizer.NT (0.2.0) iOS project - Resizetizer.NT (0.2.0)

The following two configurations however do not work:

All projects using Resizetizer.NT version 0.2.0 i.e.:

Shared project - Resizetizer.NT (0.2.0) Android project - Resizetizer.NT (0.2.0) iOS project - Resizetizer.NT (0.2.0)

All projects using Resizetizer.NT version 0.3.0 i.e.:

Shared project - Resizetizer.NT (0.3.0) Android project - Resizetizer.NT (0.3.0) iOS project - Resizetizer.NT (0.3.0)

It resolves the building part without failing, but there are no images in the apps in my case.

GerbenIndeqa commented 2 years ago

Same here. The proposed solution of @GalaxiaGuy to set ResizetizerNT to 0.3.0 in the shared project and 0.2.0 in the platform projects works. Unfortunately the build later runs into another issue, so couldn't test yet if the images do show up.

eddieyanez commented 2 years ago

You're both correct, the images are missing although the build does succeed. This can also be observed when building locally.

GerbenIndeqa commented 2 years ago

@eddieyanez Did not bother to check local build, but you are completely right. Pictures are missing in local build as well.

vsafonkin commented 2 years ago

@GalaxiaGuy, @RonnyRos, @eddieyanez, @GerbenIndeqa, @NunoBem, could you please try this command as workaround before build step to clear inactive memory?

sudo purge

Looks like it works for sample repro above: https://github.com/vsafonkin-test-organization/xf-bugs/runs/4237156103?check_suite_focus=true

GalaxiaGuy commented 2 years ago

@vsafonkin That built for me (on my actual, larger project). I'll need more time to check the output build actually has images.

eddieyanez commented 2 years ago

@GalaxiaGuy, @RonnyRos, @eddieyanez, @GerbenIndeqa, @NunoBem, could you please try this command as workaround before build step to clear inactive memory?
sudo purge
Looks like it works for sample repro above: https://github.com/vsafonkin-test-organization/xf-bugs/runs/4237156103?check_suite_focus=true

@vsafonkin, I’m using App Center. I think I can add custom scripts if that’s what you mean. If so, which one should I use? https://docs.microsoft.com/en-us/appcenter/build/custom/scripts/

vsafonkin commented 2 years ago

@eddieyanez, you can try pre-build script but I'm not sure if this will work for App Center, because looks like the memory usage increases after install dependencies.

@GalaxiaGuy, ok, thanks.

eddieyanez commented 2 years ago

@vsafonkin sure, I’ll give it a try later tonight when I have my laptop with me.

GalaxiaGuy commented 2 years ago

@vsafonkin I can confirm this app builds and does have working resizetized images.

(Also relevant to @eddieyanez) If I do the purge before nuget restore, the app fails to build, it has to be between nuget restore and the build.

eddieyanez commented 2 years ago

@vsafonkin @GalaxiaGuy I can also confirm the pre-build script means that I can use version 3.0.0 on Shared, iOS and Android projects, and the project will build successfully with working images. I've tested via App Center on an actual iPhone and an actual Android phone.

The downside is that the Android build jumped from 08:12 to 13:05, and the iPhone was worse from 07:25 to 20:11.

vsafonkin commented 2 years ago

@GalaxiaGuy, could you try to downgrade nuget version in your project? Current default version is 5.9.0 in the runner. You can do it with setup-nuget action for GitHub actions:

- uses: nuget/setup-nuget@v1
  with:
    nuget-version: 5.6.x

or nuget installer task for Azure DevOps:

- task: NuGetToolInstaller@1
  displayName: 'Install nuget 5.6.0'
  inputs:
    versionSpec: 5.6.x

Looks like nuget 5.6 works fine and the reason of issue in 5.9 version exactly.

vsafonkin commented 2 years ago

@eddieyanez, unfortunately, I don't know the way to downgrade nuget version for App Center builds.

RonnyRos commented 2 years ago

Tried changing to 0.3.0 on android, ios and shared, Added sudo purge in the end of the prebuild bash file on appcenter, same error unfortunatly.

GalaxiaGuy commented 2 years ago

@vsafonkin Using nuget 5.6.x (with no purge) built correctly.

vsafonkin commented 2 years ago

@RonnyRos, I think it can be argued that the issue is in the new version of nuget, maybe it's better to ask App Center support how to downgrade nuget tool version, sorry.

RonnyRos commented 2 years ago

Okay, thank you so far.

RonnyRos commented 2 years ago

But shouldnt Resizetizer work with the newest version of Nuget? Why the downgrades?

actions / runner-images