Xamarin.Android task cannot find NDK with latest Azure image(s)

[Maxgamerboy1]

Android application type

Classic Xamarin.Android (MonoAndroid12.0, etc.)

Affected platform version

Azure pipeline

Description

Making use of the Xamarin.Android task we have been able to build and deploy Xamarin.Android applications. New images were released which seems to have affected this task; the NDK path is no longer found when executing the "_ResolveMonoAndroidSdks" step resulting in a build failure.

Reviewing the latest releases for these images, I see there was work around NDK paths, symlinks, and environment variable changes. https://github.com/actions/virtual-environments/issues/5879 https://github.com/actions/virtual-environments/issues/5930

The image we're using is macOS-latest, here are the old and new logs showing the "_ResolveMonoAndroidSdks" step resolving the NDK path: Old working image (20220724.1)

New broken image (20220801.1)

Do we know if the Xamarin.Android task needs to be updated to reflect this?

Steps to Reproduce

1. Make use of the Azure Pipeline Xamarin.Android task for a Xamarin.Android application using the macOS-latest image.
2. See the task eventually fails with error XA5104: Could not locate the Android NDK. Please make sure the Android NDK is installed in the Android SDK Manager, or if using a custom NDK path, please ensure the $(AndroidNdkDirectory) MSBuild property is set to the custom path.

Did you find any workaround?

No response

Relevant log output

No response

[jonathanpeppers]

Did this start failing like last night?

I have a build yesterday that did:

Android NDK: /Users/runner/Library/Android/sdk/ndk/24.0.8215888/

https://dev.azure.com/jonathanpeppers/boots/_build/results?buildId=31&view=logs&j=a82d9e72-1de4-5bd6-c188-90296135479e&t=65080fd3-055e-56ae-0221-9a17a9cb5386&l=365

It is macOS-latest:

https://github.com/jonathanpeppers/boots/blob/7bf7e0875d8e4a19c3c51559521d7d0f1aa15f89/azure-pipelines.yml#L35

[jonathanpeppers]

Started a new one, let's see what it does: https://dev.azure.com/jonathanpeppers/boots/_build/results?buildId=33&view=results

[jonathanpeppers]

I think it also worked: https://dev.azure.com/jonathanpeppers/boots/_build/results?buildId=33&view=logs&j=a82d9e72-1de4-5bd6-c188-90296135479e&t=65080fd3-055e-56ae-0221-9a17a9cb5386&l=365

[Maxgamerboy1]

Thanks for the response! Our builds started failing this morning, ones running yesterday were still using the old image, due to the staggered rollout.

I do see your build is using MSBuild directly, whereas I'm using this badger for building my Xamarin Android project https://docs.microsoft.com/en-gb/azure/devops/pipelines/tasks/build/xamarin-android?view=azure-devops

[jonathanpeppers]

It seems like the bots should still have r23 and r24, can you add some ls commands to a build to find out if they are there?

[jonathanpeppers]

Mine is also using XamarinAndroidVersion: 13.0.0-0, which it downloaded and installed.

[GreatBarrier86]

Instead of the suggested mitigation of installing r21, is there a way to change the project to just use the latest version? I'm obviously having the same problem too.

@jonathanpeppers, there's something here i dont understand. I dont think we're specifying the required NDK version in our manifest or csproj file, so what's causing it to use r21 specifically instead of just $LatestVersion.

_ResolveMonoAndroidSdks: MonoAndroid Tools: /Library/Frameworks/Xamarin.Android.framework/Libraries/xbuild/Xamarin/Android/ Android Platform API level: 31 TargetFrameworkVersion: v12.0 Android NDK: Android SDK: /Users/runner/Library/Android/sdk/ Android SDK Build Tools: /Users/runner/Library/Android/sdk/build-tools/30.0.2/ Java SDK: /Library/Java/JavaVirtualMachines/Temurin-Hotspot-11.jdk/Contents/Home/ Application Java class: android.app.Application

"/Users/runner/work/8/s/Mobile/CPSMobile/CPSMobile.Android/CPSMobile.Android.csproj" (PackageForAndroid target) (1) -> (_BuildApkEmbed target) -> /Library/Frameworks/Mono.framework/External/xbuild/Xamarin/Android/Xamarin.Android.Common.targets(1954,3): error XA5104: Could not locate the Android NDK. Please make sure the Android NDK is installed in the Android SDK Manager, or if using a custom NDK path, please ensure the $(AndroidNdkDirectory) MSBuild property is set to the custom path. [/Users/runner/work/8/s/Mobile/CPSMobile/CPSMobile.Android/CPSMobile.Android.csproj]

[jonathanpeppers]

I found something that prevents the selection of r25: https://github.com/xamarin/xamarin-android-tools/pull/190

Versions with Android NDK r24 or lower should work, though. As long as one of those is installed.

[GreatBarrier86]

According to https://github.com/actions/virtual-environments/blob/main/images/macos/macos-11-Readme.md, the following NDKs are installed. 21.4.7075529 23.2.8568313 24.0.8215888

Wouldn't one of these be auto-selected? Or is there a configuration problem on my manfiest/csproj side? It just seems like a manual step shouldn't be necessary

Edit: and on top of that, that list doesn't include r25. I'm not seeing how this would work, or I just have a fundamental misunderstanding of the implementation. :)

[DevenCC]

Hey there, following the mitigation proposed in this thread, I've added this task prior to my Android build task and at least the NDK is now present.

    - task: CmdLine@2
      displayName: "XA5104 Workaround"
      inputs:
        script: '${ANDROID_HOME}/tools/bin/sdkmanager --install "ndk;21.4.7075529"'

This might not be an elegant permanent solution, since as @GreatBarrier86 stated, I also don't understand why 21.4.7075529 is not found automatically. However, this could help unblock those failing pipelines in the meantime at least.

[GreatBarrier86]

@DevenCC, did you have any success referencing a later version of the NDK that, according to that link, is also installed on the image? I tried doing the r24 and had no success.

[DevenCC]

@GreatBarrier86 I tried the same task with 24.0.8215888 and it worked fine as well. I am assuming here that using any of the ndk versions stating in the MacOS-latest virtual environments would work. I do not know if any other specific version would work however.

For my specific needs however, I went for version 21.4.7075529 as that is what my project was using prior to this issue.

[jtorvald]

I'm experiencing the same issue. This week we had one re-run on devops and that one magically succeeded while the previous one failed. All the builds after kept failing. The temp-fix from @DevenCC helps to get a successful build.

[jonathanpeppers]

Is someone able to share a .binlog of a failing build? And if you have a working example to compare against, that would also be helpful. Thanks!

You can build with -bl:nameyourlog.binlog, and then archive the file during your build.

[jtorvald]

@jonathanpeppers I tried to make one but I can't find where the binlog is placed. I copied the whole source and output directory but can't find it.

[mikequ-taggysoft]

Our App Center Xamarin.Forms Android builds just broke with the same error, too. We don't use any MSBuild task script or specify any specific NDK path/version in our Xamarin project. Things always built fine with just the source code in our DevOps repo.

I wonder why it's not using whatever is the latest on the build agent after this update. Is there a setting in the Xamarin.Forms project that we have to configure?

[jtorvald]

@mikequ-taggysoft look at the temporary fix from @DevenCC with task: CmdLine@2. Copy that block into your pipeline.yml before the task task: XamarinAndroid@1 That will make sure the NDK gets installed and your build will succeed.

[mikequ-taggysoft]

@jtorvald We're not using Azure Pipelines for the project. We build it on-commit or manually via App Center which is connected to the repo.

[jtorvald]

@mikequ-taggysoft sorry, I skipped over that. I guess you can do the same in AppCenter with a post-clone hook script and to add this line to it:

$ANDROID_HOME/tools/bin/sdkmanager --install "ndk;21.4.7075529"

Documentation about post-clone hook can be found here

[mikequ-taggysoft]

@jtorvald Thanks! We do use post clone scripts and adding that command to it did resolve the issue.

Is there any way to find out exactly which part of our Xamarin project depends on NDK v21 (or the Xamarin.Android framework itself does?) Is it possible to use the latest version somehow?

[GreatBarrier86]

Something else is strange here. I dont have any NDK version installed on my local build machine and i have no trouble building my app at all. If it works fine for me without the NDK, why is it necessary on the azure build machine? Is it something MacOS specific?

Also, i agree with @mikequ-taggysoft. If we can eliminate the specific version requirement, that would be ideal.

[dellis1972]

Are you building Debug or Release builds locally? only certain features of a Release build will need the NDK. Also the sdk I have includes an ndk in it. So it might be that your local install is picking that up.

[mikequ-taggysoft]

In my case, I believe my project requires NDK for sure because the release build profile uses AOT. But I can't see anywhere in the project config where it specifies a specific version of NDK.

[GreatBarrier86]

@dellis1972 , i bet that's it. i dont do release builds locally.

[jonathanpeppers]

I tried again this morning, and it seems to be working for me:

https://dev.azure.com/jonathanpeppers/boots/_build/results?buildId=34&view=logs&j=a82d9e72-1de4-5bd6-c188-90296135479e&t=65080fd3-055e-56ae-0221-9a17a9cb5386

XamarinAndroidVersion: 13.0.0-0
...
Android NDK: /Users/runner/Library/Android/sdk/ndk/24.0.8215888/

This is CI for boots, and it does basically:

dotnet tool install --global boots --version 1.1.0.712-preview2
boots --preview Xamarin.Android

Can anyone comment on the version of Xamarin.Android where you're having this problem? Or share a log? Does updating Xamarin.Android solve the issue?

[jonathanpeppers]

@jonathanpeppers I tried to make one but I can't find where the binlog is placed. I copied the whole source and output directory but can't find it.

@jtorvald sometimes on macOS, I found MSBuild doesn't put the .binlog where you expect unless you do something like:

-bl:$(System.DefaultWorkingDirectory)/msbuild.binlog

[jtorvald]

@jonathanpeppers thanks. I will give it a spin to see if I can send you a binlog

[Maxgamerboy1]

It seems like the bots should still have r23 and r24, can you add some ls commands to a build to find out if they are there?

Sorry this took a while. I ran the following command: echo ANDROID_NDK_HOME; ls $(ANDROID_NDK_HOME); echo "sdk path"; ls /Users/runner/Library/Android/sdk Which resulted in this output:

Generating script. Script contents: echo ANDROID_NDK_HOME; ls /Users/runner/Library/Android/sdk/ndk/25.0.8775105; echo "sdk path"; ls /Users/runner/Library/Android/sdk ========================== Starting Command Output =========================== /bin/bash /Users/runner/work/_temp/d01ea05a-ce5e-4d55-88c4-e784972de9da.sh ANDROID_NDK_HOME CHANGELOG.md NOTICE NOTICE.toolchain README.md build meta ndk-build ndk-gdb ndk-lldb ndk-stack ndk-which package.xml prebuilt python-packages shader-tools simpleperf source.properties sources toolchains wrap.sh sdk path build-tools cmake cmdline-tools emulator extras licenses ndk patcher platform-tools platform-tools-2 platforms tools

As we can see, it seems the NDK is available on the machine. After a bit more investigation, I was able to supply additional build arguments to the task: msbuildArguments: '/p:AndroidNdkDirectory="$(ANDROID_NDK_HOME)"'

The task then succeeds to find NDK 25.0.8775105 as specified by the build image https://github.com/actions/runner-images/blob/main/images/macos/macos-12-Readme.md#environment-variables-2

I was unable to get a binlog, as supplying this argument resolved our issue. It is still strange how specifying the NDK wasn't required before.

[jtorvald]

@jonathanpeppers I've a binlog. Can I share it somewhere semi-private? Could I drop you an email at your Gmail or create a private ticket as MS somehow that you can pick up?

[jonathanpeppers]

@jtorvald you can email at the address on my GitHub profile, thanks.

[jonathanpeppers]

Ok, this log message is a bit different than what I'm seeing!

Skipping NDK in '/Users/runner/Library/Android/sdk/ndk/24.0.8215888': version 24.0.8215888 is out of the accepted range (major version must be between 16 and 21
Skipping NDK in '/Users/runner/Library/Android/sdk/ndk/23.2.8568313': version 23.2.8568313 is out of the accepted range (major version must be between 16 and 21
Skipping NDK in '/Users/runner/Library/Android/sdk/ndk/25.0.8775105': version 25.0.8775105 is out of the accepted range (major version must be between 16 and 21

[mikequ-taggysoft]

@jonathanpeppers I use Xamarin.Forms and all packages are the latest. It still gives me that error. I sent App Center build logs to your email. This is the Xamarin build configuration in my csproj:

<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Release|AnyCPU' ">
<DebugSymbols>false</DebugSymbols>
<DebugType>portable</DebugType>
<Optimize>true</Optimize>
<OutputPath>bin\Release</OutputPath>
<ErrorReport>prompt</ErrorReport>
<WarningLevel>4</WarningLevel>
<AndroidManagedSymbols>true</AndroidManagedSymbols>
<AndroidUseSharedRuntime>false</AndroidUseSharedRuntime>
<AotAssemblies>true</AotAssemblies>
<!-- Hybrid mode is required to strip IL codes from DLLs in APKs, leaving only class/method names for reflection -->
<AndroidAotMode>Hybrid</AndroidAotMode>
<!-- These two aot arguments reduce APK size -->
<AndroidAotAdditionalArguments>no-write-symbols,nodebug</AndroidAotAdditionalArguments>
<EnableLLVM>false</EnableLLVM>
<AndroidEnableProfiledAot>false</AndroidEnableProfiledAot>
<BundleAssemblies>false</BundleAssemblies>
<EmbedAssembliesIntoApk>true</EmbedAssembliesIntoApk>
<AndroidUseAapt2>true</AndroidUseAapt2>
<AndroidCreatePackagePerAbi>false</AndroidCreatePackagePerAbi>
<AndroidSupportedAbis>arm64-v8a</AndroidSupportedAbis>
</PropertyGroup>

[Maxgamerboy1]

I was unable to get a binlog, as supplying this argument resolved our issue. It is still strange how specifying the NDK wasn't required before.

I thought this resolved our issue as it got past the resolve step, yet a different error occurred within the _BuildApkEmbed task; "error XA5105: Toolchain utility 'armv7a-linux-androideabi16-clang' for target Arm was not found. Tried in path: '/Users/runner/Library/Android/sdk/ndk/25.0.8775105/toolchains/llvm/prebuilt/darwin-x86'"

Downloading and examining the r25 NDK on my local machine I am seeing that armv7a-linux-androideabi16-clang does not exist. The closest thing that does exist lives under "[...]/toolchains/llvm/prebuilt/darwin-x86_64/bin/armv7a-linux-androideabi19-clang". Notice we're under the /bin folder and that the closest -clang is 19. 16 doesn't exist.

My guess as to where this new issue lies it two-fold:

1. https://github.com/xamarin/xamarin-android/blob/51acc438ef547c3ad3b2a8113a014c3539c231cd/src/Xamarin.Android.Build.Tasks/Utilities/NdkTools/WithClang.cs#L111 where we should append the bin folder (the darwin folder is based on the machine's architecture so I don't think this should change).
2. Updating the minimum level [ https://github.com/xamarin/xamarin-android/blob/51acc438ef547c3ad3b2a8113a014c3539c231cd/src/Xamarin.Android.Build.Tasks/Utilities/NdkTools/WithClang.cs#L41 , https://github.com/xamarin/xamarin-android/blob/0ab3db079d43d47ad1bc522e28e65204542fea60/build-tools/scripts/XABuildConfig.cs.in#L21 ]

Yet, I'm unsure how to make these changes only apply to the r25 NDK. Should I also perhaps look to open a separate issue, as this is more to do with using the r25 NDK rather than trying to find it.

[GreatBarrier86]

@jonathanpeppers, you're using a preview version of Xamarin.Android, and according to your pipeline, you have a step for installing it. How about trying without installing your version of it and just use whatever the latest default on the build image?

[jonathanpeppers]

@GreatBarrier86 when looking at @jtorvald's log:

https://github.com/xamarin/xamarin-android/issues/7233#issuecomment-1208332820

It has the same Xamarin.Android version as me, but it's acting as if it has a really old Xamarin.Android.Tools.AndroidSdk.dll in comparison to the Xamarin.Android.Common.targets file. It makes me think there is something very odd about the AzDO machine's image.

Mine must be working, because it installed the .pkg after and all the files are updated appropriately? I'm unsure, I'm trying to find a project where I can reproduce this.

Anyone hitting this, have you tried using boots to install Xamarin.Android? Does that fix it?

[jtorvald]

@jonathanpeppers

Adding this before the XamarinAndroid@1 step also solves the issue. You have a binlog and azure build log in your email.

- task: CmdLine@2
  displayName: "XA5104 Workaround"
  inputs:
    script: 'dotnet tool install --global boots --version 1.1.0.712-preview2 && boots --preview Xamarin.Android'

[jonathanpeppers]

I think I reproduced the problem here: https://github.com/jonathanpeppers/maui-profiling/pull/20

Going to save Xamarin.Android files as artifacts, to figure out what is going on...

[jonathanpeppers]

There is definitely something up with the AzDO image. I downloaded the suspect .dll and it is ancient:

After I install my own .pkg I get:

[jonathanpeppers]

I filed an issue here: https://github.com/actions/runner-images/issues/6038

Looking to find the right person to message at Microsoft internally.

[GreatBarrier86]

@jonathanpeppers, was the closure of https://github.com/actions/runner-images/issues/6038 supposed to resolve this issue? Wanted to follow up since it still appears to be an issue, even though they said the rollout of the update would take 3-4 days from early last week.

Current image version: '20220815.1' Current agent version: '2.206.1'

[jonathanpeppers]

@GreatBarrier86 if you have a newer .binlog from your build, can you check the value of $(XamarinAndroidVersion) now?

[GreatBarrier86]

With the workarounds in place or not?

[jonathanpeppers]

We're basically just checking if they updated Xamarin.Android on the image.

[GreatBarrier86]

Sorry but I’m not familiar with how to produce and publish the binlog file. Do you have info on how?

[jonathanpeppers]

Maybe just pass -v:diag to your build (to get diagnostic logs), and then search for XamarinAndroidVersion.

[GreatBarrier86]

@jonathanpeppers, found what you were looking for. A few things seem out of whack. I also attached the build log.

16.txt

[GreatBarrier86]

@Maxgamerboy1, i did notice i can build the app without the workaround NDK line now, so that might have fixed your issue.

[Maxgamerboy1]

Ah! Thanks for the nudge! I'll give it a go.

[rubeonline]

Same issue with vmImage: windows-2019:

[jonathanpeppers]

@rubeonline you will need to use VS 2022 to get the latest Xamarin.Android. I assume that image has VS 2019?