Building net*-android apps always restore Microsoft.NETCore.App.Host.linux

jonathanpeppers commented 2 years ago

Describe the bug

> dotnet new android
> dotnet restore

Appears to always download:

This is related to: https://github.com/dotnet/sdk/issues/19891

Since these are on NuGet.org, everything works.

However, we started noticing this with net7.0-android apps:

Since these are not on NuGet, you get errors like:

Errors
    C:\src\foo\foo.csproj error NU1102: Unable to find package Microsoft.NETCore.App.Host.linux-arm with version (= 7.0.0-preview.5.22271.4)
  - Found 88 version(s) in nuget.org [ Nearest version: 7.0.0-preview.4.22229.4 ]
  - Found 0 version(s) in Microsoft Visual Studio Offline Packages [C:\src\foo\foo.csproj]
    C:\src\foo\foo.csproj error NU1102: Unable to find package Microsoft.NETCore.App.Host.linux-arm64 with version (= 7.0.0-preview.5.22271.4)
  - Found 88 version(s) in nuget.org [ Nearest version: 7.0.0-preview.4.22229.4 ]
  - Found 0 version(s) in Microsoft Visual Studio Offline Packages [C:\src\foo\foo.csproj]
    C:\src\foo\foo.csproj error NU1102: Unable to find package Microsoft.NETCore.App.Host.linux-x64 with version (= 7.0.0-preview.5.22271.4)
  - Found 88 version(s) in nuget.org [ Nearest version: 7.0.0-preview.4.22229.4 ]
  - Found 0 version(s) in Microsoft Visual Studio Offline Packages [C:\src\foo\foo.csproj]

And so anyone dogfooding .NET 7 would have to add a nuget.config:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <add key="dotnet7" value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet7/nuget/v3/index.json" />
  </packageSources>
</configuration>

To Reproduce

Install the .NET 7 android workload
dotnet new android
dotnet restore

Exceptions (if any)

Binlog: msbuild.zip

Further technical details

.NET SDK:
 Version:   7.0.100-preview.5.22273.1
 Commit:    a3acdcf34e

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.19044
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\src\xamarin-android\bin\Release\dotnet\sdk\7.0.100-preview.5.22273.1\

global.json file:
  Not found

Host:
  Version:      7.0.0-preview.5.22271.4
  Architecture: x64
  Commit:       621cd59436

.NET SDKs installed:
  7.0.100-preview.5.22273.1 [C:\src\xamarin-android\bin\Release\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 7.0.0-preview.5.22272.3 [C:\src\xamarin-android\bin\Release\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 7.0.0-preview.5.22271.4 [C:\src\xamarin-android\bin\Release\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 7.0.0-preview.5.22269.1 [C:\src\xamarin-android\bin\Release\dotnet\shared\Microsoft.WindowsDesktop.App]

dotnet-issue-labeler[bot] commented 2 years ago

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

marcpopMSFT commented 2 years ago

@dsplaisted I believe you were looking into this before with Jonathan.

jonathanpeppers commented 2 years ago

We had a new version of this appear in: https://github.com/xamarin/xamarin-android/pull/7071

(Restore target) -> 
    App1.csproj : error NU1101: Unable to find package Microsoft.NETCore.App.Host.linux-bionic-arm. No packages exist with this id in source(s): dotnet-eng, dotnet-public, xamarin.android util
    App1.csproj : error NU1101: Unable to find package Microsoft.NETCore.App.Host.linux-bionic-arm64. No packages exist with this id in source(s): dotnet-eng, dotnet-public, xamarin.android util
    App1.csproj : error NU1101: Unable to find package Microsoft.NETCore.App.Host.linux-bionic-x86. No packages exist with this id in source(s): dotnet-eng, dotnet-public, xamarin.android util
    App1.csproj : error NU1101: Unable to find package Microsoft.NETCore.App.Host.linux-bionic-x64. No packages exist with this id in source(s): dotnet-eng, dotnet-public, xamarin.android util

marcpopMSFT commented 2 years ago

@jonathanpeppers does setting DisableTransitiveFrameworkReferenceDownloads per https://github.com/dotnet/sdk/pull/25358 work around this issue now or is it separate?

LoopedBard3 commented 2 years ago

Trying /p:DisableTransitiveFrameworkReferenceDownloads=true out locally, does not seem to fix the problem when building Maui targeting android-arm64.

jonathanpeppers commented 2 years ago

I think there might be a couple different things going on here:

linux-bionic was added to the RID graph, somehow android inherits from it. This might be correct, because "bionic" is some subset of libc, if I understand correctly?
linux-bionic-arm and linux-bionic-x86 do not actually exist, only the 64-bit versions exist

@directhex was going to look at the RIDs and/or the packages. Did we come up with something there? Maybe we should delete the 32-bit ones that shouldn't exist?

directhex commented 2 years ago

bionic is the Android C library, equivalent to glibc on Linux. linux-bionic is basically a RID for "Linux, but using Bionic libc instead of GNU libc". It was created for an internal customer, who only wanted 64-bit versions - but I was told we should have the 32-bit versions in the RID graph too so android-x86 and android-x64 would have matching dependency graphs, not android-x86->linux-x86.

The bug here is the needless restore. The existence or lack thereof of a runtime pack for linux-bionic-x86 should not be affecting Android apps. We need to figure out why that dependency is being considered as needed by the SDK.

LoopedBard3 commented 2 years ago

I am not super familiar with how the SDK uses the RID Graph, although it looks like linux-bionic* are included as imports for the respective RID's based on the RuntimeIdentifierGraph.json file used when building the Maui app:

    "android": {
      "#import": [
        "linux-bionic"
      ]
    },
    "android-arm": {
      "#import": [
        "android",
        "linux-bionic-arm"
      ]
    },
    "android-arm64": {
      "#import": [
        "android",
        "linux-bionic-arm64"
      ]
    },
    "android-x64": {
      "#import": [
        "android",
        "linux-bionic-x64"
      ]
    },
    "android-x86": {
      "#import": [
        "android",
        "linux-bionic-x86"
      ]
    },

Are these imports not treated as download/install requirements? Or is there a different place I should be looking to track down this bug? Looking at https://github.com/dotnet/runtime/commit/b5f9c224e785243944f9aabe56bfe3727649d4ea#diff-4cae40f80b790c66e01955edbd41e3ea51cec84088d1cbf6b461a5be8daee777R279-R307, it looks like android was previously importing linux, would changing it back to that fix the issue? Thanks!

directhex commented 2 years ago

The RID graph should only be traversed like that when it comes to calculating NuGet fallbacks - i.e. "if there's no .so for android in this package, try the .so for linux-bionic, followed by the .so for Linux"

steveisok commented 2 years ago

@dsplaisted pointed to https://github.com/dotnet/sdk/pull/22658 where we can exclude certain rids from the task that tries to download hosting packs.

marcpopMSFT commented 2 years ago

Change went into installer for 6.0.400 already and flowing to main here: https://github.com/dotnet/installer/pull/14288

stulda commented 2 years ago

I have this issue newly on my m1 macbook after sdk 6.0.400 update.

.NET SDK (reflecting any global.json): Version: 6.0.400 Commit: 7771abd614

Runtime Environment: OS Name: Mac OS X OS Version: 12.5 OS Platform: Darwin RID: osx.12-arm64 Base Path: /usr/local/share/dotnet/sdk/6.0.400/

global.json file: Not found

Host: Version: 6.0.8 Architecture: arm64 Commit: 55fb7ef977

/Users/stulda/Temp/test/test.csproj : error NU1102: Unable to find package Microsoft.NETCore.App.Host.linux-arm with version (= 6.0.8)
/Users/stulda/Temp/test/test.csproj : error NU1102:   - Found 94 version(s) in nuget.org [ Nearest version: 7.0.0-preview.1.22076.8 ]
/Users/stulda/Temp/test/test.csproj : error NU1102:   - Found 0 version(s) in /usr/local/share/dotnet/library-packs
/Users/stulda/Temp/test/test.csproj : error NU1102: Unable to find package Microsoft.NETCore.App.Host.linux-arm64 with version (= 6.0.8)
/Users/stulda/Temp/test/test.csproj : error NU1102:   - Found 94 version(s) in nuget.org [ Nearest version: 7.0.0-preview.1.22076.8 ]
/Users/stulda/Temp/test/test.csproj : error NU1102:   - Found 0 version(s) in /usr/local/share/dotnet/library-packs
/Users/stulda/Temp/test/test.csproj : error NU1102: Unable to find package Microsoft.NETCore.App.Host.linux-x64 with version (= 6.0.8)
/Users/stulda/Temp/test/test.csproj : error NU1102:   - Found 94 version(s) in nuget.org [ Nearest version: 7.0.0-preview.1.22076.8 ]
/Users/stulda/Temp/test/test.csproj : error NU1102:   - Found 0 version(s) in /usr/local/share/dotnet/library-packs

jonathanpeppers commented 2 years ago

These are on NuGet.org: https://www.nuget.org/packages/Microsoft.NETCore.App.Host.linux-arm/6.0.8

They shipped yesterday, I think, so maybe it just works now?

stulda commented 2 years ago

No change

I also tried this:

dotnet add package Microsoft.NETCore.App.Host.linux-arm --version 6.0.8

and here is what i got:

dotnet add package Microsoft.NETCore.App.Host.linux-arm --version 6.0.8
  Determining projects to restore...
  Writing /var/folders/m9/l6lwz36j0wxbc2lxybg9htkr0000gn/T/tmp2R0UrT.tmp
info : X.509 certificate chain validation will use the fallback certificate bundle at '/usr/local/share/dotnet/sdk/6.0.400/trustedroots/codesignctl.pem'.
info : Adding PackageReference for package 'Microsoft.NETCore.App.Host.linux-arm' into project '/Users/stulda/Temp/test/test.csproj'.
info : Restoring packages for /Users/stulda/Temp/test/test.csproj...
info :   GET https://api.nuget.org/v3-flatcontainer/microsoft.netcore.app.host.linux-arm/index.json
info :   OK https://api.nuget.org/v3-flatcontainer/microsoft.netcore.app.host.linux-arm/index.json 342ms
info :   GET https://api.nuget.org/v3-flatcontainer/microsoft.netcore.app.host.linux-arm64/index.json
info :   GET https://api.nuget.org/v3-flatcontainer/microsoft.netcore.app.host.linux-x64/index.json
info :   OK https://api.nuget.org/v3-flatcontainer/microsoft.netcore.app.host.linux-arm64/index.json 264ms
info :   OK https://api.nuget.org/v3-flatcontainer/microsoft.netcore.app.host.linux-x64/index.json 358ms
error: NU1103: Unable to find a stable package Microsoft.NETCore.App.Host.linux-arm with version (>= 6.0.8)
error:   - Found 94 version(s) in nuget.org [ Nearest version: 7.0.0-preview.1.22076.8 ]
error:   - Found 0 version(s) in /usr/local/share/dotnet/library-packs
error: NU1102: Unable to find package Microsoft.NETCore.App.Host.linux-arm with version (= 6.0.8)
error:   - Found 94 version(s) in nuget.org [ Nearest version: 7.0.0-preview.1.22076.8 ]
error:   - Found 0 version(s) in /usr/local/share/dotnet/library-packs
error: NU1102: Unable to find package Microsoft.NETCore.App.Host.linux-arm64 with version (= 6.0.8)
error:   - Found 94 version(s) in nuget.org [ Nearest version: 7.0.0-preview.1.22076.8 ]
error:   - Found 0 version(s) in /usr/local/share/dotnet/library-packs
error: NU1102: Unable to find package Microsoft.NETCore.App.Host.linux-x64 with version (= 6.0.8)
error:   - Found 94 version(s) in nuget.org [ Nearest version: 7.0.0-preview.1.22076.8 ]
error:   - Found 0 version(s) in /usr/local/share/dotnet/library-packs
error: Package 'Microsoft.NETCore.App.Host.linux-arm' is incompatible with 'all' frameworks in project '/Users/stulda/Temp/test/test.csproj'.

stulda commented 2 years ago

Change went into installer for 6.0.400 already and flowing to main here: dotnet/installer#14288

This workaround is missing in 6.0.400 release: https://github.com/xamarin/xamarin-android/pull/7237/files

dotnet / sdk