search

dotnet / source-build

A repository to track efforts to produce a source tarball of the .NET Core SDK and all its components
MIT License
265 stars 130 forks source link

Building the VMR fails on s390x with fsharp looking for crossgen for linux-s390x #4557

Closed omajid closed 1 week ago

omajid commented 4 weeks ago

Describe the Bug

I am trying to build a recent commit from the VMR (commit 1f2801003dc6785938b6116bd3c7c1717b7c060d). This fails on s390x when building fsharp

/home/omajid/dotnet/repo-projects/Directory.Build.targets(428,5): error MSB3073: The command "/home/omajid/dotnet/src/fsharp/build.sh --pack --publish --ci --configuration Release -bl /p:DotNetBuildRepo=true /p:DotNetBuildOrchestrator=true /p:RestoreConfigFile=/home/omajid/dotnet/artifacts/obj/fsharp/NuGet.config /p:SourceBuiltAssetsDir=/home/omajid/dotnet/artifacts/assets/Release/ /p:SourceBuiltAssetManifestsDir=/home/omajid/dotnet/artifacts/obj/AssetManifests/fsharp/ /p:SourceBuiltSymbolsDir=/home/omajid/dotnet/artifacts/obj/Symbols/fsharp/ /p:ArcadeBuildFromSource=true /p:DotNetBuildSourceOnly=true /p:PreviouslySourceBuiltNupkgCacheDir="/home/omajid/dotnet/prereqs/packages/previously-source-built/" /p:ReferencePackageNupkgCacheDir="/home/omajid/dotnet/prereqs/packages/reference/" /p:TreatWarningsAsErrors=false /p:GenerateResourceUsePreserializedResources=true --sourcebuild --tfm net9.0 /p:SourceBuildBootstrapTfm=net9.0 /p:SourceBuiltShippingPackagesDir=/home/omajid/dotnet/artifacts/packages/Release/Shipping/fsharp/ /p:SourceBuiltNonShippingPackagesDir=/home/omajid/dotnet/artifacts/packages/Release/NonShipping/fsharp/ > /home/omajid/dotnet/artifacts/log/Release/fsharp.log 2>&1" exited with code 1. [/home/omajid/dotnet/repo-projects/fsharp.proj]
  ##vso[task.setvariable variable=Artifacts;isSecret=false;isOutput=true]/home/omajid/dotnet/src/fsharp/artifacts
  ##vso[task.setvariable variable=Artifacts.Toolset;isSecret=false;isOutput=true]/home/omajid/dotnet/src/fsharp/artifacts/toolset
  ##vso[task.setvariable variable=Artifacts.Log;isSecret=false;isOutput=true]/home/omajid/dotnet/src/fsharp/artifacts/log/Release
  ##vso[task.setvariable variable=Temp;isSecret=false;isOutput=true]/home/omajid/dotnet/src/fsharp/artifacts/tmp/Release
  ##vso[task.setvariable variable=TMP;isSecret=false;isOutput=true]/home/omajid/dotnet/src/fsharp/artifacts/tmp/Release
  /home/omajid/dotnet/src/fsharp//FSharp.sln:
  publish /home/omajid/dotnet/src/fsharp//proto.proj /bl:"/home/omajid/dotnet/src/fsharp/artifacts/log/Release/Build.binlog"+.proto.binlog /p:Configuration=Proto /p:ArcadeBuildFromSource=true /p:DotNetBuildRepo=true /p:DotNetBuildOrchestrator=true /p:RestoreConfigFile=/home/omajid/dotnet/artifacts/obj/fsharp/NuGet.config /p:SourceBuiltAssetsDir=/home/omajid/dotnet/artifacts/assets/Release/ /p:SourceBuiltAssetManifestsDir=/home/omajid/dotnet/artifacts/obj/AssetManifests/fsharp/ /p:SourceBuiltSymbolsDir=/home/omajid/dotnet/artifacts/obj/Symbols/fsharp/ /p:ArcadeBuildFromSource=true /p:DotNetBuildSourceOnly=true /p:PreviouslySourceBuiltNupkgCacheDir=/home/omajid/dotnet/prereqs/packages/previously-source-built/ /p:ReferencePackageNupkgCacheDir=/home/omajid/dotnet/prereqs/packages/reference/ /p:TreatWarningsAsErrors=false /p:GenerateResourceUsePreserializedResources=true /p:SourceBuildBootstrapTfm=net9.0 /p:SourceBuiltShippingPackagesDir=/home/omajid/dotnet/artifacts/packages/Release/Shipping/fsharp/ /p:SourceBuiltNonShippingPackagesDir=/home/omajid/dotnet/artifacts/packages/Release/NonShipping/fsharp/

  ##vso[task.setvariable variable=NUGET_PLUGIN_HANDSHAKE_TIMEOUT_IN_SECONDS;isSecret=false;isOutput=true]20
  ##vso[task.setvariable variable=NUGET_PLUGIN_REQUEST_TIMEOUT_IN_SECONDS;isSecret=false;isOutput=true]20

    /home/omajid/dotnet/src/fsharp//FSharp.sln:
    publish /home/omajid/dotnet/src/fsharp//proto.proj /restore /bl:"/home/omajid/dotnet/src/fsharp/artifacts/log/Debug/Build.binlog"+.proto.binlog /p:Configuration=Proto /p:ArcadeBuildFromSource=false /p:SourceBuildUseMonoRuntime=

      Determining projects to restore...
  /home/omajid/dotnet/src/fsharp/buildtools/fslex/fslex.fsproj : error NU1100: Unable to resolve 'Microsoft.NETCore.App.Crossgen2.linux-s390x (= 9.0.0-rc.1.24410.5)' for 'net9.0'. PackageSourceMapping is enabled, the following source(s) were not considered: /home/omajid/dotnet/.dotnet/sdk/9.0.100-rc.1.24413.1/FSharp/library-packs, prebuilt, previously-source-built, reference-packages, source-built-transport-arcade. [/home/omajid/dotnet/artifacts/source-built-sdks/Microsoft.DotNet.Arcade.Sdk/tools/Build.proj]
  ##vso[task.logissue type=error;sourcepath=/home/omajid/dotnet/src/fsharp/buildtools/fslex/fslex.fsproj;linenumber=0;columnnumber=0;code=NU1100;]Unable to resolve 'Microsoft.NETCore.App.Crossgen2.linux-s390x (= 9.0.0-rc.1.24410.5)' for 'net9.0'. PackageSourceMapping is enabled, the following source(s) were not considered: /home/omajid/dotnet/.dotnet/sdk/9.0.100-rc.1.24413.1/FSharp/library-packs, prebuilt, previously-source-built, reference-packages, source-built-transport-arcade.
      Restored /home/omajid/dotnet/src/fsharp/src/FSharp.Core/FSharp.Core.fsproj (in 504 ms).
      Failed to restore /home/omajid/dotnet/src/fsharp/buildtools/fslex/fslex.fsproj (in 507 ms).

    real    0m3.425s
    user    0m2.882s
    sys 0m0.130s
  /home/omajid/dotnet/src/fsharp/eng/SourceBuild.props(42,5): error MSB3073: The command "./build.sh --bootstrap --skipBuild -bl --tfm net9.0 /p:SourceBuildUseMonoRuntime=" exited with code -1. [/home/omajid/dotnet/artifacts/source-built-sdks/Microsoft.DotNet.Arcade.Sdk/tools/Build.proj]
  ##vso[task.logissue type=error;sourcepath=/home/omajid/dotnet/src/fsharp/eng/SourceBuild.props;linenumber=42;columnnumber=5;code=MSB3073;]The command "./build.sh --bootstrap --skipBuild -bl --tfm net9.0 /p:SourceBuildUseMonoRuntime=" exited with code -1.

  Build FAILED.

  /home/omajid/dotnet/src/fsharp/buildtools/fslex/fslex.fsproj : error NU1100: Unable to resolve 'Microsoft.NETCore.App.Crossgen2.linux-s390x (= 9.0.0-rc.1.24410.5)' for 'net9.0'. PackageSourceMapping is enabled, the following source(s) were not considered: /home/omajid/dotnet/.dotnet/sdk/9.0.100-rc.1.24413.1/FSharp/library-packs, prebuilt, previously-source-built, reference-packages, source-built-transport-arcade. [/home/omajid/dotnet/artifacts/source-built-sdks/Microsoft.DotNet.Arcade.Sdk/tools/Build.proj]
  /home/omajid/dotnet/src/fsharp/eng/SourceBuild.props(42,5): error MSB3073: The command "./build.sh --bootstrap --skipBuild -bl --tfm net9.0 /p:SourceBuildUseMonoRuntime=" exited with code -1. [/home/omajid/dotnet/artifacts/source-built-sdks/Microsoft.DotNet.Arcade.Sdk/tools/Build.proj]
      0 Warning(s)
      2 Error(s)

Steps to Reproduce

First, produce an SDK (I had dotnet-sdk-9.0.100-rc.1.24413.1-linux-s390x.tar.gz) and a previously-source-built-artifacts (I had Private.SourceBuilt.Artifacts.9.0.100-rc.1.24413.1.linux-s390x.tar.gz) by cross-building the VMR on an x64 machine using something like this:

$ git clone https://github.com/dotnet/dotnet 
$ cd dotnet
$ export CROSS_ARCH=s390x
$ export CROSSCOMPILE=1
$ export ROOTFS_DIR=/   # or the appropriate value
$ ./prep-source-build.sh
$ ./build.sh --source-only --online --use-mono-runtime /p:TargetArchitecture=s390x /p:PortableBuild=true

Then use the just-built SDK and previously-source-built-artifacts on an s390x machine to build the VMR natively:

#!/bin/bash

set -ueo pipefail
set -x

if [ ! -d dotnet ]; then
    git clone https://github.com/dotnet/dotnet
fi
cd dotnet
git clean -xdf
git checkout -- .
git checkout main
git clean -xdf
git checkout -- .

curl -sSLO http://example.com/dotnet-sdk-9.0.100-rc.1.24413.1-linux-s390x.tar.gz
mkdir -p .dotnet
pushd .dotnet
tar xf ../dotnet-sdk*.tar.gz
popd
sdk_version=$(ls -1 .dotnet/sdk/)
sed -i -E 's/"version": .*$/"version": "'$sdk_version'",/' global.json
sed -i -E 's/"dotnet": .*$/"dotnet": "'$sdk_version'"/' global.json

curl -sSLO http://example.com/Private.SourceBuilt.Artifacts.9.0.100-rc.1.24413.1.linux-s390x.tar.gz
pushd prereqs/packages/archive/
mv ../../../Private.SourceBuilt.Artifacts.*.tar.gz .
popd

./prep-source-build.sh
./build.sh --source-only

Other Information

This was working as of Preview 7. It seems to be a regression that landed after that time. It probably affects other mono-using-platforms (eg, ppc64le) as well.

tmds commented 3 weeks ago

I ran into this issue before and had fixed it with https://github.com/dotnet/fsharp/pull/17300.

I think that worked well (preview7 includes that change). Something seems to have regressed after that.

@mthalman @ViktorHofer may be you have an idea?

ViktorHofer commented 3 weeks ago

No idea, unfortunately. I was out for the last few weeks. @omajid do you have a last known good commit so that we can narrow down the search?

tmds commented 3 weeks ago

@omajid our CI jobs that build vmr main don't show the issue. Can you still reproduce it?

omajid commented 3 weeks ago

I can still reproduce this with the VMR using the steps above:

$ git rev-parse HEAD
5215fac03da3f4f9ac067e436485dc2401041fbd
<steps from reproducer>

  ##vso[task.setvariable variable=NUGET_PLUGIN_HANDSHAKE_TIMEOUT_IN_SECONDS;isSecret=false;isOutput=true]20                                                                                                         
  ##vso[task.setvariable variable=NUGET_PLUGIN_REQUEST_TIMEOUT_IN_SECONDS;isSecret=false;isOutput=true]20                                                                                                           

    /home/omajid/dotnet/src/fsharp//FSharp.sln:                                                                                                                                                                     
    publish /home/omajid/dotnet/src/fsharp//proto.proj /restore /bl:"/home/omajid/dotnet/src/fsharp/artifacts/log/Debug/Build.binlog"+.proto.binlog /p:Configuration=Proto /p:ArcadeBuildFromSource=false /p:SourceB
uildUseMonoRuntime=false                                                                                                                                                                                            

      Determining projects to restore...                                                                                                                                                                            
  /home/omajid/dotnet/src/fsharp/buildtools/fslex/fslex.fsproj : error NU1100: Unable to resolve 'Microsoft.NETCore.App.Crossgen2.linux-s390x (= 9.0.0-rc.1.24414.5)' for 'net9.0'. PackageSourceMapping is enabled,
 the following source(s) were not considered: /home/omajid/dotnet/.dotnet/sdk/9.0.100-rc.1.24422.1/FSharp/library-packs, prebuilt, previously-source-built, reference-packages, source-built-transport-arcade. [/hom
e/omajid/dotnet/artifacts/source-built-sdks/Microsoft.DotNet.Arcade.Sdk/tools/Build.proj]                                                                                                                           
  ##vso[task.logissue type=error;sourcepath=/home/omajid/dotnet/src/fsharp/buildtools/fslex/fslex.fsproj;linenumber=0;columnnumber=0;code=NU1100;]Unable to resolve 'Microsoft.NETCore.App.Crossgen2.linux-s390x (= 
9.0.0-rc.1.24414.5)' for 'net9.0'. PackageSourceMapping is enabled, the following source(s) were not considered: /home/omajid/dotnet/.dotnet/sdk/9.0.100-rc.1.24422.1/FSharp/library-packs, prebuilt, previously-sou
rce-built, reference-packages, source-built-transport-arcade.                                                                                                                                                       
      Restored /home/omajid/dotnet/src/fsharp/src/FSharp.Core/FSharp.Core.fsproj (in 625 ms).                                                                                                                       
      Failed to restore /home/omajid/dotnet/src/fsharp/buildtools/fslex/fslex.fsproj (in 625 ms).                                                                                                                   

    real        0m3.646s                                                                                                                                                                                            
    user        0m3.788s                                                                                                                                                                                            
    sys 0m0.183s                                                                                                                                                                                                    
  /home/omajid/dotnet/src/fsharp/eng/SourceBuild.props(43,5): error MSB3073: The command "./build.sh --bootstrap --skipBuild -bl --tfm net9.0 /p:SourceBuildUseMonoRuntime=false" exited with code -1. [/home/omajid
/dotnet/artifacts/source-built-sdks/Microsoft.DotNet.Arcade.Sdk/tools/Build.proj]                                                                                                                                   
  ##vso[task.logissue type=error;sourcepath=/home/omajid/dotnet/src/fsharp/eng/SourceBuild.props;linenumber=43;columnnumber=5;code=MSB3073;]The command "./build.sh --bootstrap --skipBuild -bl --tfm net9.0 /p:Sour
ceBuildUseMonoRuntime=false" exited with code -1.                                                                                                                                                                   

  Build FAILED.                                                                                                                                                                                                     

  /home/omajid/dotnet/src/fsharp/buildtools/fslex/fslex.fsproj : error NU1100: Unable to resolve 'Microsoft.NETCore.App.Crossgen2.linux-s390x (= 9.0.0-rc.1.24414.5)' for 'net9.0'. PackageSourceMapping is enabled,
 the following source(s) were not considered: /home/omajid/dotnet/.dotnet/sdk/9.0.100-rc.1.24422.1/FSharp/library-packs, prebuilt, previously-source-built, reference-packages, source-built-transport-arcade. [/hom
e/omajid/dotnet/artifacts/source-built-sdks/Microsoft.DotNet.Arcade.Sdk/tools/Build.proj]                                                                                                                           
  /home/omajid/dotnet/src/fsharp/eng/SourceBuild.props(43,5): error MSB3073: The command "./build.sh --bootstrap --skipBuild -bl --tfm net9.0 /p:SourceBuildUseMonoRuntime=false" exited with code -1. [/home/omajid
/dotnet/artifacts/source-built-sdks/Microsoft.DotNet.Arcade.Sdk/tools/Build.proj]                                                                                                                                   
      0 Warning(s)                                                                                                                                                                                                  
      2 Error(s)                                                                                                                                                                                                   

  Time Elapsed 00:00:04.34                                                                                                                                                                                          
  Build failed with exit code 1. Check errors above.                                                                                                                                                                
  ##vso[task.logissue type=error](NETCORE_ENGINEERING_TELEMETRY=Build) Error building solution (exit code '1').                                                                                                     

  real  0m6.753s                                                                                                                                                                                                    
  user  0m7.001s                                                                                                                                                                                                    
  sys   0m0.349s                                                                                                                                                                                                    

  'fsharp' failed during build.
tmds commented 3 weeks ago

/p:SourceBuildUseMonoRuntime=false

SourceBuildUseMonoRuntime is incorrect for these architectures.

In our CI, we pass --use-mono-runtime also on the native s390x/ppc64le builds, which is why they don't run into the issue. You can do this as a workaround.