search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.22k stars 4.72k forks source link

CoreCLR test infra: why does managed test build job build native library components? #91561

Open trylek opened 1 year ago

trylek commented 1 year ago

Example run (any PR or outerloop run really):

https://dev.azure.com/dnceng-public/public/_build/results?buildId=394469&view=logs&j=c92f2c34-43c3-5f9c-356c-2e863ce9eb4e&t=628e73c6-5320-5d5e-a820-3a873876a95d

While babysitting a private PR run I have noticed that the build step

Build coreclr/libs components needed by test build

within the build job

coreclr Common Pri0 Test Build AnyOS AnyCPU checked

amongst others builds native library artifacts, e.g. here:

  [ 57%] Building C object System.Globalization.Native/CMakeFiles/System.Globalization.Native.dir/__w/1/s/artifacts/obj/_version.c.o
  [ 58%] Building C object System.Security.Cryptography.Native/CMakeFiles/objlib.dir/openssl.c.o
  [ 58%] Building C object System.Security.Cryptography.Native/CMakeFiles/objlib.dir/pal_asn1.c.o
  [ 59%] Linking C shared library libSystem.Globalization.Native.so
  [ 60%] Building C object System.Security.Cryptography.Native/CMakeFiles/objlib.dir/pal_bignum.c.o
  Stripping symbols from libSystem.Globalization.Native.so into libSystem.Globalization.Native.so.dbg
  Verifying System.Globalization.Native.so dependencies
  [ 60%] Built target System.Globalization.Native
  [ 60%] Building C object System.IO.Compression.Native/CMakeFiles/System.IO.Compression.Native.dir/pal_zlib.c.o
  [ 61%] Building C object System.IO.Compression.Native/CMakeFiles/System.IO.Compression.Native.dir/__w/1/s/src/native/external/brotli/common/constants.c.o
  [ 61%] Building C object System.Security.Cryptography.Native/CMakeFiles/objlib.dir/pal_bio.c.o
  [ 61%] Building C object System.IO.Compression.Native/CMakeFiles/System.IO.Compression.Native.dir/__w/1/s/src/native/external/brotli/common/context.c.o
  [ 62%] Building C object System.IO.Compression.Native/CMakeFiles/System.IO.Compression.Native.dir/__w/1/s/src/native/external/brotli/common/dictionary.c.o
  [ 63%] Building C object System.Security.Cryptography.Native/CMakeFiles/objlib.dir/pal_dsa.c.o
  [ 63%] Building C object System.Security.Cryptography.Native/CMakeFiles/objlib.dir/pal_ecdsa.c.o
  [ 63%] Building C object System.IO.Compression.Native/CMakeFiles/System.IO.Compression.Native.dir/__w/1/s/src/native/external/brotli/common/platform.c.o
  [ 64%] Building C object System.Security.Cryptography.Native/CMakeFiles/objlib.dir/pal_ecc_import_export.c.o

I believe we should fix the yml script to only build managed library components in this step. I don't see why we should be building native library bits in a job that's supposed to be platform-independent; narrowing the library build can also probably save us a minute or so twice in each PR run (where we run the job twice, once for CoreCLR, once for Mono).

Thanks

Tomas

/cc @dotnet/runtime-infrastructure @jkoritzinsky @stephentoub

ghost commented 1 year ago

Tagging subscribers to this area: @hoyosjs See info in area-owners.md if you want to be subscribed.

Issue Details
Example run (any PR or outerloop run really): https://dev.azure.com/dnceng-public/public/_build/results?buildId=394469&view=logs&j=c92f2c34-43c3-5f9c-356c-2e863ce9eb4e&t=628e73c6-5320-5d5e-a820-3a873876a95d While babysitting a private PR run I have noticed that the build step **Build coreclr/libs components needed by test build** within the build job **coreclr Common Pri0 Test Build AnyOS AnyCPU checked** amongst others builds native library artifacts, e.g. here: 
  [ 57%] Building C object System.Globalization.Native/CMakeFiles/System.Globalization.Native.dir/__w/1/s/artifacts/obj/_version.c.o
  [ 58%] Building C object System.Security.Cryptography.Native/CMakeFiles/objlib.dir/openssl.c.o
  [ 58%] Building C object System.Security.Cryptography.Native/CMakeFiles/objlib.dir/pal_asn1.c.o
  [ 59%] Linking C shared library libSystem.Globalization.Native.so
  [ 60%] Building C object System.Security.Cryptography.Native/CMakeFiles/objlib.dir/pal_bignum.c.o
  Stripping symbols from libSystem.Globalization.Native.so into libSystem.Globalization.Native.so.dbg
  Verifying System.Globalization.Native.so dependencies
  [ 60%] Built target System.Globalization.Native
  [ 60%] Building C object System.IO.Compression.Native/CMakeFiles/System.IO.Compression.Native.dir/pal_zlib.c.o
  [ 61%] Building C object System.IO.Compression.Native/CMakeFiles/System.IO.Compression.Native.dir/__w/1/s/src/native/external/brotli/common/constants.c.o
  [ 61%] Building C object System.Security.Cryptography.Native/CMakeFiles/objlib.dir/pal_bio.c.o
  [ 61%] Building C object System.IO.Compression.Native/CMakeFiles/System.IO.Compression.Native.dir/__w/1/s/src/native/external/brotli/common/context.c.o
  [ 62%] Building C object System.IO.Compression.Native/CMakeFiles/System.IO.Compression.Native.dir/__w/1/s/src/native/external/brotli/common/dictionary.c.o
  [ 63%] Building C object System.Security.Cryptography.Native/CMakeFiles/objlib.dir/pal_dsa.c.o
  [ 63%] Building C object System.Security.Cryptography.Native/CMakeFiles/objlib.dir/pal_ecdsa.c.o
  [ 63%] Building C object System.IO.Compression.Native/CMakeFiles/System.IO.Compression.Native.dir/__w/1/s/src/native/external/brotli/common/platform.c.o
  [ 64%] Building C object System.Security.Cryptography.Native/CMakeFiles/objlib.dir/pal_ecc_import_export.c.o
I believe we should fix the yml script to only build managed library components in this step. I don't see why we should be building native library bits in a job that's supposed to be platform-independent; narrowing the library build can also probably save us a minute or so twice in each PR run (where we run the job twice, once for CoreCLR, once for Mono). Thanks Tomas /cc @dotnet/runtime-infrastructure @jkoritzinsky @stephentoub
Author: trylek
Assignees: -
Labels: `area-Infrastructure-coreclr`
Milestone: 9.0.0
ViktorHofer commented 1 year ago

I believe we should fix the yml script to only build managed library components in this step. I don't see why we should be building native library bits in a job that's supposed to be platform-independent; narrowing the library build can also probably save us a minute or so twice in each PR run (where we run the job twice, once for CoreCLR, once for Mono).

That's for System.IO.Ports which requires the native assets to be built. If you need both the shared framework and the out-of-band libraries, you want to build the libs.sfx+libs.oob subset and pass the /p:SkipLibrariesNativeRuntimePackages=true property in to avoid the System.IO.Ports native runtime packages to be built.