Issues flowing debuginfo from corefx to final product

[omajid]

I am trying to build the source-build 3.0.0-preview3 tag (from source tarball) with extra debuginfo and the debuginfo is not showing up in the final built product. I don't know if changes are not flowing from corefx/coreclr to the final built sdk or if I am just not changing the right things.

source-build-info.txt

``` source-build: d7dd4b00dbd2b66a2bb8ee0aae3a3756c4e4b0eb . (v3.0.0-preview3) submodules: 53b80940842204f78708a538628288ff5d741a1d src/application-insights (v2.4.0-3-g53b80940) 26042347ff79a11869af8e951dd2888051c369f0 src/arcade (remotes/origin/helix-docker-sdk-notes-1-52-g26042347) 74c43bb167cf6ee60bd0ad7f11c2cac9ea109b34 src/cli (v2.1.500-preview-009335-1084-g74c43bb16) 37e3a01aa1df320a286b0a858876f9994fb6bb18 src/cli-migrate (remotes/origin/release/2.1.3xx-88-g37e3a01a) f011de8f90a23cdef7887a97d5f496570eb1808d src/clicommandlineparser (remotes/origin/mmitche-patch-1-84-gf011de8) 6e37cdfe96ac8b06a923242120169fafacd720e6 src/common (rel/1.0.1-2-g6e37cdfe9) d72abce2133c7e37c524f7933bce09a2f44e788f src/core-sdk (v3.0.100-preview3-010431) 3844df9537f1de4bc52c69e53e56fab9160d0647 src/core-setup (v3.0.0-preview3-27503-5) 817e36a41f451092efa26c1478abfcd5c69a3bcf src/coreclr (v3.0.0-preview3-27422-72) 218c8e595095f0cb8f12d64eeb36e0b2c50590c6 src/corefx (v3.0.0-preview3.19128.7) 51a53a16e8915983ce11783da45af0aa14328586 src/fsharp (Visual-Studio-2019-Preview-2-Version-16.0-83-g51a53a16e) 9e8bcb40755c4ec007b6ca89618968b7c5f6fe58 src/linker (9e8bcb4) 8629bf5014f3cbfaf3d72e9dda6f7c8f91209f81 src/linker/cecil (0.9.5-767-g8629bf5) 5775d0d6bb7690510aa58717bbfdc139906a4f98 src/msbuild (v15.9.21.664-456-g5775d0d6) e43dae94c26f0c30e9095327a3a9eac87193923d src/newtonsoft-json (9.0.1-3-ge43dae94) 7a5c6ae18ae07c22a6e3ad46f18f558a75830955 src/nuget-client (5.0.0.5754) e6fac8061686c18531e2621ccef97dd5e0687a65 src/nuget-client/submodules/Common (1.0.0-nuget-client) f1f3f0820a573b96b2faaf5b7e6be9a036e4c7aa src/nuget-client/submodules/FileSystem (1.0.0-beta3-118-gf1f3f08) f15db7b7c6f5affbea268632ef8333d2687c8031 src/nuget-client/submodules/NuGet.Build.Localization (4.3.0.4273~28) c32ec2904856f0b64f20e82329ba69680bdeca70 src/roslyn (version-2.10.0-6440-gc32ec29048) 87f00dbd00e72ca5e3e42b2e1f7b553c2e2f4f32 src/sdk (v3.0.100-preview3.19124.1) f3add9198e40e017c064188481e978d09631eba3 src/standard (v2.1.0-prerelease.19126.1) ac8f12b90238542dfd2ea596178ae24ea20a0f55 src/templating (2.1.0-preview2-162-gac8f12b9) 99484cb8867ce380fc5d433eabdbbde1e2c80565 src/toolset (v3.0.100-preview3.19154.4) df96f832f28de9798bf3a1b2d6b7933439e00ae4 src/vstest (v15.8.0-preview-20180610-02-32-gdf96f832) 8e7f5f9da867d9a809f11b2aab7540ee3999252a src/websdk (3.0.100-Preview3) 27d43b762aa6dac3a0a6ba48fe55000942d75c1c src/xliff-tasks (27d43b7) ```

AFAIK, source-build disables stripping debuginfo. So everything built locally should contain debuginfo anyway. To be extra sure, I added this patch to core-setup to not strip anything: https://gist.github.com/omajid/f3ca14bed056b4a33a2ddf057b0a49b3

Once built, I scan over the files with this script: https://gist.github.com/omajid/766aef59857de9fc1eab173119143abc

$ ./check-debug-symbols.py -v $(find -iname System.Net.Security.Native.so)
error: missing .debug_info section in /home/omajid/dotnet-3-0/dotnet-v3.0.0-preview3/src/core-sdk/artifacts/bin/redist/Release/dotnet/shared/Microsoft.NETCore.App/3.0.0-preview-27503-5/System.Net.Security.Native.so
error: missing .debug_abbrev section in /home/omajid/dotnet-3-0/dotnet-v3.0.0-preview3/src/core-sdk/artifacts/bin/redist/Release/dotnet/shared/Microsoft.NETCore.App/3.0.0-preview-27503-5/System.Net.Security.Native.so
error: missing FILE symbols in /home/omajid/dotnet-3-0/dotnet-v3.0.0-preview3/src/core-sdk/artifacts/bin/redist/Release/dotnet/shared/Microsoft.NETCore.App/3.0.0-preview-27503-5/System.Net.Security.Native.so
error: missing .debug_info section in /home/omajid/dotnet-3-0/dotnet-v3.0.0-preview3/src/core-setup/bin/obj/fedora.28-x64.Release/SFP.packages/runtime.fedora.28-x64.microsoft.netcore.app/3.0.0-preview-27503-5/runtimes/fedora.28-x64/native/System.Net.Security.Native.so
error: missing .debug_abbrev section in /home/omajid/dotnet-3-0/dotnet-v3.0.0-preview3/src/core-setup/bin/obj/fedora.28-x64.Release/SFP.packages/runtime.fedora.28-x64.microsoft.netcore.app/3.0.0-preview-27503-5/runtimes/fedora.28-x64/native/System.Net.Security.Native.so
error: missing FILE symbols in /home/omajid/dotnet-3-0/dotnet-v3.0.0-preview3/src/core-setup/bin/obj/fedora.28-x64.Release/SFP.packages/runtime.fedora.28-x64.microsoft.netcore.app/3.0.0-preview-27503-5/runtimes/fedora.28-x64/native/System.Net.Security.Native.so
error: missing .debug_info section in /home/omajid/dotnet-3-0/dotnet-v3.0.0-preview3/src/core-setup/bin/obj/fedora.28-x64.Release/sharedFrameworkPublish/shared/Microsoft.NETCore.App/3.0.0-preview-27503-5/System.Net.Security.Native.so
error: missing .debug_abbrev section in /home/omajid/dotnet-3-0/dotnet-v3.0.0-preview3/src/core-setup/bin/obj/fedora.28-x64.Release/sharedFrameworkPublish/shared/Microsoft.NETCore.App/3.0.0-preview-27503-5/System.Net.Security.Native.so
error: missing FILE symbols in /home/omajid/dotnet-3-0/dotnet-v3.0.0-preview3/src/core-setup/bin/obj/fedora.28-x64.Release/sharedFrameworkPublish/shared/Microsoft.NETCore.App/3.0.0-preview-27503-5/System.Net.Security.Native.so
error: missing .debug_info section in /home/omajid/dotnet-3-0/dotnet-v3.0.0-preview3/src/core-setup/bin/obj/fedora.28-x64.Release/sharedFx/shared/Microsoft.NETCore.App/3.0.0-preview-27503-5/System.Net.Security.Native.so
error: missing .debug_abbrev section in /home/omajid/dotnet-3-0/dotnet-v3.0.0-preview3/src/core-setup/bin/obj/fedora.28-x64.Release/sharedFx/shared/Microsoft.NETCore.App/3.0.0-preview-27503-5/System.Net.Security.Native.so
error: missing FILE symbols in /home/omajid/dotnet-3-0/dotnet-v3.0.0-preview3/src/core-setup/bin/obj/fedora.28-x64.Release/sharedFx/shared/Microsoft.NETCore.App/3.0.0-preview-27503-5/System.Net.Security.Native.so
error: missing .debug_info section in /home/omajid/dotnet-3-0/dotnet-v3.0.0-preview3/src/core-setup/bin/obj/fedora.28-x64.Release/combined-framework-host/shared/Microsoft.NETCore.App/3.0.0-preview-27503-5/System.Net.Security.Native.so
error: missing .debug_abbrev section in /home/omajid/dotnet-3-0/dotnet-v3.0.0-preview3/src/core-setup/bin/obj/fedora.28-x64.Release/combined-framework-host/shared/Microsoft.NETCore.App/3.0.0-preview-27503-5/System.Net.Security.Native.so
error: missing FILE symbols in /home/omajid/dotnet-3-0/dotnet-v3.0.0-preview3/src/core-setup/bin/obj/fedora.28-x64.Release/combined-framework-host/shared/Microsoft.NETCore.App/3.0.0-preview-27503-5/System.Net.Security.Native.so
OK:  /home/omajid/dotnet-3-0/dotnet-v3.0.0-preview3/src/corefx/artifacts/bin/testhost/netcoreapp-Linux-Release-x64/shared/Microsoft.NETCore.App/9.9.9/System.Net.Security.Native.so
OK:  /home/omajid/dotnet-3-0/dotnet-v3.0.0-preview3/src/corefx/artifacts/bin/runtime/netcoreapp-Linux-Release-x64/System.Net.Security.Native.so
OK:  /home/omajid/dotnet-3-0/dotnet-v3.0.0-preview3/src/corefx/artifacts/bin/native/netcoreapp-Linux-Release-x64/System.Net.Security.Native.so
OK:  /home/omajid/dotnet-3-0/dotnet-v3.0.0-preview3/src/corefx/artifacts/obj/native/netcoreapp-Linux-Release-x64/System.Net.Security.Native/System.Net.Security.Native.so
error: missing .debug_info section in /home/omajid/dotnet-3-0/dotnet-v3.0.0-preview3/src/toolset/artifacts/bin/redist/Release/dotnet/shared/Microsoft.NETCore.App/3.0.0-preview-27122-01/System.Net.Security.Native.so
error: missing .debug_abbrev section in /home/omajid/dotnet-3-0/dotnet-v3.0.0-preview3/src/toolset/artifacts/bin/redist/Release/dotnet/shared/Microsoft.NETCore.App/3.0.0-preview-27122-01/System.Net.Security.Native.so
error: missing FILE symbols in /home/omajid/dotnet-3-0/dotnet-v3.0.0-preview3/src/toolset/artifacts/bin/redist/Release/dotnet/shared/Microsoft.NETCore.App/3.0.0-preview-27122-01/System.Net.Security.Native.so
error: missing .debug_info section in /home/omajid/dotnet-3-0/dotnet-v3.0.0-preview3/Tools/dotnetcli3/shared/Microsoft.NETCore.App/3.0.0-preview-27122-01/System.Net.Security.Native.so
error: missing .debug_abbrev section in /home/omajid/dotnet-3-0/dotnet-v3.0.0-preview3/Tools/dotnetcli3/shared/Microsoft.NETCore.App/3.0.0-preview-27122-01/System.Net.Security.Native.so
error: missing FILE symbols in /home/omajid/dotnet-3-0/dotnet-v3.0.0-preview3/Tools/dotnetcli3/shared/Microsoft.NETCore.App/3.0.0-preview-27122-01/System.Net.Security.Native.so
error: missing .debug_info section in /home/omajid/dotnet-3-0/dotnet-v3.0.0-preview3/Tools/dotnetcli/shared/Microsoft.NETCore.App/2.1.7/System.Net.Security.Native.so
error: missing .debug_abbrev section in /home/omajid/dotnet-3-0/dotnet-v3.0.0-preview3/Tools/dotnetcli/shared/Microsoft.NETCore.App/2.1.7/System.Net.Security.Native.so
error: missing FILE symbols in /home/omajid/dotnet-3-0/dotnet-v3.0.0-preview3/Tools/dotnetcli/shared/Microsoft.NETCore.App/2.1.7/System.Net.Security.Native.so
error: missing .debug_info section in /home/omajid/dotnet-3-0/dotnet-v3.0.0-preview3/Tools/ilasm/System.Net.Security.Native.so
error: missing .debug_abbrev section in /home/omajid/dotnet-3-0/dotnet-v3.0.0-preview3/Tools/ilasm/System.Net.Security.Native.so
error: missing FILE symbols in /home/omajid/dotnet-3-0/dotnet-v3.0.0-preview3/Tools/ilasm/System.Net.Security.Native.so
error: missing .debug_info section in /home/omajid/dotnet-3-0/dotnet-v3.0.0-preview3/packages/runtime.linux-x64.microsoft.netcore.app/2.1.0/runtimes/linux-x64/native/System.Net.Security.Native.so
error: missing .debug_abbrev section in /home/omajid/dotnet-3-0/dotnet-v3.0.0-preview3/packages/runtime.linux-x64.microsoft.netcore.app/2.1.0/runtimes/linux-x64/native/System.Net.Security.Native.so
error: missing FILE symbols in /home/omajid/dotnet-3-0/dotnet-v3.0.0-preview3/packages/runtime.linux-x64.microsoft.netcore.app/2.1.0/runtimes/linux-x64/native/System.Net.Security.Native.so
error: missing .debug_info section in /home/omajid/dotnet-3-0/dotnet-v3.0.0-preview3/packages/runtime.linux-x64.microsoft.private.corefx.netcoreapp/4.6.0-preview3.19128.7/runtimes/linux-x64/native/System.Net.Security.Native.so
error: missing .debug_abbrev section in /home/omajid/dotnet-3-0/dotnet-v3.0.0-preview3/packages/runtime.linux-x64.microsoft.private.corefx.netcoreapp/4.6.0-preview3.19128.7/runtimes/linux-x64/native/System.Net.Security.Native.so
error: missing FILE symbols in /home/omajid/dotnet-3-0/dotnet-v3.0.0-preview3/packages/runtime.linux-x64.microsoft.private.corefx.netcoreapp/4.6.0-preview3.19128.7/runtimes/linux-x64/native/System.Net.Security.Native.so
error: missing .debug_info section in /home/omajid/dotnet-3-0/dotnet-v3.0.0-preview3/packages/runtime.fedora.28-x64.microsoft.netcore.app/3.0.0-preview-27503-5/runtimes/fedora.28-x64/native/System.Net.Security.Native.so
error: missing .debug_abbrev section in /home/omajid/dotnet-3-0/dotnet-v3.0.0-preview3/packages/runtime.fedora.28-x64.microsoft.netcore.app/3.0.0-preview-27503-5/runtimes/fedora.28-x64/native/System.Net.Security.Native.so
error: missing FILE symbols in /home/omajid/dotnet-3-0/dotnet-v3.0.0-preview3/packages/runtime.fedora.28-x64.microsoft.netcore.app/3.0.0-preview-27503-5/runtimes/fedora.28-x64/native/System.Net.Security.Native.so

It looks like System.Net.Security.Native.so built by corefx has the right debuginfo, but nothing else does. The version in core-setup is missing the debuginfo.

The file modifiaction times of two examples from the list above are interesting:

[omajid@localhost dotnet-3-0]$ exiftool /home/omajid/dotnet-3-0/dotnet-v3.0.0-preview3/src/core-setup/bin/obj/fedora.28-x64.Release/sharedFx/shared/Microsoft.NETCore.App/3.0.0-preview-27503-5/System.Net.Security.Native.so                
ExifTool Version Number         : 11.30
File Name                       : System.Net.Security.Native.so
Directory                       : /home/omajid/dotnet-3-0/dotnet-v3.0.0-preview3/src/core-setup/bin/obj/fedora.28-x64.Release/sharedFx/shared/Microsoft.NETCore.App/3.0.0-preview-27503-5                                                    
File Size                       : 10 kB
File Modification Date/Time     : 2019:03:01 00:49:38-05:00
File Access Date/Time           : 2019:04:11 15:03:37-04:00
File Inode Change Date/Time     : 2019:04:11 15:03:36-04:00
File Permissions                : rwxr-xr-x
File Type                       : ELF shared library
File Type Extension             : so
MIME Type                       : application/octet-stream
CPU Architecture                : 64 bit
CPU Byte Order                  : Little endian
Object File Type                : Shared object file
CPU Type                        : AMD x86-64
[omajid@localhost dotnet-3-0]$ exiftool /home/omajid/dotnet-3-0/dotnet-v3.0.0-preview3/src/corefx/artifacts/bin/runtime/netcoreapp-Linux-Release-x64/System.Net.Security.Native.so
ExifTool Version Number         : 11.30
File Name                       : System.Net.Security.Native.so
Directory                       : /home/omajid/dotnet-3-0/dotnet-v3.0.0-preview3/src/corefx/artifacts/bin/runtime/netcoreapp-Linux-Release-x64
File Size                       : 31 kB
File Modification Date/Time     : 2019:04:11 14:54:19-04:00
File Access Date/Time           : 2019:04:11 15:00:24-04:00
File Inode Change Date/Time     : 2019:04:11 14:55:38-04:00
File Permissions                : rwxr-xr-x
File Type                       : ELF shared library
File Type Extension             : so
MIME Type                       : application/octet-stream
CPU Architecture                : 64 bit
CPU Byte Order                  : Little endian
Object File Type                : Shared object file
CPU Type                        : AMD x86-64

The file built by corefx was modified today, while the copy in core-setup was modified in March?

[dagood]

The way the shared framework is put together is pretty unusual and I think it'd interfere with prebuilt detection: https://github.com/dotnet/core-setup/blob/1758b4706e9387d579bc00cbd7b7b64b07c48f1a/src/sharedFramework/sharedFramework.proj#L158. It should source only a local directory, so a clean/isolated build should be ok, but there are other NuGet things to contend with....

@omajid I wonder if it managed to pick up a package with a matching version from a NuGet cache on your machine? Can you check for this so in ~/.nuget etc. for a match?

@dseefeld @crummel

[dagood]

For now, fixing it by adding --no-cache to the restore calls is simple enough. The source-build package cache dir should also be flowed through to stop "hiding" these restores.

It would be nice to switch away from a restore call and assemble the shared framework in a more controlled way. Right now it depends on dotnet publish/dotnet restore but I'm not sure what it would take.

[omajid]

@dagood My nuget on this vm is empty:

$ find ~/.nuget/
/home/omajid/.nuget/
/home/omajid/.nuget/NuGet
/home/omajid/.nuget/NuGet/NuGet.Config

And for the sake of testing, I am modifying my build scripts to run rm -rf ~/.nuget ~/.dotnet ~/.aspnet/ before calling build.sh now. I will know in a little bit if it makes a difference.

[dagood]

There's also an HTTP cache at ~/.local/share/NuGet/v3-cache/, although I wouldn't expect NuGet to look at it for a local dir source.

[omajid]

I actually added:

rm -rf ~/.aspnet ~/.dotnet/ ~/.nuget/ ~/.local/share/NuGet ~/.templateengine
rm -rf /tmp/NuGet /tmp/NuGetScratch /tmp/.NETCore* /tmp/.NETStandard* /tmp/.dotnet /tmp/clr-debug-pipe* /tmp/Razor-Server /tmp/CoreFxPipe* /tmp/VBCSCompiler /tmp/.NETFramework*

And it didn't help :(

[dagood]

I'd expect our offline builds to fail if there's some prebuilt required here... maybe it's sneaky enough to only get one if the build's online. 😕 That list of cleanup rms looks pretty extensive.

I'll kick off a couple builds locally with my usual online and offline steps to see if this repros in Docker isolation.

(FWIW, I wouldn't expect the Core-Setup patch to be affecting this, because System.Net.Security.Native.so is built in CoreFX and they should be simply redistributed in Core-Setup.)

[dagood]

Since preview2, in online builds, the identity package is a prebuilt:

https://github.com/dotnet/source-build/blob/058f070349183187b86aecea2138f398254c626c/tools-local/prebuilt-baseline-online.xml#L241

In my tarball production build I have these in my bin/prebuilt-report/annotated-usage.xml:

  <AnnotatedUsage Id="Microsoft.Private.CoreFx.NETCoreApp" Version="4.6.0-preview3.19128.7" File="src/core-setup/bin/obj/centos.7-x64.Release/Microsoft.NETCore.App/centos.7-x64/centos.7-x64/project.assets.json" IsDirectDependency="true" Project="src/core-setup/" SourceBuildPackageIdCreator="corefx MicrosoftPrivateCoreFxNETCoreAppPackageVersion/4.6.0-dev.19128.7" />
  <AnnotatedUsage Id="Microsoft.Private.CoreFx.NETCoreApp" Version="4.6.0-preview3.19128.7" File="src/core-setup/bin/obj/centos.7-x64.Release/Microsoft.NETCore.App/project.assets.json" IsDirectDependency="true" Project="src/core-setup/" SourceBuildPackageIdCreator="corefx MicrosoftPrivateCoreFxNETCoreAppPackageVersion/4.6.0-dev.19128.7" />
...
  <AnnotatedUsage Id="runtime.linux-x64.Microsoft.Private.CoreFx.NETCoreApp" Version="4.6.0-preview3.19128.7" File="src/core-setup/bin/obj/centos.7-x64.Release/Microsoft.NETCore.App/centos.7-x64/centos.7-x64/project.assets.json" Rid="linux-x64" Project="src/core-setup/" />

The PVP sets the version, but since it matches official I guess it's up to resolution:

<MicrosoftPrivateCoreFxNETCoreAppPackageVersion>4.6.0-dev.19128.7</MicrosoftPrivateCoreFxNETCoreAppPackageVersion>

Both project.assets.json files for Microsoft.Private.CoreFx.NETCoreApp list the source-build dir as the first source, maybe it's not priority order? (Anymore?)

      "sources": {
        "/work/bin/obj/x64/Release/blob-feed/packages/": {},
        "https://api.nuget.org/v3/index.json": {},
        "https://dotnet.myget.org/F/dotnet-buildtools/api/v3/index.json": {},
        "https://dotnet.myget.org/F/dotnet-core/api/v3/index.json": {},
        "https://dotnet.myget.org/F/dotnet-corefxlab/api/v3/index.json": {},
        "https://dotnetfeed.blob.core.windows.net/dotnet-core/index.json": {},
        "https://dotnetfeed.blob.core.windows.net/dotnet-coreclr/index.json": {},
        "https://www.myget.org/F/nugetbuild/api/v3/index.json": {}
      },

The identity package isn't listed as a prebuilt, but it is, it's just omitted because it's ref-only. (#866) The runtime package does show up, in any case:

https://github.com/dotnet/source-build/blob/058f070349183187b86aecea2138f398254c626c/tools-local/prebuilt-baseline-offline.xml#L346

I can confirm my offline build picks up the prebuilt identity package and prebuilt runtime package.

So... this is known in the sense that the baseline includes it.

@dseefeld @crummel do you know if this caused by any known issue? Will it show up in https://github.com/dotnet/source-build/issues/885 at some point as something to drive down?

[dseefeld]

It will show up at some point in work required to drive down prebuilts. The current issues that we have logged are just to drive down prebuilts that are direct dependencies. The assumption is that these will eliminate the most of the transitive dependencies as well, but in this case, I think we have more work to do.

[dagood]

Ah... since the identity package is the direct dependency but it's ref-only, the runtime packages fly under the radar because those come from the identity package's runtime.json.

About fixing, I think at some point I had a hypothesis that the search in the local dir and online are happening in parallel, and the online one might go faster and win. I'm not sure offhand how to verify this. A fix could be enforcing offline in this particular case.

I don't have time to look at this further right now, but this is a more serious problem than most prebuilts IMO because corefx is being entirely ignored.

[dseefeld]

@omajid It appears that the version of Microsoft.Private.CoreFx.NETCoreApp from PackageVersions.props is being overwritten from eng/Versions.props. The proposed fix is to swap these two lines: https://github.com/dotnet/core-setup/blob/328e1e262b67443fe47f091cf6bdde0b1b09bec4/dir.props#L55-L56 I'm testing that out now.

[omajid]

That seems to have done the trick for me! Thanks!