search

dotnet / machinelearning

ML.NET is an open source and cross-platform machine learning framework for .NET.
https://dot.net/ml
MIT License
8.93k stars 1.86k forks source link

Symbols misssing from symbol servers for ML.NET 3.0 #6934

Closed ericstj closed 5 months ago

ericstj commented 6 months ago

Describe the bug Using the latest Microsoft.ML packages, I can't resolve symbols.

To Reproduce

  1. Build a project using ML.NET 3.0. (or use attached ml-example.zip)
  2. Run symchk to check for symbols.

Output with public server:

\debuggers\SymChk.exe /s https://msdl.microsoft.com/download/symbols *.dll
SYMCHK: CpuMathNative.dll    FAILED  - CpuMathNative.pdb mismatched or not found
SYMCHK: FastTreeNative.dll   FAILED  - FastTreeNative.pdb mismatched or not found
SYMCHK: LdaNative.dll        FAILED  - LdaNative.pdb mismatched or not found
SYMCHK: Microsoft.ML.Core.dll FAILED  - /_/artifacts/obj/Microsoft.ML.Core/Release/netstandard2.0/Microsoft.ML.Core.pdb mismatched or not found
SYMCHK: Microsoft.ML.CpuMath.dll FAILED  - /_/artifacts/obj/Microsoft.ML.CpuMath/Release/netstandard2.0/Microsoft.ML.CpuMath.pdb mismatched or not found
SYMCHK: Microsoft.ML.Data.dll FAILED  - /_/artifacts/obj/Microsoft.ML.Data/Release/netstandard2.0/Microsoft.ML.Data.pdb mismatched or not found
SYMCHK: Microsoft.ML.DataView.dll FAILED  - /_/artifacts/obj/Microsoft.ML.DataView/Release/netstandard2.0/Microsoft.ML.DataView.pdb mismatched or not found
SYMCHK: Microsoft.ML.dll     FAILED  - /_/artifacts/obj/Microsoft.ML/Release/netstandard2.0/Microsoft.ML.pdb mismatched or not found
SYMCHK: Microsoft.ML.FastTree.dll FAILED  - /_/artifacts/obj/Microsoft.ML.FastTree/Release/netstandard2.0/Microsoft.ML.FastTree.pdb mismatched or not found
SYMCHK: Microsoft.ML.KMeansClustering.dll FAILED  - /_/artifacts/obj/Microsoft.ML.KMeansClustering/Release/netstandard2.0/Microsoft.ML.KMeansClustering.pdb mismatched or not found
SYMCHK: Microsoft.ML.PCA.dll FAILED  - /_/artifacts/obj/Microsoft.ML.PCA/Release/netstandard2.0/Microsoft.ML.PCA.pdb mismatched or not found
SYMCHK: Microsoft.ML.StandardTrainers.dll FAILED  - /_/artifacts/obj/Microsoft.ML.StandardTrainers/Release/netstandard2.0/Microsoft.ML.StandardTrainers.pdb mismatched or not found
SYMCHK: Microsoft.ML.Transforms.dll FAILED  - /_/artifacts/obj/Microsoft.ML.Transforms/Release/netstandard2.0/Microsoft.ML.Transforms.pdb mismatched or not found
SYMCHK: Newtonsoft.Json.dll  FAILED  - /_/Src/Newtonsoft.Json/obj/Release/net45/Newtonsoft.Json.pdb mismatched or not found
SYMCHK: System.Memory.dll    FAILED  - System.Memory.pdb mismatched or not found

SYMCHK: FAILED files = 15
SYMCHK: PASSED + IGNORED files = 9

Output with internal server:

C:\scratch\ml-example\bin\Debug\net4.8>\debuggers\SymChk.exe /s http://symweb *.dll
SYMCHK: CpuMathNative.dll    FAILED  - CpuMathNative.pdb mismatched or not found
SYMCHK: FastTreeNative.dll   FAILED  - FastTreeNative.pdb mismatched or not found
SYMCHK: LdaNative.dll        FAILED  - LdaNative.pdb mismatched or not found
SYMCHK: Microsoft.ML.Core.dll FAILED  - /_/artifacts/obj/Microsoft.ML.Core/Release/netstandard2.0/Microsoft.ML.Core.pdb mismatched or not found
SYMCHK: Microsoft.ML.CpuMath.dll FAILED  - /_/artifacts/obj/Microsoft.ML.CpuMath/Release/netstandard2.0/Microsoft.ML.CpuMath.pdb mismatched or not found
SYMCHK: Microsoft.ML.Data.dll FAILED  - /_/artifacts/obj/Microsoft.ML.Data/Release/netstandard2.0/Microsoft.ML.Data.pdb mismatched or not found
SYMCHK: Microsoft.ML.DataView.dll FAILED  - /_/artifacts/obj/Microsoft.ML.DataView/Release/netstandard2.0/Microsoft.ML.DataView.pdb mismatched or not found
SYMCHK: Microsoft.ML.dll     FAILED  - /_/artifacts/obj/Microsoft.ML/Release/netstandard2.0/Microsoft.ML.pdb mismatched or not found
SYMCHK: Microsoft.ML.FastTree.dll FAILED  - /_/artifacts/obj/Microsoft.ML.FastTree/Release/netstandard2.0/Microsoft.ML.FastTree.pdb mismatched or not found
SYMCHK: Microsoft.ML.KMeansClustering.dll FAILED  - /_/artifacts/obj/Microsoft.ML.KMeansClustering/Release/netstandard2.0/Microsoft.ML.KMeansClustering.pdb mismatched or not found
SYMCHK: Microsoft.ML.PCA.dll FAILED  - /_/artifacts/obj/Microsoft.ML.PCA/Release/netstandard2.0/Microsoft.ML.PCA.pdb mismatched or not found
SYMCHK: Microsoft.ML.StandardTrainers.dll FAILED  - /_/artifacts/obj/Microsoft.ML.StandardTrainers/Release/netstandard2.0/Microsoft.ML.StandardTrainers.pdb mismatched or not found
SYMCHK: Microsoft.ML.Transforms.dll FAILED  - /_/artifacts/obj/Microsoft.ML.Transforms/Release/netstandard2.0/Microsoft.ML.Transforms.pdb mismatched or not found
SYMCHK: System.Memory.dll    FAILED  - System.Memory.pdb mismatched or not found

SYMCHK: FAILED files = 14
SYMCHK: PASSED + IGNORED files = 10

Sample verbose output for a single assembly:

\debuggers\SymChk.exe /s https://msdl.microsoft.com/download/symbols Microsoft.ML.Core.dll
SYMCHK: Microsoft.ML.Core.dll FAILED  - /_/artifacts/obj/Microsoft.ML.Core/Release/netstandard2.0/Microsoft.ML.Core.pdb mismatched or not found

SYMCHK: FAILED files = 1
SYMCHK: PASSED + IGNORED files = 0

C:\scratch\ml-example\bin\Debug\net4.8>\debuggers\SymChk.exe /s https://msdl.microsoft.com/download/symbols Microsoft.ML.Core.dll /v
[SYMCHK] Searching for symbols to C:\scratch\ml-example\bin\Debug\net4.8\Microsoft.ML.Core.dll in path https://msdl.microsoft.com/download/symbols
DBGHELP: Symbol Search Path: https://msdl.microsoft.com/download/symbols
[SYMCHK] Using search path "https://msdl.microsoft.com/download/symbols"
DBGHELP: No header for C:\scratch\ml-example\bin\Debug\net4.8\Microsoft.ML.Core.dll.  Searching for image on disk
DBGHELP: C:\scratch\ml-example\bin\Debug\net4.8\Microsoft.ML.Core.dll - OK
SYMSRV:  BYINDEX: 0x1
         https://msdl.microsoft.com/download/symbols
         Microsoft.ML.Core.pdb
         CC84F9CEA3EC4F749E7CFCAEEA342D8A1
SYMSRV:  UNC: C:\debuggers\sym\Microsoft.ML.Core.pdb\CC84F9CEA3EC4F749E7CFCAEEA342D8A1\Microsoft.ML.Core.pdb - path not found
SYMSRV:  UNC: C:\debuggers\sym\Microsoft.ML.Core.pdb\CC84F9CEA3EC4F749E7CFCAEEA342D8A1\Microsoft.ML.Core.pd_ - path not found
SYMSRV:  UNC: C:\debuggers\sym\Microsoft.ML.Core.pdb\CC84F9CEA3EC4F749E7CFCAEEA342D8A1\file.ptr - path not found
SYMSRV:  HTTPGET: /download/symbols/index2.txt
SYMSRV:  HttpQueryInfo: 80190194 - HTTP_STATUS_NOT_FOUND
SYMSRV:  HTTPGET: /download/symbols/Microsoft.ML.Core.pdb/CC84F9CEA3EC4F749E7CFCAEEA342D8A1/Microsoft.ML.Core.pdb
SYMSRV:  HttpQueryInfo: 80190194 - HTTP_STATUS_NOT_FOUND
SYMSRV:  HTTPGET: /download/symbols/Microsoft.ML.Core.pdb/CC84F9CEA3EC4F749E7CFCAEEA342D8A1/Microsoft.ML.Core.pd_
SYMSRV:  HttpQueryInfo: 80190194 - HTTP_STATUS_NOT_FOUND
SYMSRV:  HTTPGET: /download/symbols/Microsoft.ML.Core.pdb/CC84F9CEA3EC4F749E7CFCAEEA342D8A1/file.ptr
SYMSRV:  HttpQueryInfo: 80190194 - HTTP_STATUS_NOT_FOUND
SYMSRV:  RESULT: 0x80190194
SYMSRV:  BYINDEX: 0x2
         https://msdl.microsoft.com/download/symbols
         Microsoft.ML.Core.pdb
         CC84F9CEA3EC4F749E7CFCAEEA342D8Affffffff
SYMSRV:  UNC: C:\debuggers\sym\Microsoft.ML.Core.pdb\CC84F9CEA3EC4F749E7CFCAEEA342D8Affffffff\Microsoft.ML.Core.pdb - path not found
SYMSRV:  UNC: C:\debuggers\sym\Microsoft.ML.Core.pdb\CC84F9CEA3EC4F749E7CFCAEEA342D8Affffffff\Microsoft.ML.Core.pd_ - path not found
SYMSRV:  UNC: C:\debuggers\sym\Microsoft.ML.Core.pdb\CC84F9CEA3EC4F749E7CFCAEEA342D8Affffffff\file.ptr - path not found
SYMSRV:  HTTPGET: /download/symbols/Microsoft.ML.Core.pdb/CC84F9CEA3EC4F749E7CFCAEEA342D8Affffffff/Microsoft.ML.Core.pdb
SYMSRV:  HttpQueryInfo: 80190194 - HTTP_STATUS_NOT_FOUND
SYMSRV:  HTTPGET: /download/symbols/Microsoft.ML.Core.pdb/CC84F9CEA3EC4F749E7CFCAEEA342D8Affffffff/Microsoft.ML.Core.pd_
SYMSRV:  HttpQueryInfo: 80190194 - HTTP_STATUS_NOT_FOUND
SYMSRV:  HTTPGET: /download/symbols/Microsoft.ML.Core.pdb/CC84F9CEA3EC4F749E7CFCAEEA342D8Affffffff/file.ptr
SYMSRV:  HttpQueryInfo: 80190194 - HTTP_STATUS_NOT_FOUND
SYMSRV:  RESULT: 0x80190194
DBGHELP: Microsoft.ML.Core - no symbols loaded
[SYMCHK] MODULE64 Info ----------------------
[SYMCHK] Struct size: 1680 bytes
[SYMCHK] Base: 0x10000000
[SYMCHK] Image size: 434176 bytes
[SYMCHK] Date: 0xed43a25e
[SYMCHK] Checksum: 0x00070a89
[SYMCHK] NumSyms: 0
[SYMCHK] SymType: SymNone
[SYMCHK] ModName: Microsoft.ML.Core
[SYMCHK] ImageName: C:\scratch\ml-example\bin\Debug\net4.8\Microsoft.ML.Core.dll
[SYMCHK] LoadedImage: C:\scratch\ml-example\bin\Debug\net4.8\Microsoft.ML.Core.dll
[SYMCHK] PDB: ""
[SYMCHK] CV: RSDS
[SYMCHK] CV DWORD: 0x53445352
[SYMCHK] CV Data:  /_/artifacts/obj/Microsoft.ML.Core/Release/netstandard2.0/Microsoft.ML.Core.pdb
[SYMCHK] PDB Sig:  0
[SYMCHK] PDB7 Sig: {CC84F9CE-A3EC-4F74-9E7C-FCAEEA342D8A}
[SYMCHK] Age: 1
[SYMCHK] PDB Matched:  TRUE
[SYMCHK] DBG Matched:  TRUE
[SYMCHK] Line numbers: FALSE
[SYMCHK] Global syms:  FALSE
[SYMCHK] Type Info:    FALSE
[SYMCHK] ------------------------------------
SymbolCheckVersion  0x00000002
Result              0x00010001
DbgFilename         Microsoft.ML.Core.dbg
DbgTimeDateStamp    0x00000000
DbgSizeOfImage      0x00000000
DbgChecksum         0x00000000
PdbFilename         /_/artifacts/obj/Microsoft.ML.Core/Release/netstandard2.0/Microsoft.ML.Core.pdb
PdbSignature        {CC84F9CE-A3EC-4F74-9E7C-FCAEEA342D8A}
PdbDbiAge           0x00000001
[SYMCHK] [ 0x00000000 - 0x00010001 ] Checked "C:\scratch\ml-example\bin\Debug\net4.8\Microsoft.ML.Core.dll"
SYMCHK: Microsoft.ML.Core.dll FAILED  - /_/artifacts/obj/Microsoft.ML.Core/Release/netstandard2.0/Microsoft.ML.Core.pdb mismatched or not found

SYMCHK: FAILED files = 1
SYMCHK: PASSED + IGNORED files = 0

Expected behavior All ML assembly's symbols are published to both internal and public symbol servers.

I suspect at some point our symbol publishing broke. I spot checked 2.0.1 and it was working, but only for managed assemblies. Modifying the above sample to use 2.0.1 instead of 3.0.0 I get:

C:\scratch\ml-example\bin\Debug\net4.8>\debuggers\SymChk.exe /s https://msdl.microsoft.com/download/symbols *.dll
SYMCHK: CpuMathNative.dll    FAILED  - CpuMathNative.pdb mismatched or not found
SYMCHK: FastTreeNative.dll   FAILED  - FastTreeNative.pdb mismatched or not found
SYMCHK: LdaNative.dll        FAILED  - LdaNative.pdb mismatched or not found

SYMCHK: FAILED files = 3
SYMCHK: PASSED + IGNORED files = 19

Going back we had this same problem with native binaries in previous releases as well. We should make sure we fix that as part of this.

ericstj commented 6 months ago

Oddly enough, I just checked and the latest build out of main is available. I also double checked a build out of main from 2 BS 3 months ago and they also available. Odd that only this release build is missing. (note all are still missing native symbols)

So it's only the release build. The release build was also never actually published to our feeds. So perhaps symbol publishing is part of the pipeline that's "delayed" in a stable release build. Maybe we need to kick off the actual publishing of that build to get it's symbols published.

ericstj commented 6 months ago

Ok, so with the latest arcade changes to switch to .symbols.nupkg native symbols publishing is now working correctly.

The only remaining gap here is why we don't get publishing for release builds. I believe that's because the builds don't get promoted to any channel by darc publish. I think we can fix this by adding a step to our release process to manually promote the build with a darc command (once we decide which build we want to publish). I'm waiting for confirmation on this.

ericstj commented 6 months ago

We discovered the problem with the release builds. It had to do with our darc channel configuration for release braches. That's been fixed now.

Last piece here is to port the changes to release.

ericstj commented 5 months ago

Fixed in 3.0.1