dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.26k stars 4.73k forks source link

CertGetCertificateChain memory leak in pure Windows 7 system #68892

Closed lindexi closed 2 years ago

lindexi commented 2 years ago

Description

I find my application memory leak in Windows 7 system which install by cn_windows_7_ultimate_with_sp1_x64_dvd_u_677408.iso

I debug it with WinDbg and gflags.exe tools. And the memory leak call stack is:

0:042> !heap -p -a 1fa93750
    address 1fa93750 found in
    _HEAP @ 490000
      HEAP_ENTRY Size Prev Flags    UserPtr UserSize - state
        1fa93738 4f36 0000  [00]   1fa93750    27994 - (busy)
        7741df42 ntdll!RtlAllocateHeap+0x00000274
        76874ec3 KERNELBASE!LocalAlloc+0x0000005f
        76424b84 CRYPT32!PkiAlloc+0x00000032
        764516b3 CRYPT32!ChainCreateCyclicPathObject+0x000000b8
        764515c7 CRYPT32!ExtractEncodedCtlFromCab+0x000001b0
        7645142c CRYPT32!ExtractAuthRootAutoUpdateCtlFromCab+0x00000041
        764504d3 CRYPT32!CCertChainEngine::GetAuthRootAutoUpdateCtl+0x000001f8
        764c047c CRYPT32!CChainPathObject::GetAuthRootAutoUpdateUrlStore+0x00000082
        76469850 CRYPT32!CChainPathObject::CChainPathObject+0x000003d0
        76437934 CRYPT32!ChainCreatePathObject+0x0000005e
        76437da9 CRYPT32!CCertIssuerList::AddIssuer+0x0000006c
        764387ac CRYPT32!CChainPathObject::FindAndAddIssuersFromStoreByMatchType+0x0000018b
        764386bd CRYPT32!CChainPathObject::FindAndAddIssuersByMatchType+0x00000096
        7643bbc6 CRYPT32!CChainPathObject::FindAndAddIssuers+0x00000063
        764697e0 CRYPT32!CChainPathObject::CChainPathObject+0x0000035b
        76437934 CRYPT32!ChainCreatePathObject+0x0000005e
        76438c8d CRYPT32!CCertChainEngine::CreateChainContextFromPathGraph+0x000001ae
        76438a6e CRYPT32!CCertChainEngine::GetChainContext+0x00000046
        76436d42 CRYPT32!CertGetCertificateChain+0x00000072

And we use the CRYPT32!CertGetCertificateChain in ChainPal.BuildChain, see:

https://github.com/dotnet/runtime/blob/c12bea880a2f1290d16adf97ec1000aa63631da2/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/ChainPal.Windows.BuildChain.cs#L66


WinDbg step:

0:024> !heap -s
LFH Key                   : 0x5327c840
Termination on corruption : ENABLED
  Heap     Flags   Reserv  Commit  Virt   Free  List   UCR  Virt  Lock  Fast 
                    (k)     (k)    (k)     (k) length      blocks cont. heap 
-----------------------------------------------------------------------------
00420000 00000002   48768  43096  48768   1929   715    16    0      3   LFH
006b0000 00001002    1088    680   1088      8    21     2    0      0   LFH
00e30000 00001002     256    204    256      2    21     1    0      0   LFH
00df0000 00041002     256      4    256      2     1     1    0      0      
01170000 00001002    1088    196   1088     16     8     2    0      0   LFH
05970000 00041002     256      4    256      2     1     1    0      0      
05920000 00001002     256    160    256      3     7     1    0      0   LFH
083a0000 00001002     256    172    256    118     3     1    0      0      
0b240000 00001002     256    168    256      5    10     1    0      0   LFH
0a3f0000 00041002     256     16    256      5     1     1    0      0      
0e510000 00011002     256     12    256      9     6     1    0      0      
0ec10000 00001002     256    148    256      6     5     1    0      0   LFH
0ee20000 00001002     256    256    256    111    11     1    0      0   LFH
0ed10000 00001002      64     52     64      7     3     1    0      0      
0f990000 00001002     256      4    256      1     2     1    0      0      
0fdb0000 00001002   12096   4048  12096   2601    32     8    0      0   LFH
    External fragmentation  64 % (32 free blocks)
08700000 00001002      64      4     64      2     1     1    0      0      
-----------------------------------------------------------------------------
0:024> g
(7c0.1874): CLR exception - code e0434352 (first chance)
(7c0.1874): CLR exception - code e0434352 (first chance)
(7c0.e64): Break instruction exception - code 80000003 (first chance)
eax=fff9c000 ebx=00000000 ecx=00000000 edx=7743f7ea esi=00000000 edi=00000000
eip=773b000c esp=0a5efe4c ebp=0a5efe78 iopl=0         nv up ei pl zr na pe nc
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246
ntdll!DbgBreakPoint:
773b000c cc              int     3

The application get some https urls.

0:007> !heap -s
LFH Key                   : 0x5327c840
Termination on corruption : ENABLED
  Heap     Flags   Reserv  Commit  Virt   Free  List   UCR  Virt  Lock  Fast 
                    (k)     (k)    (k)     (k) length      blocks cont. heap 
-----------------------------------------------------------------------------
00420000 00000002   81152  67244  81152   1992   723    18    0      3   LFH
006b0000 00001002    1088    680   1088      8    22     2    0      0   LFH
00e30000 00001002     256    204    256      2    21     1    0      0   LFH
00df0000 00041002     256      4    256      2     1     1    0      0      
01170000 00001002    1088    196   1088     16     9     2    0      0   LFH
05970000 00041002     256      4    256      2     1     1    0      0      
05920000 00001002     256    160    256      3     7     1    0      0   LFH
083a0000 00001002     256    172    256    118     3     1    0      0      
0b240000 00001002     256    168    256      5    10     1    0      0   LFH
0a3f0000 00041002     256     16    256      5     1     1    0      0      
0e510000 00011002     256     12    256      9     6     1    0      0      
0ec10000 00001002     256    148    256      6     5     1    0      0   LFH
0ee20000 00001002     256    256    256    111    11     1    0      0   LFH
0ed10000 00001002      64     52     64      7     3     1    0      0      
0f990000 00001002     256      4    256      1     2     1    0      0      
0fdb0000 00001002   12096   4048  12096   2601    32     8    0      0   LFH
    External fragmentation  64 % (32 free blocks)
08700000 00001002      64      4     64      2     1     1    0      0      
-----------------------------------------------------------------------------

You can find the size of 00420000 ranges from 48768 to 81152 .

0:007> !heap -stat -h 00420000
 heap @ 00420000
group-by: TOTSIZE max-display: 20
    size     #blocks     total     ( %) (percent of total busy bytes)
    27994 71 - 117aa54  (37.88)
    269f8 6f - 10bf288  (36.29)
    fdcc 67 - 661d14  (13.83)
    10 7560 - 75600  (0.99)
    1c 2fec - 53dd0  (0.71)
    49a9c 1 - 49a9c  (0.62)
    390 e3 - 328b0  (0.43)
    711 68 - 2dee8  (0.39)
    284 108 - 29820  (0.35)
    618 64 - 26160  (0.32)
    40 934 - 24d00  (0.31)
    20 11f8 - 23f00  (0.30)
    70 49e - 20520  (0.27)
    50 639 - 1f1d0  (0.26)
    60 4b2 - 1c2c0  (0.24)
    dce0 2 - 1b9c0  (0.23)
    84 2d7 - 176dc  (0.20)
    15f13 1 - 15f13  (0.19)
    15eee 1 - 15eee  (0.19)
    30 6c5 - 144f0  (0.17)

And then I continue run the application. And the application get some https urls...

0:019> !heap -s
LFH Key                   : 0x5327c840
Termination on corruption : ENABLED
  Heap     Flags   Reserv  Commit  Virt   Free  List   UCR  Virt  Lock  Fast 
                    (k)     (k)    (k)     (k) length      blocks cont. heap 
-----------------------------------------------------------------------------
00420000 00000002   97344  91356  97344   2082   730    19    0      3   LFH
006b0000 00001002    1088    680   1088      9    22     2    0      0   LFH
00e30000 00001002     256    204    256      2    21     1    0      0   LFH
00df0000 00041002     256      4    256      2     1     1    0      0      
01170000 00001002    1088    196   1088     17     9     2    0      0   LFH
05970000 00041002     256      4    256      2     1     1    0      0      
05920000 00001002     256    160    256      3     7     1    0      0   LFH
083a0000 00001002     256    172    256    118     3     1    0      0      
0b240000 00001002     256    172    256      5    11     1    0      0   LFH
0a3f0000 00041002     256     16    256      5     1     1    0      0      
0e510000 00011002     256     12    256      9     6     1    0      0      
0ec10000 00001002     256    148    256      6     5     1    0      0   LFH
0ee20000 00001002     256    256    256    111    11     1    0      0   LFH
0ed10000 00001002      64     52     64      7     3     1    0      0      
0f990000 00001002     256      4    256      1     2     1    0      0      
0fdb0000 00001002   12096   4048  12096   2601    32     8    0      0   LFH
    External fragmentation  64 % (32 free blocks)
08700000 00001002      64      4     64      2     1     1    0      0      
-----------------------------------------------------------------------------

You can see the 00420000 uses more memory.

0:019> !heap -stat -h 00420000
 heap @ 00420000
group-by: TOTSIZE max-display: 20
    size     #blocks     total     ( %) (percent of total busy bytes)
    27994 b1 - 1b60f54  (39.25)
    269f8 af - 1a67088  (37.85)
    fdcc a6 - a49248  (14.75)
    10 757a - 757a0  (0.66)
    1c 2ff4 - 53eb0  (0.47)
    49a9c 1 - 49a9c  (0.41)
    711 97 - 42b07  (0.37)
    618 86 - 33090  (0.29)
    390 e3 - 328b0  (0.28)
    284 108 - 29820  (0.23)
    40 935 - 24d40  (0.21)
    20 1236 - 246c0  (0.20)
    70 4a2 - 206e0  (0.18)
    50 63a - 1f220  (0.17)
    60 4b2 - 1c2c0  (0.16)
    dce0 2 - 1b9c0  (0.15)
    84 2d7 - 176dc  (0.13)
    15f13 1 - 15f13  (0.12)
    15eee 1 - 15eee  (0.12)
    30 6c5 - 144f0  (0.11)

You can see that there are more blocks of size 27994

0:019> !heap -flt s 27994
    _HEAP @ 420000
      HEAP_ENTRY Size Prev Flags    UserPtr UserSize - state
        05fd2880 4f34 0000  [00]   05fd2888    27994 - (busy)
        06020c20 4f34 4f34  [00]   06020c28    27994 - (busy)
        0614cc18 4f34 4f34  [00]   0614cc20    27994 - (busy)
        08a719d0 4f34 4f34  [00]   08a719d8    27994 - (busy)
        08b05028 4f34 4f34  [00]   08b05030    27994 - (busy)
        08b9e4f0 4f34 4f34  [00]   08b9e4f8    27994 - (busy)
        0b366408 4f34 4f34  [00]   0b366410    27994 - (busy)
        0b493108 4f34 4f34  [00]   0b493110    27994 - (busy)
        0b4eaf60 4f34 4f34  [00]   0b4eaf68    27994 - (busy)
        0b598b40 4f34 4f34  [00]   0b598b48    27994 - (busy)
        0b5d8b40 4f34 4f34  [00]   0b5d8b48    27994 - (busy)
        0b6b1e68 4f34 4f34  [00]   0b6b1e70    27994 - (busy)
        0b6f3208 4f34 4f34  [00]   0b6f3210    27994 - (busy)
    ...............
        1dde4ce0 4f34 4f34  [00]   1dde4ce8    27994 - (busy)
        1de42e58 4f34 4f34  [00]   1de42e60    27994 - (busy)
        1dea0fd0 4f34 4f34  [00]   1dea0fd8    27994 - (busy)
        1deff148 4f34 4f34  [00]   1deff150    27994 - (busy)
        1df5d2c0 4f34 4f34  [00]   1df5d2c8    27994 - (busy)
        1dfbb438 4f34 4f34  [00]   1dfbb440    27994 - (busy)
        1e0195b0 4f34 4f34  [00]   1e0195b8    27994 - (busy)
        1e077728 4f34 4f34  [00]   1e077730    27994 - (busy)
        1e0d58a0 4f34 4f34  [00]   1e0d58a8    27994 - (busy)
        1e133a18 4f34 4f34  [00]   1e133a20    27994 - (busy)
        1e191b90 4f34 4f34  [00]   1e191b98    27994 - (busy)
        1e1efd08 4f34 4f34  [00]   1e1efd10    27994 - (busy)
        1e24de80 4f34 4f34  [00]   1e24de88    27994 - (busy)
        1e2abff8 4f34 4f34  [00]   1e2ac000    27994 - (busy)
        1e31a178 4f34 4f34  [00]   1e31a180    27994 - (busy)
        1e3782f0 4f34 4f34  [00]   1e3782f8    27994 - (busy)
        1e3d6468 4f34 4f34  [00]   1e3d6470    27994 - (busy)
    _HEAP @ 6b0000
    _HEAP @ e30000
    _HEAP @ df0000
    _HEAP @ 1170000
    _HEAP @ 5970000
    _HEAP @ 5920000
    _HEAP @ 83a0000
    _HEAP @ b240000
    _HEAP @ a3f0000
    _HEAP @ e510000
    _HEAP @ ec10000
    _HEAP @ ee20000
    _HEAP @ ed10000
    _HEAP @ f990000
    _HEAP @ fdb0000
    _HEAP @ 8700000

The alloc call stack as follow:

!heap -p -a 2004dc80
    address 2004dc80 found in
    _HEAP @ 490000
      HEAP_ENTRY Size Prev Flags    UserPtr UserSize - state
        2004dc68 4f36 0000  [00]   2004dc80    27994 - (busy)
        7741df42 ntdll!RtlAllocateHeap+0x00000274
        76874ec3 KERNELBASE!LocalAlloc+0x0000005f
        76424b84 CRYPT32!PkiAlloc+0x00000032
        764516b3 CRYPT32!ChainCreateCyclicPathObject+0x000000b8
        764515c7 CRYPT32!ExtractEncodedCtlFromCab+0x000001b0
        7645142c CRYPT32!ExtractAuthRootAutoUpdateCtlFromCab+0x00000041
        764504d3 CRYPT32!CCertChainEngine::GetAuthRootAutoUpdateCtl+0x000001f8
        764c047c CRYPT32!CChainPathObject::GetAuthRootAutoUpdateUrlStore+0x00000082
        76469850 CRYPT32!CChainPathObject::CChainPathObject+0x000003d0
        76437934 CRYPT32!ChainCreatePathObject+0x0000005e
        76437da9 CRYPT32!CCertIssuerList::AddIssuer+0x0000006c
        764387ac CRYPT32!CChainPathObject::FindAndAddIssuersFromStoreByMatchType+0x0000018b
        764386bd CRYPT32!CChainPathObject::FindAndAddIssuersByMatchType+0x00000096
        7643bbc6 CRYPT32!CChainPathObject::FindAndAddIssuers+0x00000063
        764697e0 CRYPT32!CChainPathObject::CChainPathObject+0x0000035b
        76437934 CRYPT32!ChainCreatePathObject+0x0000005e
        76438c8d CRYPT32!CCertChainEngine::CreateChainContextFromPathGraph+0x000001ae
        76438a6e CRYPT32!CCertChainEngine::GetChainContext+0x00000046
        76436d42 CRYPT32!CertGetCertificateChain+0x00000072
0:042> !heap -p -a 1fa93750
    address 1fa93750 found in
    _HEAP @ 490000
      HEAP_ENTRY Size Prev Flags    UserPtr UserSize - state
        1fa93738 4f36 0000  [00]   1fa93750    27994 - (busy)
        7741df42 ntdll!RtlAllocateHeap+0x00000274
        76874ec3 KERNELBASE!LocalAlloc+0x0000005f
        76424b84 CRYPT32!PkiAlloc+0x00000032
        764516b3 CRYPT32!ChainCreateCyclicPathObject+0x000000b8
        764515c7 CRYPT32!ExtractEncodedCtlFromCab+0x000001b0
        7645142c CRYPT32!ExtractAuthRootAutoUpdateCtlFromCab+0x00000041
        764504d3 CRYPT32!CCertChainEngine::GetAuthRootAutoUpdateCtl+0x000001f8
        764c047c CRYPT32!CChainPathObject::GetAuthRootAutoUpdateUrlStore+0x00000082
        76469850 CRYPT32!CChainPathObject::CChainPathObject+0x000003d0
        76437934 CRYPT32!ChainCreatePathObject+0x0000005e
        76437da9 CRYPT32!CCertIssuerList::AddIssuer+0x0000006c
        764387ac CRYPT32!CChainPathObject::FindAndAddIssuersFromStoreByMatchType+0x0000018b
        764386bd CRYPT32!CChainPathObject::FindAndAddIssuersByMatchType+0x00000096
        7643bbc6 CRYPT32!CChainPathObject::FindAndAddIssuers+0x00000063
        764697e0 CRYPT32!CChainPathObject::CChainPathObject+0x0000035b
        76437934 CRYPT32!ChainCreatePathObject+0x0000005e
        76438c8d CRYPT32!CCertChainEngine::CreateChainContextFromPathGraph+0x000001ae
        76438a6e CRYPT32!CCertChainEngine::GetChainContext+0x00000046
        76436d42 CRYPT32!CertGetCertificateChain+0x00000072
0:042> !heap -p -a 106b9378
    address 106b9378 found in
    _HEAP @ 490000
      HEAP_ENTRY Size Prev Flags    UserPtr UserSize - state
        106b9360 4f36 0000  [00]   106b9378    27994 - (busy)
        7741df42 ntdll!RtlAllocateHeap+0x00000274
        76874ec3 KERNELBASE!LocalAlloc+0x0000005f
        76424b84 CRYPT32!PkiAlloc+0x00000032
        764516b3 CRYPT32!ChainCreateCyclicPathObject+0x000000b8
        764515c7 CRYPT32!ExtractEncodedCtlFromCab+0x000001b0
        7645142c CRYPT32!ExtractAuthRootAutoUpdateCtlFromCab+0x00000041
        764504d3 CRYPT32!CCertChainEngine::GetAuthRootAutoUpdateCtl+0x000001f8
        764c047c CRYPT32!CChainPathObject::GetAuthRootAutoUpdateUrlStore+0x00000082
        76469850 CRYPT32!CChainPathObject::CChainPathObject+0x000003d0
        76437934 CRYPT32!ChainCreatePathObject+0x0000005e
        76438c8d CRYPT32!CCertChainEngine::CreateChainContextFromPathGraph+0x000001ae
        76438a6e CRYPT32!CCertChainEngine::GetChainContext+0x00000046
        76436d42 CRYPT32!CertGetCertificateChain+0x00000072

After I use Fiddler to capture the https, the application no longer has memory leaks.


The dump file is too large for me to upload. And I can share you dump file in private.

Reproduction Steps

  1. Prepare the win7 system. Install the Win7 system by cn_windows_7_ultimate_with_sp1_x64_dvd_u_677408.iso.
  2. Install the KB2533623 to run the dotnet application
  3. Create and run a dotnet project which will get some https urls

Expected behavior

The application can work well without memory leak.

Actual behavior

The application will memory leak by CRYPT32!CertGetCertificateChain.

Regression?

It work can work well in .NET Core 3.1 and memory leak in .NET 6.

Known Workarounds

No response

Configuration

Which version of .NET is the code running on?

.NET 6 6.0.4 (also in 6.0.1)

What OS and version, and what distro if applicable?

Windows 7 (cn_windows_7_ultimate_with_sp1_x64_dvd_u_677408.iso)

Other information

See IE crashes due to SSL certificate check - Problem with MSVCR80.dll, - Microsoft Community

dotnet-issue-labeler[bot] commented 2 years ago

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

ghost commented 2 years ago

Tagging subscribers to this area: @dotnet/area-system-security, @vcsjones See info in area-owners.md if you want to be subscribed.

Issue Details
### Description I find my application memory leak in Windows 7 system which install by `cn_windows_7_ultimate_with_sp1_x64_dvd_u_677408.iso` I debug it with WinDbg and gflags.exe tools. And the memory leak call stack is: ``` 0:042> !heap -p -a 1fa93750 address 1fa93750 found in _HEAP @ 490000 HEAP_ENTRY Size Prev Flags UserPtr UserSize - state 1fa93738 4f36 0000 [00] 1fa93750 27994 - (busy) 7741df42 ntdll!RtlAllocateHeap+0x00000274 76874ec3 KERNELBASE!LocalAlloc+0x0000005f 76424b84 CRYPT32!PkiAlloc+0x00000032 764516b3 CRYPT32!ChainCreateCyclicPathObject+0x000000b8 764515c7 CRYPT32!ExtractEncodedCtlFromCab+0x000001b0 7645142c CRYPT32!ExtractAuthRootAutoUpdateCtlFromCab+0x00000041 764504d3 CRYPT32!CCertChainEngine::GetAuthRootAutoUpdateCtl+0x000001f8 764c047c CRYPT32!CChainPathObject::GetAuthRootAutoUpdateUrlStore+0x00000082 76469850 CRYPT32!CChainPathObject::CChainPathObject+0x000003d0 76437934 CRYPT32!ChainCreatePathObject+0x0000005e 76437da9 CRYPT32!CCertIssuerList::AddIssuer+0x0000006c 764387ac CRYPT32!CChainPathObject::FindAndAddIssuersFromStoreByMatchType+0x0000018b 764386bd CRYPT32!CChainPathObject::FindAndAddIssuersByMatchType+0x00000096 7643bbc6 CRYPT32!CChainPathObject::FindAndAddIssuers+0x00000063 764697e0 CRYPT32!CChainPathObject::CChainPathObject+0x0000035b 76437934 CRYPT32!ChainCreatePathObject+0x0000005e 76438c8d CRYPT32!CCertChainEngine::CreateChainContextFromPathGraph+0x000001ae 76438a6e CRYPT32!CCertChainEngine::GetChainContext+0x00000046 76436d42 CRYPT32!CertGetCertificateChain+0x00000072 ``` And we use the `CRYPT32!CertGetCertificateChain` in `ChainPal.BuildChain`, see: https://github.com/dotnet/runtime/blob/c12bea880a2f1290d16adf97ec1000aa63631da2/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/ChainPal.Windows.BuildChain.cs#L66 -------- WinDbg step: ``` 0:024> !heap -s LFH Key : 0x5327c840 Termination on corruption : ENABLED Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast (k) (k) (k) (k) length blocks cont. heap ----------------------------------------------------------------------------- 00420000 00000002 48768 43096 48768 1929 715 16 0 3 LFH 006b0000 00001002 1088 680 1088 8 21 2 0 0 LFH 00e30000 00001002 256 204 256 2 21 1 0 0 LFH 00df0000 00041002 256 4 256 2 1 1 0 0 01170000 00001002 1088 196 1088 16 8 2 0 0 LFH 05970000 00041002 256 4 256 2 1 1 0 0 05920000 00001002 256 160 256 3 7 1 0 0 LFH 083a0000 00001002 256 172 256 118 3 1 0 0 0b240000 00001002 256 168 256 5 10 1 0 0 LFH 0a3f0000 00041002 256 16 256 5 1 1 0 0 0e510000 00011002 256 12 256 9 6 1 0 0 0ec10000 00001002 256 148 256 6 5 1 0 0 LFH 0ee20000 00001002 256 256 256 111 11 1 0 0 LFH 0ed10000 00001002 64 52 64 7 3 1 0 0 0f990000 00001002 256 4 256 1 2 1 0 0 0fdb0000 00001002 12096 4048 12096 2601 32 8 0 0 LFH External fragmentation 64 % (32 free blocks) 08700000 00001002 64 4 64 2 1 1 0 0 ----------------------------------------------------------------------------- 0:024> g (7c0.1874): CLR exception - code e0434352 (first chance) (7c0.1874): CLR exception - code e0434352 (first chance) (7c0.e64): Break instruction exception - code 80000003 (first chance) eax=fff9c000 ebx=00000000 ecx=00000000 edx=7743f7ea esi=00000000 edi=00000000 eip=773b000c esp=0a5efe4c ebp=0a5efe78 iopl=0 nv up ei pl zr na pe nc cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246 ntdll!DbgBreakPoint: 773b000c cc int 3 ``` The application get some https urls. ``` 0:007> !heap -s LFH Key : 0x5327c840 Termination on corruption : ENABLED Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast (k) (k) (k) (k) length blocks cont. heap ----------------------------------------------------------------------------- 00420000 00000002 81152 67244 81152 1992 723 18 0 3 LFH 006b0000 00001002 1088 680 1088 8 22 2 0 0 LFH 00e30000 00001002 256 204 256 2 21 1 0 0 LFH 00df0000 00041002 256 4 256 2 1 1 0 0 01170000 00001002 1088 196 1088 16 9 2 0 0 LFH 05970000 00041002 256 4 256 2 1 1 0 0 05920000 00001002 256 160 256 3 7 1 0 0 LFH 083a0000 00001002 256 172 256 118 3 1 0 0 0b240000 00001002 256 168 256 5 10 1 0 0 LFH 0a3f0000 00041002 256 16 256 5 1 1 0 0 0e510000 00011002 256 12 256 9 6 1 0 0 0ec10000 00001002 256 148 256 6 5 1 0 0 LFH 0ee20000 00001002 256 256 256 111 11 1 0 0 LFH 0ed10000 00001002 64 52 64 7 3 1 0 0 0f990000 00001002 256 4 256 1 2 1 0 0 0fdb0000 00001002 12096 4048 12096 2601 32 8 0 0 LFH External fragmentation 64 % (32 free blocks) 08700000 00001002 64 4 64 2 1 1 0 0 ----------------------------------------------------------------------------- ``` You can find the size of `00420000` ranges from 48768 to 81152 . ``` 0:007> !heap -stat -h 00420000 heap @ 00420000 group-by: TOTSIZE max-display: 20 size #blocks total ( %) (percent of total busy bytes) 27994 71 - 117aa54 (37.88) 269f8 6f - 10bf288 (36.29) fdcc 67 - 661d14 (13.83) 10 7560 - 75600 (0.99) 1c 2fec - 53dd0 (0.71) 49a9c 1 - 49a9c (0.62) 390 e3 - 328b0 (0.43) 711 68 - 2dee8 (0.39) 284 108 - 29820 (0.35) 618 64 - 26160 (0.32) 40 934 - 24d00 (0.31) 20 11f8 - 23f00 (0.30) 70 49e - 20520 (0.27) 50 639 - 1f1d0 (0.26) 60 4b2 - 1c2c0 (0.24) dce0 2 - 1b9c0 (0.23) 84 2d7 - 176dc (0.20) 15f13 1 - 15f13 (0.19) 15eee 1 - 15eee (0.19) 30 6c5 - 144f0 (0.17) ``` And then I continue run the application. And the application get some https urls... ``` 0:019> !heap -s LFH Key : 0x5327c840 Termination on corruption : ENABLED Heap Flags Reserv Commit Virt Free List UCR Virt Lock Fast (k) (k) (k) (k) length blocks cont. heap ----------------------------------------------------------------------------- 00420000 00000002 97344 91356 97344 2082 730 19 0 3 LFH 006b0000 00001002 1088 680 1088 9 22 2 0 0 LFH 00e30000 00001002 256 204 256 2 21 1 0 0 LFH 00df0000 00041002 256 4 256 2 1 1 0 0 01170000 00001002 1088 196 1088 17 9 2 0 0 LFH 05970000 00041002 256 4 256 2 1 1 0 0 05920000 00001002 256 160 256 3 7 1 0 0 LFH 083a0000 00001002 256 172 256 118 3 1 0 0 0b240000 00001002 256 172 256 5 11 1 0 0 LFH 0a3f0000 00041002 256 16 256 5 1 1 0 0 0e510000 00011002 256 12 256 9 6 1 0 0 0ec10000 00001002 256 148 256 6 5 1 0 0 LFH 0ee20000 00001002 256 256 256 111 11 1 0 0 LFH 0ed10000 00001002 64 52 64 7 3 1 0 0 0f990000 00001002 256 4 256 1 2 1 0 0 0fdb0000 00001002 12096 4048 12096 2601 32 8 0 0 LFH External fragmentation 64 % (32 free blocks) 08700000 00001002 64 4 64 2 1 1 0 0 ----------------------------------------------------------------------------- ``` You can see the `00420000` uses more memory. ``` 0:019> !heap -stat -h 00420000 heap @ 00420000 group-by: TOTSIZE max-display: 20 size #blocks total ( %) (percent of total busy bytes) 27994 b1 - 1b60f54 (39.25) 269f8 af - 1a67088 (37.85) fdcc a6 - a49248 (14.75) 10 757a - 757a0 (0.66) 1c 2ff4 - 53eb0 (0.47) 49a9c 1 - 49a9c (0.41) 711 97 - 42b07 (0.37) 618 86 - 33090 (0.29) 390 e3 - 328b0 (0.28) 284 108 - 29820 (0.23) 40 935 - 24d40 (0.21) 20 1236 - 246c0 (0.20) 70 4a2 - 206e0 (0.18) 50 63a - 1f220 (0.17) 60 4b2 - 1c2c0 (0.16) dce0 2 - 1b9c0 (0.15) 84 2d7 - 176dc (0.13) 15f13 1 - 15f13 (0.12) 15eee 1 - 15eee (0.12) 30 6c5 - 144f0 (0.11) ``` You can see that there are more blocks of size 27994 ``` 0:019> !heap -flt s 27994 _HEAP @ 420000 HEAP_ENTRY Size Prev Flags UserPtr UserSize - state 05fd2880 4f34 0000 [00] 05fd2888 27994 - (busy) 06020c20 4f34 4f34 [00] 06020c28 27994 - (busy) 0614cc18 4f34 4f34 [00] 0614cc20 27994 - (busy) 08a719d0 4f34 4f34 [00] 08a719d8 27994 - (busy) 08b05028 4f34 4f34 [00] 08b05030 27994 - (busy) 08b9e4f0 4f34 4f34 [00] 08b9e4f8 27994 - (busy) 0b366408 4f34 4f34 [00] 0b366410 27994 - (busy) 0b493108 4f34 4f34 [00] 0b493110 27994 - (busy) 0b4eaf60 4f34 4f34 [00] 0b4eaf68 27994 - (busy) 0b598b40 4f34 4f34 [00] 0b598b48 27994 - (busy) 0b5d8b40 4f34 4f34 [00] 0b5d8b48 27994 - (busy) 0b6b1e68 4f34 4f34 [00] 0b6b1e70 27994 - (busy) 0b6f3208 4f34 4f34 [00] 0b6f3210 27994 - (busy) ............... 1dde4ce0 4f34 4f34 [00] 1dde4ce8 27994 - (busy) 1de42e58 4f34 4f34 [00] 1de42e60 27994 - (busy) 1dea0fd0 4f34 4f34 [00] 1dea0fd8 27994 - (busy) 1deff148 4f34 4f34 [00] 1deff150 27994 - (busy) 1df5d2c0 4f34 4f34 [00] 1df5d2c8 27994 - (busy) 1dfbb438 4f34 4f34 [00] 1dfbb440 27994 - (busy) 1e0195b0 4f34 4f34 [00] 1e0195b8 27994 - (busy) 1e077728 4f34 4f34 [00] 1e077730 27994 - (busy) 1e0d58a0 4f34 4f34 [00] 1e0d58a8 27994 - (busy) 1e133a18 4f34 4f34 [00] 1e133a20 27994 - (busy) 1e191b90 4f34 4f34 [00] 1e191b98 27994 - (busy) 1e1efd08 4f34 4f34 [00] 1e1efd10 27994 - (busy) 1e24de80 4f34 4f34 [00] 1e24de88 27994 - (busy) 1e2abff8 4f34 4f34 [00] 1e2ac000 27994 - (busy) 1e31a178 4f34 4f34 [00] 1e31a180 27994 - (busy) 1e3782f0 4f34 4f34 [00] 1e3782f8 27994 - (busy) 1e3d6468 4f34 4f34 [00] 1e3d6470 27994 - (busy) _HEAP @ 6b0000 _HEAP @ e30000 _HEAP @ df0000 _HEAP @ 1170000 _HEAP @ 5970000 _HEAP @ 5920000 _HEAP @ 83a0000 _HEAP @ b240000 _HEAP @ a3f0000 _HEAP @ e510000 _HEAP @ ec10000 _HEAP @ ee20000 _HEAP @ ed10000 _HEAP @ f990000 _HEAP @ fdb0000 _HEAP @ 8700000 ``` The alloc call stack as follow: ``` !heap -p -a 2004dc80 address 2004dc80 found in _HEAP @ 490000 HEAP_ENTRY Size Prev Flags UserPtr UserSize - state 2004dc68 4f36 0000 [00] 2004dc80 27994 - (busy) 7741df42 ntdll!RtlAllocateHeap+0x00000274 76874ec3 KERNELBASE!LocalAlloc+0x0000005f 76424b84 CRYPT32!PkiAlloc+0x00000032 764516b3 CRYPT32!ChainCreateCyclicPathObject+0x000000b8 764515c7 CRYPT32!ExtractEncodedCtlFromCab+0x000001b0 7645142c CRYPT32!ExtractAuthRootAutoUpdateCtlFromCab+0x00000041 764504d3 CRYPT32!CCertChainEngine::GetAuthRootAutoUpdateCtl+0x000001f8 764c047c CRYPT32!CChainPathObject::GetAuthRootAutoUpdateUrlStore+0x00000082 76469850 CRYPT32!CChainPathObject::CChainPathObject+0x000003d0 76437934 CRYPT32!ChainCreatePathObject+0x0000005e 76437da9 CRYPT32!CCertIssuerList::AddIssuer+0x0000006c 764387ac CRYPT32!CChainPathObject::FindAndAddIssuersFromStoreByMatchType+0x0000018b 764386bd CRYPT32!CChainPathObject::FindAndAddIssuersByMatchType+0x00000096 7643bbc6 CRYPT32!CChainPathObject::FindAndAddIssuers+0x00000063 764697e0 CRYPT32!CChainPathObject::CChainPathObject+0x0000035b 76437934 CRYPT32!ChainCreatePathObject+0x0000005e 76438c8d CRYPT32!CCertChainEngine::CreateChainContextFromPathGraph+0x000001ae 76438a6e CRYPT32!CCertChainEngine::GetChainContext+0x00000046 76436d42 CRYPT32!CertGetCertificateChain+0x00000072 ``` ``` 0:042> !heap -p -a 1fa93750 address 1fa93750 found in _HEAP @ 490000 HEAP_ENTRY Size Prev Flags UserPtr UserSize - state 1fa93738 4f36 0000 [00] 1fa93750 27994 - (busy) 7741df42 ntdll!RtlAllocateHeap+0x00000274 76874ec3 KERNELBASE!LocalAlloc+0x0000005f 76424b84 CRYPT32!PkiAlloc+0x00000032 764516b3 CRYPT32!ChainCreateCyclicPathObject+0x000000b8 764515c7 CRYPT32!ExtractEncodedCtlFromCab+0x000001b0 7645142c CRYPT32!ExtractAuthRootAutoUpdateCtlFromCab+0x00000041 764504d3 CRYPT32!CCertChainEngine::GetAuthRootAutoUpdateCtl+0x000001f8 764c047c CRYPT32!CChainPathObject::GetAuthRootAutoUpdateUrlStore+0x00000082 76469850 CRYPT32!CChainPathObject::CChainPathObject+0x000003d0 76437934 CRYPT32!ChainCreatePathObject+0x0000005e 76437da9 CRYPT32!CCertIssuerList::AddIssuer+0x0000006c 764387ac CRYPT32!CChainPathObject::FindAndAddIssuersFromStoreByMatchType+0x0000018b 764386bd CRYPT32!CChainPathObject::FindAndAddIssuersByMatchType+0x00000096 7643bbc6 CRYPT32!CChainPathObject::FindAndAddIssuers+0x00000063 764697e0 CRYPT32!CChainPathObject::CChainPathObject+0x0000035b 76437934 CRYPT32!ChainCreatePathObject+0x0000005e 76438c8d CRYPT32!CCertChainEngine::CreateChainContextFromPathGraph+0x000001ae 76438a6e CRYPT32!CCertChainEngine::GetChainContext+0x00000046 76436d42 CRYPT32!CertGetCertificateChain+0x00000072 ``` ``` 0:042> !heap -p -a 106b9378 address 106b9378 found in _HEAP @ 490000 HEAP_ENTRY Size Prev Flags UserPtr UserSize - state 106b9360 4f36 0000 [00] 106b9378 27994 - (busy) 7741df42 ntdll!RtlAllocateHeap+0x00000274 76874ec3 KERNELBASE!LocalAlloc+0x0000005f 76424b84 CRYPT32!PkiAlloc+0x00000032 764516b3 CRYPT32!ChainCreateCyclicPathObject+0x000000b8 764515c7 CRYPT32!ExtractEncodedCtlFromCab+0x000001b0 7645142c CRYPT32!ExtractAuthRootAutoUpdateCtlFromCab+0x00000041 764504d3 CRYPT32!CCertChainEngine::GetAuthRootAutoUpdateCtl+0x000001f8 764c047c CRYPT32!CChainPathObject::GetAuthRootAutoUpdateUrlStore+0x00000082 76469850 CRYPT32!CChainPathObject::CChainPathObject+0x000003d0 76437934 CRYPT32!ChainCreatePathObject+0x0000005e 76438c8d CRYPT32!CCertChainEngine::CreateChainContextFromPathGraph+0x000001ae 76438a6e CRYPT32!CCertChainEngine::GetChainContext+0x00000046 76436d42 CRYPT32!CertGetCertificateChain+0x00000072 ``` ------ After I use Fiddler to capture the https, the application no longer has memory leaks. ------ The dump file is too large for me to upload. And I can share you dump file in private. ### Reproduction Steps 1. Prepare the win7 system. Install the Win7 system by `cn_windows_7_ultimate_with_sp1_x64_dvd_u_677408.iso`. 2. Install the [KB2533623](http://www.microsoft.com/download/details.aspx?familyid=c79c41b0-fbfb-4d61-b5d8-cadbe184b9fc) to run the dotnet application 3. Create and run a dotnet project which will get some https urls ### Expected behavior The application can work well without memory leak. ### Actual behavior The application will memory leak by `CRYPT32!CertGetCertificateChain`. ### Regression? It work can work well in .NET Core 3.1 and memory leak in .NET 6. ### Known Workarounds _No response_ ### Configuration > Which version of .NET is the code running on? .NET 6 6.0.4 (also in 6.0.1) > What OS and version, and what distro if applicable? Windows 7 (cn_windows_7_ultimate_with_sp1_x64_dvd_u_677408.iso) ### Other information See [IE crashes due to SSL certificate check - Problem with MSVCR80.dll, - Microsoft Community](https://answers.microsoft.com/en-us/ie/forum/all/ie-crashes-due-to-ssl-certificate-check-problem/fc3365bb-3583-4813-9e59-dc3cc8f92d46 )
Author: lindexi
Assignees: -
Labels: `area-System.Security`, `untriaged`
Milestone: -
lindexi commented 2 years ago

I guess this is the problem handled by the caller. As we can see, the ChainPal.BuildChain will return null when the CertGetCertificateChain return false.

https://github.com/dotnet/runtime/blob/c12bea880a2f1290d16adf97ec1000aa63631da2/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/ChainPal.Windows.BuildChain.cs#L66-L69

But the CertGetCertificateChain requires calling CertFreeCertificateChain to release the resource, as the document says. See https://docs.microsoft.com/en-us/windows/win32/api/wincrypt/nf-wincrypt-certgetcertificatechain

We can not call the CertFreeCertificateChain by ChainPal.ReleaseSafeX509ChainHandle when the ChainPal.BuildChain will return null.

https://github.com/dotnet/runtime/blob/215b39abf947da7a40b0cb137eab4bceb24ad3e3/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/ChainPal.Windows.cs#L110-L114

It means that the CertGetCertificateChain may make the application memory leak.

Should we call the CertFreeCertificateChain when CertGetCertificateChain return false?

lindexi commented 2 years ago

Should we call the CertFreeCertificateChain when CertGetCertificateChain return false?

I read the code from mozilla. And I find mozilla do not call the CertFreeCertificateChain when CertGetCertificateChain return false.

See https://hg.mozilla.org/releases/mozilla-release/rev/d9659c22b3c5#l3.347

bartonjs commented 2 years ago

A couple of points:

My instincts say it's just a finalization latency problem.

lindexi commented 2 years ago

Thank you @bartonjs

If you aren't already, you should add GC.Collect(); GC.WaitForPendingFinalizers(); to your program before determining if there was a leak.

The bad news is that no any .NET objects have been leaked. And all the .NET objects be gc.

Even if CertGetCertificateChain returns false, we're tracking the PCCERT_CHAIN_CONTEXT via a SafeHandle

But the chain was not released by CertFreeCertificateChain.

Windows 7 is mostly out of support.

The bad news is there are still many users using Windows 7. Unless .NET abandons the Windows desktop developers.

lindexi commented 2 years ago

I wrote a Chinese blog about it.

bartonjs commented 2 years ago

Given that we're tracking the native resources with a SafeHandle, the release will get called eventually (even if CertGetCertificateChain returns false). It doesn't sound like the leak is coming from .NET.

Windows 7 is long out of main support and Extended Support ends soon. .NET 8 will not support running on Windows 7.

Without demonstration of the problem on a more modern OS there's nothing we can really do here.

lindexi commented 2 years ago

@bartonjs Will Win7 be supported until 2026 ?

aallidina-cb commented 2 years ago

I'm debugging something similar on Windows 2003 server at the moment (different context, but similar leak).

Notice the leaking function is always: ExtractAuthRootAutoUpdateCtlFromCab

That's not directly related to the certificate chain - it looks like the code is trying to grab the latest root certificates. These days, that function will fail on older operating systems because the auto certificate updates are signed with SHA256 and older operating systems don't understand this. That appears to trigger an edge case in the code, resulting in a leak.

As a test, could you install the SHA2 patch on Windows 7 and see if the problem resolves?

Here's the KB article: https://support.microsoft.com/en-us/topic/sha-2-code-signing-support-update-for-windows-server-2008-r2-windows-7-and-windows-server-2008-september-23-2019-84a8aad5-d8d9-2d5c-6d78-34f9aa5f8339

The article points to this download site: https://www.catalog.update.microsoft.com/search.aspx?q=kb4474419

aallidina-cb commented 2 years ago

In my case, when I disabled Automatic Root Certificate Updates on my 2003 Server machine, the leak went away. You could also try that on your Windows 7 box.