search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.38k stars 4.75k forks source link

Leaking System.Net.Http.WinHttpRequestState in WCF Streaming over Http #16309

Closed KKhurin closed 4 years ago

KKhurin commented 8 years ago

This is really fast - ~1GB per 1minute. To repro:

dnx  --configuration RETAIL run .  Program2Run:Perf Binding:Http Async:true Test:Streaming

Here are some details:

0:012> !gcroot 000000859f603ab8
HandleTable:
    00000086a7c81668 (strong handle)
    -> 000000859e5f5780 System.Net.Http.WinHttpRequestState
    -> 000000859e5f5718 System.Threading.Tasks.TaskCompletionSource`1[[System.Net.Http.HttpResponseMessage, System.Net.Http]]
    -> 000000859e5f5730 System.Threading.Tasks.Task`1[[System.Net.Http.HttpResponseMessage, System.Net.Http]]
    -> 000000841e3aa2a8 System.Net.Http.HttpResponseMessage
    -> 000000859e5f4ab0 System.Net.Http.HttpRequestMessage
    -> 000000859e5f4af8 System.Net.Http.Headers.HttpRequestHeaders
    -> 000000859e5f4c30 System.Collections.Generic.Dictionary`2[[System.String, mscorlib],[System.Net.Http.Headers.HttpHeaders+HeaderStoreItemInfo, System.Net.Http]]
    -> 000000859e5f4ca8 System.Collections.Generic.Dictionary`2+Entry[[System.String, mscorlib],[System.Net.Http.Headers.HttpHeaders+HeaderStoreItemInfo, System.Net.Http]][]
    -> 000000859f603ab8 System.Net.Http.Headers.HttpHeaders+HeaderStoreItemInfo

Found 1 unique roots (run '!GCRoot -all' to see all roots).
0:012> !do 000000859e5f5780 
Name:        System.Net.Http.WinHttpRequestState
MethodTable: 00007fff0f29a520
EEClass:     00007fff0f28fc98
Size:        184(0xb8) bytes
File:        C:\Users\kkhurin\.dnx\packages\System.Net.Http.WinHttpHandler\4.0.0-rc3-23720\lib\DNXCore50\System.Net.Http.WinHttpHandler.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
00007fff696e15e8  4000044        8        System.Object  0 instance 000000859e5f5838 _lock
00007fff696e3628  4000045       a0 ...Services.GCHandle  1 instance 000000859e5f5820 _operationHandle
00007fff696e5d90  4000046       98       System.Boolean  1 instance                0 _disposed
00007fff6fbf5c58  4000047       10 ...System.Net.Http]]  0 instance 000000859e5f5718 <Tcs>k__BackingField
00007fff696ef250  4000048       a8 ...CancellationToken  1 instance 000000859e5f5828 <CancellationToken>k__BackingField
00007fff6fbf7f90  4000049       18 ...ttpRequestMessage  0 instance 000000859e5f4ab0 <RequestMessage>k__BackingField
00007fff0f292580  400004a       20 ...tp.WinHttpHandler  0 instance 000000839dd16e30 <Handler>k__BackingField
00007fff0f29b288  400004b       28 ...SafeWinHttpHandle  0 instance 000000831e798458 <RequestHandle>k__BackingField
00007fff696e1330  400004c       30     System.Exception  0 instance 0000000000000000 <SavedException>k__BackingField
00007fff696e5d90  400004d       99       System.Boolean  1 instance                0 <CheckCertificateRevocationList>k__BackingField
00007fff6fbf3738  400004e       38 ...olean, mscorlib]]  0 instance 0000000000000000 <ServerCertificateValidationCallback>k__BackingField
00007fff0f29a900  400004f       40 ...pTransportContext  0 instance 000000859e5f5850 <TransportContext>k__BackingField
00007fff0f292428  4000050       90         System.Int32  1 instance                2 <WindowsProxyUsePolicy>k__BackingField
00007fff69f7fd18  4000051       48 System.Net.IWebProxy  0 instance 0000000000000000 <Proxy>k__BackingField
00007fff69f7fc78  4000052       50 ....Net.ICredentials  0 instance 0000000000000000 <ServerCredentials>k__BackingField
00007fff69f7fc78  4000053       58 ....Net.ICredentials  0 instance 000000839dcaed18 <DefaultProxyCredentials>k__BackingField
00007fff696e5d90  4000054       9a       System.Boolean  1 instance                1 <PreAuthenticate>k__BackingField
00007fff69f7fc18  4000055       94         System.Int32  1 instance                0 <LastStatusCode>k__BackingField
00007fff696e5d90  4000056       9b       System.Boolean  1 instance                0 <RetryRequest>k__BackingField
00007fff0f11cde8  4000057       60 ...olean, mscorlib]]  0 instance 000000831e79a068 <TcsSendRequest>k__BackingField
00007fff0f11cde8  4000058       68 ...olean, mscorlib]]  0 instance 0000000000000000 <TcsWriteToRequestStream>k__BackingField
00007fff0f11cde8  4000059       70 ...olean, mscorlib]]  0 instance 000000831e79a430 <TcsInternalWriteDataToRequestStream>k__BackingField
00007fff0f11cde8  400005a       78 ...olean, mscorlib]]  0 instance 000000831e79a498 <TcsReceiveResponseHeaders>k__BackingField
00007fff0eda9d10  400005b       80 ...Int32, mscorlib]]  0 instance 000000859e5f8428 <TcsQueryDataAvailable>k__BackingField
00007fff0eda9d10  400005c       88 ...Int32, mscorlib]]  0 instance 000000859e5f83c0 <TcsReadFromResponseStream>k__BackingField
0:012> !do 000000841e3aa2a8 ; !do 000000859e5f4ab0 
Name:        System.Net.Http.HttpResponseMessage
MethodTable: 00007fff6fbf8120
EEClass:     00007fff6fb5b240
Size:        64(0x40) bytes
File:        C:\Users\kkhurin\.dnx\runtimes\dnx-coreclr-win-x64.1.0.0-rc1-update1\bin\System.Net.Http.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
00007fff69f7fc18  40000b6       30         System.Int32  1 instance              200 _statusCode
00007fff6fbf6f20  40000b7        8 ...tpResponseHeaders  0 instance 000000841e3aa3e8 _headers
00007fff696dfc28  40000b8       10        System.String  0 instance 000000841e3aa2e8 _reasonPhrase
00007fff6fbf7f90  40000b9       18 ...ttpRequestMessage  0 instance 000000859e5f4ab0 _requestMessage
00007fff696ec228  40000ba       20       System.Version  0 instance 000000831daf5a88 _version
00007fff6fbf6a60  40000bb       28 ....Http.HttpContent  0 instance 000000841e3aa340 _content
00007fff696e5d90  40000bc       34       System.Boolean  1 instance                0 _disposed
Name:        System.Net.Http.HttpRequestMessage
MethodTable: 00007fff6fbf7f90
EEClass:     00007fff6fb5b138
Size:        72(0x48) bytes
File:        C:\Users\kkhurin\.dnx\runtimes\dnx-coreclr-win-x64.1.0.0-rc1-update1\bin\System.Net.Http.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
00007fff696f7db8  40000ad       38         System.Int32  1 instance                1 _sendStatus
00007fff6fbf8cf8  40000ae        8 ...t.Http.HttpMethod  0 instance 000000839dcaf910 _method
00007fff7cf55c38  40000af       10           System.Uri  0 instance 000000839dd11a08 _requestUri
00007fff6fbf6e98  40000b0       18 ...ttpRequestHeaders  0 instance 000000859e5f4af8 _headers
00007fff696ec228  40000b1       20       System.Version  0 instance 000000839dcafa40 _version
00007fff6fbf6a60  40000b2       28 ....Http.HttpContent  0 instance 000000859e5f51f0 _content
00007fff696e5d90  40000b3       3c       System.Boolean  1 instance                1 _disposed
00007fff68db3b78  40000b4       30 ...bject, mscorlib]]  0 instance 0000000000000000 _properties
davidsh commented 8 years ago

We fixed a bug of this nature a while ago. According to this repro log, you are using an old version of System.Net.Http:

C:\Users\kkhurin\.dnx\runtimes\dnx-coreclr-win-x64.1.0.0-rc1-update1\bin\System.Net.Http.dll

Please make sure you use the latest version of System.Net.Http from the CoreFx dev feed:

https://dotnet.myget.org/gallery/dotnet-core System.Net.Http v4.0.1-rc3-23803

Note that DNX currently "pins" the System.Net.* libraries in its runtimes folders. This makes it difficult to force it to use the latest version of packages. DNX is going away and being replaced by "dotnet cli".

Can you please redo this repro and verify that you are using the latest version of System.Net.Http library?

davidsh commented 8 years ago

Also, this WinHttpRequestState object will always be strongly pinned during the lifetime of the HttpResponseMessage. It shows from your log that the HttpResponseMessage object is still alive which means the .Content (response content) is still alive.

0:012> !do 000000841e3aa2a8 ; !do 000000859e5f4ab0 
Name:        System.Net.Http.HttpResponseMessage
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
00007fff696e5d90  40000bc       34       System.Boolean  1 instance                0 _disposed

So, you need to "dispose" the HttpResponseMessage in order for the memory to be freed and the WinHttpRequestState to be unpinned.

Does your stress test dispose the HttpResponseMessage at all?

davidsh commented 8 years ago

Turns out we still have a cycle between strongly root WinHttpRequestState and the inner field of RequestHandle.

00007ffcec778d28 42738 7863792 System.Net.Http.WinHttpRequestState 0:012> !gcroot 000000e691bf1610 HandleTable: 000000e50ff1b890 (strong handle) -> 000000e691bf1610 System.Net.Http.WinHttpRequestState Plus I see a huge native heap: 0:012> !address -summary --- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal Free 164 7ffa`a770c000 ( 127.979 TB) 99.98%

566 4`8272b000 ( 18.038 GB) 84.36% 0.01% Heap 339 0`c69d1000 ( 3.103 GB) 14.51% 0.00% Stack 447 0`09040000 ( 144.250 MB) 0.66% 0.00% Image 675 0`0469c200 ( 70.610 MB) 0.32% 0.00% MappedFile 22 0`01e21e00 ( 30.132 MB) 0.14% 0.00% Other 8 0`001bf000 ( 1.746 MB) 0.01% 0.00% TEB 149 0`0012a000 ( 1.164 MB) 0.01% 0.00% PEB 1 0`00001000 ( 4.000 kB) 0.00% 0.00% 0:012> !heapstats 0 Heap Total Total Total WOB WOB WOC WOC Largest Handle Count Size Reserved Size Bytes Size Count Allocation --- 000000e5`0fd00000 2 1,868 65,536 1,807 1,807 0 1 1,807 000000e5`0ffa0000 69,560 1,464,145,196 1,474,506,752 32,768 1,133,772,800 32,768 34,600 32,768 000000e5`0ffb0000 1,446,618 1,760,457,444 1,852,621,664 16,384 567,033,856 20 149,815 9,796,080