search

dotnet / aspnetcore

ASP.NET Core is a cross-platform .NET framework for building modern cloud-based web applications on Windows, Mac, or Linux.
https://asp.net
MIT License
35.52k stars 10.04k forks source link

Excessive memory usage using endpoint.MapControllerRoute (up to 4GB) for Routes Candidates #30092

Closed Sbaia closed 3 years ago

Sbaia commented 3 years ago

Describe the bug

Using the MapControllerRoute function with various route templates the server memory explodes in the order of GB. A project with about 1K of Controllers and each with 30/40 Actions, with a dozen MapControllerRoute endpoints can weight up to 3GB.

We are not using the RouteAttribute decorations, but attributing the routes with MapControllerRoute, for a precise choice and for compatibility with NET.

app.UseEndpoint(endpoint => {
    endpoint.MapControllerRoute({
        name: "RouteName",
        pattern: "api/{controller}/{action}",
    });
    ...
    endpoint.MapControllerRoute({
        name: "RouteName",
        pattern: "api/{controller}/{id}/{action}",
    })
    ..
});

The memory is almost completely occupied by the Microsoft.AspNetCore.Mvc.ModelBinding.Metadata.DefaultModelMetadata class and the internal routing dictionaries.

If, instead of with the UseEndpoint, the same identical routes are linked with UseMvc, there is not problems, and the application has a normal use of RAM (<500MB in Debug, 150MB in Release)

// this works without leak
app.UseMvc(route => {
    route.MapRoute(
                name: name,
                template: routeTemplate,
                defaults: defaults,
                constraints: constraints);
});

To Reproduce

To replicate the problem I created a test project, with 1k controller with 30 actions, and I defined a series of Route patterns, and the RAM usage is also excessive, reaching 3GB.

https://github.com/Sbaia/WebApp.Core.MemoryUsage

Exceptions (if any)

No Exceptions was thrown

Further technical details


dotnet --info
.NET SDK (che rispecchia un qualsiasi file global.json):
 Version:   5.0.102
 Commit:    71365b4d42

Ambiente di runtime:
 OS Name:     Windows
 OS Version:  10.0.17763
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\5.0.102\

Host (useful for support):
  Version: 5.0.2
  Commit:  cb5f173b96

.NET SDKs installed:
  2.1.202 [C:\Program Files\dotnet\sdk]
  2.1.500 [C:\Program Files\dotnet\sdk]
  2.1.801 [C:\Program Files\dotnet\sdk]
  2.2.401 [C:\Program Files\dotnet\sdk]
  3.1.401 [C:\Program Files\dotnet\sdk]
  5.0.100 [C:\Program Files\dotnet\sdk]
  5.0.102 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.All 2.1.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.23 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.23 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.9 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.0 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 5.0.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.23 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.8 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.0 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 5.0.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.1.7 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.1.9 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.0 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 5.0.2 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
ghost commented 3 years ago

Thanks for contacting us. We're moving this issue to the Next sprint planning milestone for future evaluation / consideration. We will evaluate the request when we are planning the work for the next milestone. To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.

NinjaCross commented 3 years ago

I have this problem too, and the performances are terrible due to the absurd memory pressure / allocation :(

davidfowl commented 3 years ago

@NinjaCross how many controllers do you have?

NinjaCross commented 3 years ago

@NinjaCross how many controllers do you have?

I have between 200 and 800 statically/dinamically compiled controllers. The more controllers I have, the heaviest the memory consumption is.

javiercn commented 3 years ago

I did take some time to provide a more detailed explanation here, this is scheduled for our next sprint planning since it requires a more in depth response. However, here are a few pointers:

I compiled in Release mode with the ROUTE constraint defined.

The total memory of the app amounts to 803MB on my system. image

There are ~39K actions on the application (38961 to be precise).

The generated DFA tree by routing contains 26982 nodes as reported by !dumpobj which is (26980 entries) +1 for the root node and +1 for the no match state.

Microsoft.AspNetCore.Routing.Matching.DfaState[]
Array:       Rank 1, Number of elements 26982

The inclusive size for the DFAMatcher (the thing that does the matching is 296740344 (0x11afe5f8) bytes (Microsoft.AspNetCore.Routing.Matching.DfaMatcher) as reported by !objsize in windbg

Here is everything that contributes more than 100K:

Count TotalSize TotalSize in MB "Class Name"
882650 64705984 61.70843506 System.String
520361 36257168 34.57752991 System.Object[]
223056 23197824 22.12316895 System.Reflection.RuntimeMethodInfo
366545 15147108 14.44540787 System.Int32[]
155897 14966112 14.27279663 Microsoft.AspNetCore.Routing.Patterns.RoutePattern
57958 14837248 14.14990234 Microsoft.AspNetCore.Mvc.ModelBinding.Metadata.DefaultModelMetadata
247851 13879120 13.23616028 Microsoft.AspNetCore.Routing.Patterns.RoutePatternPart[]
172865 13181416 12.57077789 System.Collections.Generic.KeyValuePair`2[[System.String, System.Private.CoreLib],[System.Object, System.Private.CoreLib]][]
147866 11829280 11.28128052 System.Collections.Generic.Dictionary`2[[System.Object, System.Private.CoreLib],[System.Object, System.Private.CoreLib]]
127758 11242704 10.72187805 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[Microsoft.AspNetCore.Mvc.ModelBinding.Metadata.ModelMetadataIdentity, Microsoft.AspNetCore.Mvc.Abstractions],[Microsoft.AspNetCore.Mvc.ModelBinding.Metadata.DefaultModelMetadataProvider+ModelMetadataCacheEntry, Microsoft.AspNetCore.Mvc.Core]]
233809 11222832 10.70292664 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Type, System.Private.CoreLib],[System.Object[], System.Private.CoreLib]]
57955 11127360 10.61187744 Microsoft.AspNetCore.Mvc.ModelBinding.Metadata.DefaultModelBindingMessageProvider
193821 11045752 10.53404999 Microsoft.AspNetCore.Routing.Matching.DfaNode[]
111929 10816536 10.31545258 Microsoft.AspNetCore.Http.Endpoint[]
38983 10603376 10.11216736 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.Type, System.Private.CoreLib],[System.Object[], System.Private.CoreLib]][]
198864 10497320 10.01102448 Microsoft.AspNetCore.Routing.Patterns.RoutePatternPathSegment[]
267889 8572448 8.175323486 System.Collections.Generic.List`1[[System.Object, System.Private.CoreLib]]
33 8162280 7.784156799 System.ValueTuple3[[Microsoft.AspNetCore.Routing.RouteEndpoint, Microsoft.AspNetCore.Routing],[System.Int32, System.Private.CoreLib],[System.Collections.Generic.List1[[Microsoft.AspNetCore.Routing.Matching.DfaNode, Microsoft.AspNetCore.Routing]], System.Private.CoreLib]][]
247829 7930528 7.563140869 System.Collections.Generic.List`1[[Microsoft.AspNetCore.Routing.Patterns.RoutePatternPart, Microsoft.AspNetCore.Routing]]
57958 7882288 7.51713562 Microsoft.AspNetCore.Mvc.ModelBinding.Metadata.DefaultMetadataDetails
133885 7497560 7.150230408 System.Collections.Generic.Dictionary`2+Enumerator[[System.String, System.Private.CoreLib],[System.Object, System.Private.CoreLib]]
76924 7384704 7.042602539 System.Collections.Generic.Dictionary`2+Entry[[System.Object, System.Private.CoreLib],[System.Object, System.Private.CoreLib]][]
125645 7036120 6.710166931 System.Collections.Generic.Dictionary`2+Enumerator[[System.String, System.Private.CoreLib],[System.String, System.Private.CoreLib]]
121254 6790192 6.475631714 Microsoft.AspNetCore.Mvc.Filters.FilterDescriptor[]
274980 6599520 6.293792725 System.Object
116923 6547752 6.244422913 Microsoft.AspNetCore.Routing.Patterns.RoutePatternParameterPart[]
82578 6417950 6.120634079 System.Char[]
80116 6409280 6.112365723 System.Collections.Generic.Dictionary`2[[System.String, System.Private.CoreLib],[System.String, System.Private.CoreLib]]
246654 6027104 5.747894287 Free
187826 6010432 5.731994629 System.Collections.Generic.List`1[[Microsoft.AspNetCore.Routing.Matching.DfaNode, Microsoft.AspNetCore.Routing]]
247850 5948400 5.672836304 Microsoft.AspNetCore.Routing.Patterns.RoutePatternPathSegment
77952 5613344 5.353302002 System.Collections.Generic.HashSet`1+Entry[[System.String, System.Private.CoreLib]][]
77943 5611896 5.351921082 System.Collections.Generic.HashSet`1[[System.String, System.Private.CoreLib]]
58366 5603136 5.343566895 System.Reflection.RuntimeParameterInfo
163857 5243424 5.000518799 Microsoft.AspNetCore.Routing.Patterns.RoutePatternLiteralPart
155844 4987008 4.755981445 Microsoft.AspNetCore.Mvc.Filters.FilterDescriptor
38961 4987008 4.755981445 Microsoft.AspNetCore.Mvc.Controllers.ControllerActionDescriptor
62264 4981120 4.750366211 System.Signature
83993 4703608 4.485710144 Microsoft.AspNetCore.Routing.Patterns.RoutePatternParameterPart
117002 4680080 4.463272095 Microsoft.AspNetCore.Routing.RouteValueDictionary
116884 4675344 4.458755493 Microsoft.AspNetCore.Mvc.ActionConstraints.IActionConstraintMetadata[]
57954 4636320 4.421539307 Microsoft.AspNetCore.Mvc.ModelBinding.Metadata.BindingMetadata
18986 4547824 4.337142944 Microsoft.AspNetCore.Routing.Matching.Candidate[]
77939 4477520 4.270095825 Microsoft.AspNetCore.Mvc.Abstractions.ActionDescriptor[]
94138 4438648 4.233024597 System.String[]
77940 4364640 4.162445068 Microsoft.AspNetCore.Routing.Patterns.RoutePatternParser+Context
53962 4316960 4.116973877 Microsoft.AspNetCore.Routing.Matching.DfaNode
57942 4171824 3.978561401 Microsoft.AspNetCore.Mvc.ApplicationModels.ParameterModel
82546 3962208 3.778656006 System.Text.StringBuilder
41152 3952656 3.769546509 System.Collections.Generic.Dictionary`2+Entry[[System.String, System.Private.CoreLib],[System.String, System.Private.CoreLib]][]
37962 3948048 3.765151978 Microsoft.AspNetCore.Mvc.ApplicationModels.ActionModel
122139 3908448 3.727386475 System.Collections.Generic.List`1[[Microsoft.AspNetCore.Mvc.ActionConstraints.IActionConstraintMetadata, Microsoft.AspNetCore.Mvc.Abstractions]]
38976 3741888 3.56854248 System.Collections.Generic.Dictionary`2+Entry[[System.String, System.Private.CoreLib],[System.Object, System.Private.CoreLib]][]
116917 3741344 3.568023682 System.Collections.Generic.List`1[[Microsoft.AspNetCore.Routing.Patterns.RoutePatternParameterPart, Microsoft.AspNetCore.Routing]]
57942 3708288 3.536499023 Microsoft.AspNetCore.Mvc.ModelBinding.BindingInfo
77931 3428960 3.270111084 Microsoft.AspNetCore.Routing.IHttpMethodMetadata[]
38963 3428744 3.26990509 System.Reflection.AssemblyName
5 3402624 3.244995117 System.Collections.Generic.Dictionary2+Entry[[System.Reflection.MethodInfo, System.Private.CoreLib],[System.Collections.Generic.List1[[System.ValueTuple`2[[Microsoft.AspNetCore.Mvc.ApplicationModels.ActionModel, Microsoft.AspNetCore.Mvc.Core],[Microsoft.AspNetCore.Mvc.ApplicationModels.SelectorModel, Microsoft.AspNetCore.Mvc.Core]], System.Private.CoreLib]], System.Private.CoreLib]][]
37963 3340680 3.185920715 System.ValueTuple`2[[Microsoft.AspNetCore.Mvc.ApplicationModels.ActionModel, Microsoft.AspNetCore.Mvc.Core],[Microsoft.AspNetCore.Mvc.ApplicationModels.SelectorModel, Microsoft.AspNetCore.Mvc.Core]][]
81050 3242000 3.091812134 Microsoft.AspNetCore.Mvc.ApplicationModels.SelectorModel
55946 3132976 2.987838745 System.Collections.Generic.Dictionary2+Enumerator[[System.String, System.Private.CoreLib],[System.Collections.Generic.IReadOnlyList1[[Microsoft.AspNetCore.Routing.Patterns.RoutePatternParameterPolicyReference, Microsoft.AspNetCore.Routing]], System.Private.CoreLib]]
38977 3118160 2.973709106 System.Collections.Generic.Dictionary`2[[System.String, System.Private.CoreLib],[System.Object, System.Private.CoreLib]]
77922 3116880 2.972488403 System.Collections.Generic.List`1+Enumerator[[System.Object, System.Private.CoreLib]]
75940 3037600 2.896881104 System.Collections.Generic.List`1+Enumerator[[Microsoft.AspNetCore.Http.Endpoint, Microsoft.AspNetCore.Http.Abstractions]]
91929 2941728 2.805450439 System.Collections.Generic.List`1[[Microsoft.AspNetCore.Http.Endpoint, Microsoft.AspNetCore.Http.Abstractions]]
58941 2829168 2.698104858 Microsoft.AspNetCore.Mvc.Controllers.ControllerParameterDescriptor
57958 2781984 2.653106689 Microsoft.AspNetCore.Mvc.ModelBinding.ModelAttributes
47364 2652384 2.529510498 Microsoft.AspNetCore.Mvc.ApplicationModels.AttributeRouteModel
41149 2633536 2.511535645 System.Comparison`1[[System.Int32, System.Private.CoreLib]]
77940 2494080 2.378540039 System.Collections.Generic.List`1[[Microsoft.AspNetCore.Routing.Patterns.RoutePatternPathSegment, Microsoft.AspNetCore.Routing]]
38961 2493504 2.377990723 System.Linq.OrderedEnumerable`1+d__17[[Microsoft.AspNetCore.Mvc.Filters.FilterDescriptor, Microsoft.AspNetCore.Mvc.Abstractions]]
77922 2493504 2.377990723 System.Collections.Generic.List`1[[Microsoft.AspNetCore.Mvc.Abstractions.ParameterDescriptor, Microsoft.AspNetCore.Mvc.Abstractions]]
17986 2414064 2.302230835 System.Collections.Generic.Dictionary2+Entry[[Microsoft.AspNetCore.Routing.Matching.HttpMethodMatcherPolicy+EdgeKey, Microsoft.AspNetCore.Routing],[System.Collections.Generic.List1[[Microsoft.AspNetCore.Http.Endpoint, Microsoft.AspNetCore.Http.Abstractions]], System.Private.CoreLib]][]
49950 2397600 2.286529541 Microsoft.AspNetCore.Mvc.HttpGetAttribute
55947 2349784 2.24092865 System.ValueTuple`3[[System.String, System.Private.CoreLib],[System.Int32, System.Private.CoreLib],[System.Int32, System.Private.CoreLib]][]
41146 2304176 2.197433472 System.Linq.EnumerableSorter`2[[Microsoft.AspNetCore.Mvc.Filters.FilterDescriptor, Microsoft.AspNetCore.Mvc.Abstractions],[Microsoft.AspNetCore.Mvc.Filters.FilterDescriptor, Microsoft.AspNetCore.Mvc.Abstractions]]
41146 2304176 2.197433472 System.Linq.OrderedEnumerable`2[[Microsoft.AspNetCore.Mvc.Filters.FilterDescriptor, Microsoft.AspNetCore.Mvc.Abstractions],[Microsoft.AspNetCore.Mvc.Filters.FilterDescriptor, Microsoft.AspNetCore.Mvc.Abstractions]]
41091 2301064 2.194465637 Microsoft.AspNetCore.Mvc.ApplicationModels.SelectorModel[]
45954 2205792 2.103607178 Microsoft.AspNetCore.Mvc.HttpPostAttribute
38982 2182992 2.081863403 Microsoft.AspNetCore.Routing.RouteEndpoint
38980 2182880 2.081756592 Microsoft.AspNetCore.Routing.RouteEndpointBuilder
38964 2181952 2.080871582 Microsoft.AspNetCore.Mvc.Filters.IFilterMetadata[]
38961 2181816 2.080741882 System.Linq.Enumerable+d__62`1[[Microsoft.AspNetCore.Routing.HttpMethodMetadata, Microsoft.AspNetCore.Routing]]
38961 2181816 2.080741882 System.Linq.Enumerable+SelectIPartitionIterator`2[[Microsoft.AspNetCore.Mvc.Filters.FilterDescriptor, Microsoft.AspNetCore.Mvc.Abstractions],[Microsoft.AspNetCore.Mvc.Filters.IFilterMetadata, Microsoft.AspNetCore.Mvc.Abstractions]]
4 2150400 2.05078125 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[Microsoft.AspNetCore.Mvc.ModelBinding.Metadata.ModelMetadataIdentity, Microsoft.AspNetCore.Mvc.Abstractions],[Microsoft.AspNetCore.Mvc.ModelBinding.Metadata.DefaultModelMetadataProvider+ModelMetadataCacheEntry, Microsoft.AspNetCore.Mvc.Core]][]
37963 2125896 2.027412415 Microsoft.AspNetCore.Mvc.ApplicationModels.ParameterModel[]
31 2097832 2.000648499 Microsoft.AspNetCore.Routing.RouteEndpoint[]
62411 1966936 1.875816345 System.RuntimeType[]
81038 1944912 1.854812622 System.Int32
17986 1918464 1.829589844 System.Collections.Generic.Dictionary`2+Entry[[System.Object, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.Matching.DfaNode, Microsoft.AspNetCore.Routing]][]
38983 1871184 1.784500122 System.Collections.Concurrent.ConcurrentDictionary`2+Tables[[System.Type, System.Private.CoreLib],[System.Object[], System.Private.CoreLib]]
38983 1871184 1.784500122 System.Collections.Concurrent.ConcurrentDictionary`2[[System.Type, System.Private.CoreLib],[System.Object[], System.Private.CoreLib]]
1106 1819000 1.734733582 System.Reflection.RuntimeMethodInfo[]
55030 1760960 1.679382324 System.Collections.Generic.List`1[[System.String, System.Private.CoreLib]]
13 1679616 1.601806641 System.Collections.Generic.Dictionary`2+Entry[[Microsoft.AspNetCore.Routing.Matching.DfaNode, Microsoft.AspNetCore.Routing],[System.Int32, System.Private.CoreLib]][]
50284 1674304 1.596740723 System.Reflection.ParameterInfo[]
38961 1558440 1.486244202 System.Collections.Generic.List`1+Enumerator[[Microsoft.AspNetCore.Mvc.ActionConstraints.IActionConstraintMetadata, Microsoft.AspNetCore.Mvc.Abstractions]]
38961 1558440 1.486244202 System.Collections.Generic.Dictionary`2+KeyCollection+Enumerator[[System.String, System.Private.CoreLib],[System.String, System.Private.CoreLib]]
38961 1558440 1.486244202 Microsoft.AspNetCore.Mvc.Routing.AttributeRouteInfo
38963 1406640 1.34147644 Microsoft.AspNetCore.Mvc.Abstractions.ParameterDescriptor[]
57949 1390776 1.326347351 System.SerializableAttribute
41090 1314880 1.253967285 System.Collections.Generic.List`1[[Microsoft.AspNetCore.Mvc.ApplicationModels.SelectorModel, Microsoft.AspNetCore.Mvc.Core]]
15988 1279040 1.219787598 System.Collections.Generic.Dictionary2[[Microsoft.AspNetCore.Routing.Matching.HttpMethodMatcherPolicy+EdgeKey, Microsoft.AspNetCore.Routing],[System.Collections.Generic.List1[[Microsoft.AspNetCore.Http.Endpoint, Microsoft.AspNetCore.Http.Abstractions]], System.Private.CoreLib]]
15988 1279040 1.219787598 System.Collections.Generic.Dictionary`2[[System.Object, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.Matching.DfaNode, Microsoft.AspNetCore.Routing]]
12026 1250704 1.192764282 System.Reflection.RuntimePropertyInfo
38983 1247456 1.189666748 Microsoft.AspNetCore.Http.EndpointMetadataCollection
38965 1246880 1.189117432 System.Collections.Generic.List`1[[Microsoft.AspNetCore.Routing.IHttpMethodMetadata, Microsoft.AspNetCore.Routing]]
38963 1246816 1.189056396 System.Collections.Generic.List`1[[Microsoft.AspNetCore.Mvc.Filters.IFilterMetadata, Microsoft.AspNetCore.Mvc.Abstractions]]
38963 1246816 1.189056396 Microsoft.AspNetCore.Routing.HttpMethodMetadata
38962 1246784 1.189025879 Microsoft.AspNetCore.Mvc.ApplicationModels.ApiExplorerModel
38962 1246784 1.189025879 System.Collections.Generic.List`1[[Microsoft.AspNetCore.Mvc.Abstractions.ActionDescriptor, Microsoft.AspNetCore.Mvc.Abstractions]]
38962 1246784 1.189025879 System.Version
38961 1246752 1.188995361 System.Collections.Generic.List`1[[Microsoft.AspNetCore.Mvc.Filters.FilterDescriptor, Microsoft.AspNetCore.Mvc.Abstractions]]
37970 1215040 1.158752441 Microsoft.AspNetCore.Routing.Matching.HttpMethodMatcherPolicy+EdgeKey
37962 1214784 1.158508301 System.Collections.Generic.List1[[System.ValueTuple2[[Microsoft.AspNetCore.Mvc.ApplicationModels.ActionModel, Microsoft.AspNetCore.Mvc.Core],[Microsoft.AspNetCore.Mvc.ApplicationModels.SelectorModel, Microsoft.AspNetCore.Mvc.Core]], System.Private.CoreLib]]
37962 1214784 1.158508301 System.Collections.Generic.List`1[[Microsoft.AspNetCore.Mvc.ApplicationModels.ParameterModel, Microsoft.AspNetCore.Mvc.Core]]
41486 1179728 1.125076294 System.Byte[]
45954 1102896 1.051803589 Microsoft.AspNetCore.Mvc.FromBodyAttribute
9025 1048896 1.000305176 System.Collections.Generic.Dictionary`2+Entry[[System.String, System.Private.CoreLib],[System.Int32, System.Private.CoreLib]][]
16002 1024128 0.97668457 Microsoft.AspNetCore.Http.RequestDelegate
15988 991232 0.9453125 Microsoft.AspNetCore.Routing.Matching.PolicyNodeEdge[]
38978 935472 0.89213562 Microsoft.AspNetCore.Routing.RouteNameMetadata
38962 935088 0.891769409 System.Collections.ObjectModel.ReadOnlyCollection`1[[System.String, System.Private.CoreLib]]
38961 935064 0.891746521 System.Collections.Generic.Dictionary`2+KeyCollection[[System.String, System.Private.CoreLib],[System.String, System.Private.CoreLib]]
38961 935064 0.891746521 Microsoft.AspNetCore.Mvc.ApiDescriptionActionData
38961 935064 0.891746521 Microsoft.AspNetCore.Mvc.ActionConstraints.HttpMethodActionConstraint
4 917600 0.875091553 Microsoft.AspNetCore.Mvc.Controllers.ControllerActionDescriptor[]
1 863448 0.823448181 Microsoft.AspNetCore.Routing.Matching.DfaState[]
11988 863136 0.823150635 Microsoft.AspNetCore.Mvc.ApplicationModels.PropertyModel
34971 839304 0.800422668 System.Runtime.CompilerServices.TypeForwardedFromAttribute
32969 791256 0.754600525 System.Runtime.CompilerServices.IsReadOnlyAttribute
32661 783864 0.747550964 System.Char
4016 750336 0.715576172 System.Collections.Generic.Dictionary`2+Entry[[System.String, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.Matching.DfaNode, Microsoft.AspNetCore.Routing]][]
10989 703296 0.670715332 System.Comparison`1[[Microsoft.AspNetCore.Http.Endpoint, Microsoft.AspNetCore.Http.Abstractions]]
8018 641440 0.611724854 System.Collections.Generic.Dictionary`2[[System.String, System.Private.CoreLib],[System.Int32, System.Private.CoreLib]]
15989 639560 0.609931946 System.Collections.Generic.List`1+Enumerator[[System.String, System.Private.CoreLib]]
15988 639520 0.609893799 Microsoft.AspNetCore.Http.Endpoint
25981 623544 0.594657898 Microsoft.AspNetCore.Routing.Matching.ZeroEntryJumpTable
12000 576000 0.549316406 Microsoft.Extensions.Internal.PropertyHelper
1000 535488 0.510681152 Microsoft.AspNetCore.Mvc.ApplicationModels.ActionModel[]
7994 495616 0.47265625 Microsoft.AspNetCore.Routing.Matching.PolicyJumpTableEdge[]
6555 471960 0.45009613 System.Linq.Enumerable+SelectListIterator`2[[Microsoft.AspNetCore.Mvc.Filters.IFilterMetadata, Microsoft.AspNetCore.Mvc.Abstractions],[Microsoft.AspNetCore.Mvc.Filters.FilterDescriptor, Microsoft.AspNetCore.Mvc.Abstractions]]
10989 439560 0.419197083 Microsoft.AspNetCore.Mvc.RouteAttribute
6995 391720 0.373573303 System.Collections.Generic.Dictionary`2+Enumerator[[System.String, System.Private.CoreLib],[System.Int32, System.Private.CoreLib]]
15988 383712 0.365936279 Microsoft.AspNetCore.Routing.Matching.HttpMethodMatcherPolicy+<>c__DisplayClass15_0
6555 367080 0.350074768 System.Collections.Generic.Dictionary`2+Enumerator[[System.Object, System.Private.CoreLib],[System.Object, System.Private.CoreLib]]
6995 335760 0.320205688 Microsoft.AspNetCore.Routing.Matching.HttpMethodSingleEntryPolicyJumpTable
2129 204384 0.194915771 Microsoft.AspNetCore.Mvc.ApplicationModels.ActionAttributeRouteModel+d__0
3996 191808 0.182922363 Microsoft.AspNetCore.Mvc.HttpPutAttribute
1208 183616 0.175109863 System.RuntimeType+RuntimeTypeCache
3009 178496 0.170227051 System.SByte[]
2185 174800 0.166702271 System.Linq.Enumerable+WhereSelectListIterator`2[[Microsoft.AspNetCore.Mvc.ApplicationModels.PropertyModel, Microsoft.AspNetCore.Mvc.Core],[Microsoft.AspNetCore.Mvc.Abstractions.ParameterDescriptor, Microsoft.AspNetCore.Mvc.Abstractions]]
2002 160160 0.152740479 System.Collections.Generic.Dictionary`2[[System.String, System.Private.CoreLib],[Microsoft.AspNetCore.Routing.Matching.DfaNode, Microsoft.AspNetCore.Routing]]
2185 157320 0.150032043 System.Linq.Enumerable+WhereListIterator`1[[Microsoft.AspNetCore.Mvc.ApplicationModels.PropertyModel, Microsoft.AspNetCore.Mvc.Core]]
2128 153216 0.146118164 System.Linq.Enumerable+WhereListIterator`1[[Microsoft.AspNetCore.Mvc.ApplicationModels.SelectorModel, Microsoft.AspNetCore.Mvc.Core]]
1000 151872 0.144836426 Microsoft.AspNetCore.Mvc.ApplicationModels.PropertyModel[]
2185 139840 0.133361816 System.Linq.Enumerable+ConcatNIterator`1[[Microsoft.AspNetCore.Mvc.Filters.FilterDescriptor, Microsoft.AspNetCore.Mvc.Abstractions]]
2185 139840 0.133361816 System.Func`2[[Microsoft.AspNetCore.Mvc.ApplicationModels.PropertyModel, Microsoft.AspNetCore.Mvc.Core],[Microsoft.AspNetCore.Mvc.Abstractions.ParameterDescriptor, Microsoft.AspNetCore.Mvc.Abstractions]]
1998 127872 0.121948242 System.Comparison`1[[System.String, System.Private.CoreLib]]
2187 122472 0.116798401 System.Collections.Generic.Marker[]
2185 122360 0.116691589 System.Linq.Enumerable+Concat2Iterator`1[[Microsoft.AspNetCore.Mvc.Filters.FilterDescriptor, Microsoft.AspNetCore.Mvc.Abstractions]]
1061 122304 0.116638184 System.Reflection.RuntimePropertyInfo[]
1001 120024 0.114463806 Microsoft.Extensions.Internal.PropertyHelper[]
1013 104456 0.099617004 System.ValueTuple`2[[System.String, System.Private.CoreLib],[System.Int32, System.Private.CoreLib]][]

I believe the large size is caused mostly by the sheer amount of actions in the app (39K) and likely to a degree by some additional memory required by the router to avoid backtracking.

In short, routing uses a DFA that avoids backtracking by trading off memory for compute time (it "precomputes" all possible destinations ahead of time).

A route is "always evaluated in M steps" being "M" the number of segments of the url + the number of additional matching policies independent of the number of routes it defines.

There are strategies to reduce the amount of nodes in the tree, but we believe in general this is not a problem for the majority of apps. Apps with these many routes are not common, and even in these cases the advantage of running matching in linear time is normally worth the memory trade-off.

Once we do actual work on this issue we'll provide more details on how to structure your routes in large apps like this and other strategies you can follow to "prune" the amount of nodes in the tree.

NinjaCross commented 3 years ago

First of all, thankyou for your answer :)

In short, routing uses a DFA that avoids backtracking by trading off memory for compute time (it "precomputes" all possible destinations ahead of time).

That's indeed a great feature, but IMHO the developer should be able to decide if and when to apply or modulate this "precomputation". There are many factors to consider when balancing the resources (CPU and RAM) allocation in an application, and the fact that a single component of the system decide how to balance itself in an opaque/untunable way is a bit offsetting.

There are strategies to reduce the amount of nodes in the tree, but we believe in general this is not a problem for the majority of apps. Apps with these many routes are not common, and even in these cases the advantage of running matching in linear time is normally worth the memory trade-off.

I appreciate your technical analysis, and I see your general point... but I also think there is a flaw in this last sentence. Maybe it's just a misunderstanding from my side. If it is so, please accept my apologies in advance :)

I agree that big applications are not as common as small/average ones, but with this statement you are implicitly saying that it's ok to build small applications using ASP.NET CORE MVC, and it's not ok to use it for big applications. It's an invitation to (average/big) companies (usually creating big products) to move away from ASP.NET CORE MVC and transition to other technologies (... and that's exactly what my customers are asking me to do right now due to this unsolvable problem). I feel this is a very dangerous statement, and accepting it could cause an unwanted precedence.

davidfowl commented 3 years ago

There's big and there's big. This is huge. When we say things like "we believe in general this is not a problem for the majority of apps", it's because we haven't seen this complaint before. It doesn't mean that we don't care but it certainly lowers the priority of the issue for us.

sstraus commented 3 years ago

@davidfowl Probably many others had the same problem, especially when porting big old Net Framework applications, and decide to quit instead of discussing it here. So Net Core is just for simple and small applications? I dont think so, but your answer is pointing to a different direction...

davidfowl commented 3 years ago

If many people are having this issue they should absolutely file bugs. That's how we know the issue is affecting customers... We can't guess as to which issues people give up on, we make a judgement call based on how loud people are and how bad we think the issue is.

I just ran the project and its a really bad experience so I'm hoping we can at least look into it more to understand how feasible it is to shrink some of the data structures. That said, we don't have a good sense (at least I don't) of the types of applications that have thousands of controllers loaded at the same time. I'd love to understand how common this is in enterprise applications, we don't have any data on this other than what people share.

NinjaCross commented 3 years ago

but it certainly lowers the priority of the issue for us.

I only partially see your point.

I would like to stress on a (IMHO) 2 very significant aspects:

  1. As a professional developer, my priorities are money-driven. I don't develop software only for pleasure: I do it because I'm payed to do it. If the technology doesn't support the needs of my customer's projects, I'll simply use another technology that doesn't do assumptions on what is important based on the fact that it's rare. Even a rare problem can be show-stopper. This is exactly the case.

  2. Of course big/huge project are rare, that's how statistics works :) Nonetheless, they exists, and since they are big, they tipically involve tens or hundreds of people (developers, analysts, testers, and so on). Just because it's rare, doesn't mean it's not statistically relevant. In a money-driven world, the number of the projects impacted by the problem is meaningless on itself. To know if a problem has priority, you should count the total number of people involved into the project, not the number of the projects impacted by the problem. I know it's not possible from your POV, but I think you get the meaning of what I'm saying.

Please, don't see this as just a "rambling"... here we are debating the future usage of this technology for many big (even huge, as you defined them) projects.

That said, we don't have a good sense (at least I don't) of the types of applications that have thousands of controllers loaded at the same time. I'd love to understand how common this is in enterprise applications, we don't have any data on this other than what people share.

Applications in the following categories contains hundreds or thousands of different database entities, which are exposed through APIs.

In my experience, an average CRM contains at least 200-400 tables. An average ERP contains at least 300-600 tables Banking management applications contains many thousands tables (I've seen systems with tens of thousands of tables).

Dynamic-generation of data models and APIs is also a very real, very frequent scenario in big systems that needs for flexibility and extensibility. As you can imagine, the situation goes haywire if your system allows to "extend" native entities structures through plugins, and those "extended models" are exposed too using dynamically generated APIs.

I strongly hope the situation will be patched ASAP.

javiercn commented 3 years ago

That's indeed a great feature, but IMHO the developer should be able to decide if and when to apply or modulate this "precomputation". There are many factors to consider when balancing the resources (CPU and RAM) allocation in an application, and the fact that a single component of the system decide how to balance itself in an opaque/untunable way is a bit offsetting.

We take these things into account, and there are ways to "modulate" it. It just isn't a problem until you enter the realm of "very large" applications. To give you an idea of the sheer size of the sample app provided, it's about 10x bigger than the routing space for all Azure APIs combined (which are distributed across many services). We've validated this approach with many large applications including CMSs like Orchard and many products at Microsoft without this being a problem.

I agree that big applications are not as common as small/average ones, but with this statement you are implicitly saying that it's ok to build small applications using ASP.NET CORE MVC, and it's not ok to use it for big applications. It's an invitation to (average/big) companies (usually creating big products) to move away from ASP.NET CORE MVC and transition to other technologies (... and that's exactly what my customers are asking me to do right now due to this unsolvable problem). I feel this is a very dangerous statement, and accepting it could cause an unwanted precedence.

The problem here is "expectations", you can totally use ASP.NET Core to build internet facing hyper-scale applications, but when you reach a certain scale, you should expect that the default patterns the system use might not work for you and that you might need to reach for other tools on the toolbelt or get a deeper understanding of how the system works to ensure you optimize for it.

For an application hosting 40K endpoints we believe it's perfectly valid to allocate a couple hundred megabytes upfront to handle routing to all those endpoints. If you want more control, you can use things like MapDynamicControllerRoute which allows you define how a given route matches to a controller/action without creating individual endpoints for each action.

NinjaCross commented 3 years ago

We take these things into account, and there are ways to "modulate" it.

Very interesting ! Could you please provide some hints about how to "modulate" this behaviour ?

It just isn't a problem until you enter the realm of "very large" applications. To give you an idea of the sheer size of the sample app provided, it's about 10x bigger than the routing space for all Azure APIs combined (which are distributed across many services). We've validated this approach with many large applications including CMSs like Orchard and many products at Microsoft without this being a problem.

I'm absolutely sure you did everything you could to satisfy the majority of common scenarios :) Anyway, I'm also not sure this is a valid comparison.

The size of an API surface is strongly related to the specific functionalities implemented by the application. In our specific case, the routes are thousands because we allow our customers to dynamically define strongly-typed data entities, and expose them with strongly-typed, dynamically generated WebApi controllers.

This is just one case, anyway. As I said, there are other scenarios having thousands of statically-typed entities, having their own controllers.

The problem here is "expectations"

You are right, expectations are everything :) I need to be able to decide on my own which is the correct trade-off for my application. So, my expectation for this specific problem is just to be able to "tune" (or, worst case scenario, disable) the "precomputation" mechanisms mentioned by @davidfowl I would be totally satisfited if I could trade memory for speed, and slightly reduce the performances in order to significantly decrease the memory allocation. User's time is important (indeed !), but is also free (from my POV, of course). RAM, on the other hand, is not (especially in cloud-based environments) :)

For an application hosting 40K endpoints we believe it's perfectly valid to allocate a couple hundred megabytes upfront to handle routing to all those endpoints.

Maybe you misread. @Sbaia is talking about (3) gigabytes, not a couple hundred megabytes. My scenario is very similar to his.

If you want more control, you can use things like MapDynamicControllerRoute which allows you define how a given route matches to a controller/action without creating individual endpoints for each action.

I will give that a spin, thanks for the suggestion !

javiercn commented 3 years ago
  1. As a professional developer, my priorities are money-driven. I don't develop software only for pleasure: I do it because I'm payed to do it. If the technology doesn't support the needs of my customer's projects, I'll simply use another technology that doesn't do assumptions on what is important based on the fact that it's rare. Even a rare problem can be show-stopper. This is exactly the case.

I agree, however I don't think the sample you are providing is using the right routing configuration/options for an app this size.

At a given point in app size you need to have some understanding of the inner workings of the primitives you use and their performance characteristics. The same way you know what sorting algorithms you can use based on the set size or the collection you should use based on the operations you need to perform, you need to understand how routing works and what options are available to you.

2. Of course big/huge project are rare, that's how statistics works :) Nonetheless, they exists, and since they are big, they typically involve tens or hundreds of people (developers, analysts, testers, and so on). Just because it's rare, doesn't mean it's not statistically relevant. In a money-driven world, the number of the projects impacted by the problem is meaningless on itself. To know if a problem has priority, you should count the total number of people involved into the project, not the number of the projects impacted by the problem. I know it's not possible from your POV, but I think you get the meaning of what I'm saying.

We do care about these types of projects and we offer options and flexibility on how you configure routing to scale as much as you need to, however, there are different tools for different app sizes and different approaches you can take within routing that have different trade-offs.

Our default approach is a static route table which in our opinion offers the best trade-off for the majority of applications between consumed memory and CPU time, but we provide multiple options and extensibility points that allow you to control how the routing process works.

Applications in the following categories contains hundreds or thousands of different database entities, which are exposed through APIs.

  • ERP
  • CRM
  • Engineering calculation apps and 3D modeling (CADs and so on) apps
  • Medical treatment
  • Banking management
  • ... and so on

In my experience, an average CRM contains at least 200-400 tables. An average ERP contains at least 300-600 tables Banking management applications contains many thousands tables (I've seen systems with tens of thousands of tables).

Sure, however I believe the common practice is to break such large applications into multiple services and perform multiple levels of routing instead of hosting everything in the same process. You can't expect the same approach you follow for an app with 40 endpoints to work for an app with 40000; the scale matters. That said, it doesn't mean there are no alternative approaches for those situations.

I also believe that with the correct routing configuration, it's not problematic to handle 300-600 tables. After all, with the sample you provided with attribute routing enabled, the 40K endpoints take less than 1GB memory which is one order of magnitude above what 300-600 tables would represent in terms of number of endpoints.

NinjaCross commented 3 years ago

I also believe that with the correct routing configuration, it's not problematic to handle 300-600 tables.

I will try with MapDynamicControllerRoute as you suggested, thankyou.

javiercn commented 3 years ago

@NinjaCross I totally got people mixed up on the thread, sorry about the confusion.

The original sample that @Sbaia provided offers different configurations it can run on.

We take these things into account, and there are ways to "modulate" it.

Very interesting ! Could you please provide some hints about how to "modulate" this behaviour ?

MapDynamicControllerRoute is the approach in these cases

It just isn't a problem until you enter the realm of "very large" applications. To give you an idea of the sheer size of the sample app provided, it's about 10x bigger than the routing space for all Azure APIs combined (which are distributed across many services). We've validated this approach with many large applications including CMSs like Orchard and many products at Microsoft without this being a problem.

I'm absolutely sure you did everything you could to satisfy the majority of common scenarios :) Anyway, I'm also not sure this is a valid comparison.

It was meant to point out that we've researched this space and that our defaults are optimized for the majority of apps out there, but that we do offer other options for when the common approach is not enough.

The size of an API surface is strongly related to the specific functionalities implemented by the application. In our specific case, the routes are thousands because we allow our customers to dynamically define strongly-typed data entities, and expose them with strongly-typed, dynamically generated WebApi controllers.

This is just one case, anyway. As I said, there are other scenarios having thousands of statically-typed entities, having their own controllers.

This is exactly the type of scenario dynamic endpoints are for.

The problem here is "expectations"

You are right, expectations are everything :) I need to be able to decide on my own which is the correct trade-off for my application. So, my expectation for this specific problem is just to be able to "tune" (or, worst case scenario, disable) the "precomputation" mechanisms mentioned by @davidfowl I would be totally satisfited if I could trade memory for speed, and slightly reduce the performances in order to significantly decrease the memory allocation. User's time is important (indeed !), but is also free (from my POV, of course). RAM, on the other hand, is not (especially in cloud-based environments) :)

I agree, and if you run into trouble with the suggestion we provided, let us know and we can help, it's important for us that these types of scenarios are possible within our routing system.

For an application hosting 40K endpoints we believe it's perfectly valid to allocate a couple hundred megabytes upfront to handle routing to all those endpoints.

Maybe you misread. @Sbaia is talking about (3) gigabytes, not a couple hundred megabytes.

Yeah I got things a bit mixed up, however the sample had different configurations and the one using 3GB is not something we recommend in this situation.

My scenario is very similar to his.

If you want more control, you can use things like MapDynamicControllerRoute which allows you define how a given route matches to a controller/action without creating individual endpoints for each action.

I will give that a spin, thanks for the suggestion !

I hope this suggestion help you, and if not, please file a separate issue and we can help.

When in doubt, file a separate issue and additionally mention an existing/similar issue. That way it helps us avoid confusions.

davidfowl commented 3 years ago

We're here to help. We should also improve the docs and warn about this API for an extra large number of routes.

droyad commented 3 years ago

We have just run into this as well. Some info from our internal investigation:

We are about 25% through our conversion to ASP.NET Core (from NancyFX) and we found a ratio of 240:1 "anti-match" route to defined route. Memory usage is currently at 350MB for the routing, so end result would be 1.4GB.

A rough test we've done shows that removing all the Regex constraints drops the size of this DFA tree by ~72%.

mkArtakMSFT commented 3 years ago

This is now fixed and will be released as part of the upcoming .NET 6 RC1 release.