search

toddams / RazorLight

Template engine based on Microsoft's Razor parsing engine for .NET Core
Apache License 2.0
1.5k stars 259 forks source link

System.OutOfMemoryException #446

Closed glararan closed 3 years ago

glararan commented 3 years ago

Describe the bug Today I upgraded site from NET Core 3 to NET 5. I also moved from beta-8 to rc-3. However after few hours I am getting following exception

2021-05-17 17:24:42.930 +02:00 [ERR] An unhandled exception has occurred while executing the request.
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at RazorLight.Compilation.RoslynCompilationService.CompileAndEmit(IGeneratedRazorTemplate razorTemplate)
   at RazorLight.Compilation.RazorTemplateCompiler.CompileAndEmit(RazorLightProjectItem projectItem)
   at RazorLight.Compilation.RazorTemplateCompiler.OnCacheMissAsync(String templateKey)
--- End of stack trace from previous location ---
   at RazorLight.EngineHandler.CompileTemplateAsync(String key)
   at RazorLight.EngineHandler.CompileRenderAsync[T](String key, T model, ExpandoObject viewBag)
   at CallSite.Target(Closure , CallSite , Object )
   at EmailBuilder.BuildAsync(String template, Object model, Object viewBag)

However app is not crashing nor is system out of memory.. there is about 10GB RAM left.

To Reproduce My implementation EmailBuilder.BuildAsync:

RazorLightEngine engine = new RazorLightEngineBuilder().UseFileSystemProject(templatePath).UseMemoryCachingProvider().Build();

return await engine.CompileRenderAsync($"{template}.cshtml", model, viewBag);

App is blazor-serverside/webassembly based. Using now like 500MB RAM. Limit is 6GB. All actions works except building emails. Maybe I miss something after upgrade?

Expected behavior Exception less.

Information (please complete the following information):

jzabroski commented 3 years ago

Hi @glararan , Thanks for using RazorLight.

Normally, .NET developers use a tool like Redgate ANTS Memory Profiler to detect memory leaks or possibly places in their application where they are not freeing memory soon enough. It can profile memory up to the OOM Exception, and then you can take a final snapshot of what's been allocated in order to catch the offender. Another trick you can do to try to make the OOM happen faster is allocate a useless byte array in a commonly used object, to see if that commonly used object's class has bad memory management.

Did you try that before reporting an exception here? It's not clear to me that this OOM Exception happens on the first call, or after many calls.

App is blazor-serverside/webassembly based.

I don't think that tells me much since I have not built apps like these before (just looked at some demoes about 2 years ago). What does your csproj look like?

glararan commented 3 years ago

Welp.. its not definitely mem leak if I can tell - app is not crashing and still works, it sounds like wrong exception. To analyze memory I will have to rebuild it using debug and check later when visitors returns.

Csproj looks like

<Project Sdk="Microsoft.NET.Sdk.Web">
  <PropertyGroup>
    <TargetFramework>net5.0</TargetFramework>
    <UserSecretsId>my-secret</UserSecretsId>
  </PropertyGroup>

  <PropertyGroup>
    <TypeScriptCompileBlocked>true</TypeScriptCompileBlocked>
    <RootNamespace>MyRootNamespace</RootNamespace>
    <AssemblyName>MyAssemblyName</AssemblyName>
  </PropertyGroup>

  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|AnyCPU'">
    <OutputPath>F:\_build\...\blazor\Debug\</OutputPath>
  </PropertyGroup>

  <PropertyGroup>
    <EnableDefaultContentItems>false</EnableDefaultContentItems>
  </PropertyGroup>

  <ItemGroup>
    <!-- Publish everything under wwwroot, all JSON files, all config files and all Razor files -->
    <Content Include="wwwroot\**" ExcludeFromSingleFile="true" CopyToPublishDirectory="PreserveNewest" Exclude="$(DefaultItemExcludes);$(DefaultExcludesInProjectFolder)" />
    <Content Include="**\*.config" ExcludeFromSingleFile="true" CopyToOutputDirectory="PreserveNewest" CopyToPublishDirectory="PreserveNewest" Exclude="$(DefaultItemExcludes);$(DefaultExcludesInProjectFolder);$(DefaultWebContentItemExcludes)" />
    <Content Include="**\*.json" ExcludeFromSingleFile="true" CopyToOutputDirectory="PreserveNewest" CopyToPublishDirectory="PreserveNewest" Exclude="$(DefaultItemExcludes);$(DefaultExcludesInProjectFolder);$(DefaultWebContentItemExcludes)" />
    <Content Include="**\*.cshtml" ExcludeFromSingleFile="true" CopyToPublishDirectory="PreserveNewest" Exclude="$(DefaultItemExcludes);$(DefaultExcludesInProjectFolder);$(DefaultWebContentItemExcludes)" />
    <Content Include="**\*.razor" ExcludeFromSingleFile="true" Exclude="$(DefaultItemExcludes);$(DefaultExcludesInProjectFolder);$(DefaultWebContentItemExcludes)" />

    <!-- Set CopyToPublishDirectory to Never for items under AppDesignerFolder ("Properties", by default) to avoid publishing launchSettings.json -->
    <Content Update="$(AppDesignerFolder)\**" CopyToPublishDirectory="Never" Condition="'$(AppDesignerFolder)' != ''" />

    <!-- Remove Content items from other item types (in a way that CPS understands) -->
    <None Remove="**\*.cshtml" />
    <None Remove="**\*.razor" />
    <None Remove="wwwroot\**;**\*.json;**\*.config" />
    <Compile Remove="wwwroot\**" />
    <EmbeddedResource Remove="wwwroot\**" />
    <Compile Remove="Themes\**" />
    <Content Remove="Themes\**" />
    <Content Remove="wwwroot\uploads\**" />
    <EmbeddedResource Remove="Themes\**" />
    <EmbeddedResource Remove="wwwroot\uploads\**" />
    <None Remove="Themes\**" />
    <None Remove="wwwroot\uploads\**" />
  </ItemGroup>

  <ItemGroup>
    <_ContentIncludedByDefault Remove="Areas\Admin\Views\Features\CreateValue.cshtml" />
    <_ContentIncludedByDefault Remove="wwwroot\sass\maintenance.min.css" />
  </ItemGroup>

  <ItemGroup>
    <PackageReference Include="Blazored.LocalStorage" Version="3.0.0" />
    <PackageReference Include="BlazorPolyfill.Server" Version="5.0.100.6" />
    <PackageReference Include="ChartJs.Blazor" Version="1.1.0" />
    <PackageReference Include="CurrieTechnologies.Razor.SweetAlert2" Version="4.3.1" />
    <PackageReference Include="DotNetCore.NPOI" Version="1.2.3" />
    <PackageReference Include="Google.Apis.AnalyticsReporting.v4" Version="1.49.0.2197" />
    <PackageReference Include="Magick.NET-Q16-AnyCPU" Version="7.23.0" />
    <PackageReference Include="MailKit" Version="2.10.1" />
    <PackageReference Include="Microsoft.AspNetCore.Components.Analyzers" Version="5.0.2" />
    <PackageReference Include="Microsoft.AspNetCore.Components.WebAssembly.Server" Version="5.0.2" />
    <PackageReference Include="Microsoft.AspNetCore.Diagnostics.EntityFrameworkCore" Version="5.0.2" />
    <PackageReference Include="Microsoft.AspNetCore.Identity.EntityFrameworkCore" Version="5.0.2" />
    <PackageReference Include="Microsoft.AspNetCore.Mvc.NewtonsoftJson" Version="5.0.2" />
    <PackageReference Include="Microsoft.AspNetCore.Mvc.Razor.RuntimeCompilation" Version="5.0.2" />
    <PackageReference Include="Microsoft.AspNetCore.SignalR.Client" Version="5.0.2" />
    <PackageReference Include="Microsoft.EntityFrameworkCore" Version="5.0.2" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.Design" Version="5.0.2">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
    <PackageReference Include="Microsoft.EntityFrameworkCore.SqlServer" Version="5.0.2" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.Tools" Version="5.0.2">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
    <PackageReference Include="Microsoft.Extensions.Hosting" Version="5.0.0" />
    <PackageReference Include="Microsoft.VisualStudio.Web.CodeGeneration.Design" Version="5.0.1" />
    <PackageReference Include="ncrontab" Version="3.3.1" />
    <PackageReference Include="Newtonsoft.Json" Version="12.0.3" />
    <PackageReference Include="RazorLight" Version="2.0.0-rc.3" />
    <PackageReference Include="Serilog.AspNetCore" Version="3.4.0" />
    <PackageReference Include="SimpleMvcSitemap" Version="4.0.0" />
    <PackageReference Include="System.ServiceModel.Duplex" Version="4.8.0" />
    <PackageReference Include="System.ServiceModel.Http" Version="4.8.0" />
    <PackageReference Include="System.ServiceModel.NetTcp" Version="4.8.0" />
    <PackageReference Include="System.ServiceModel.Security" Version="4.8.0" />
    <PackageReference Include="TextFieldParserCore" Version="1.0.0" />
    <PackageReference Include="TinyCsvParser" Version="2.6.0" />
  </ItemGroup>

  <ItemGroup>
    <Folder Include="wwwroot\images\" />
    <Folder Include="wwwroot\lib\signalr\" />
  </ItemGroup>

  <ItemGroup>
    <None Include="EmailTemplates\ForgotPassword.cshtml" />
    <None Include="EmailTemplates\ResetPassword.cshtml" />
    <None Include="wwwroot\sass\vendors\pace-progress\css\pace.css.map" />
    <None Include="wwwroot\sass\vendors\pace-progress\css\pace.min.css.map" />
  </ItemGroup>

  <ItemGroup>
    <ProjectReference Include="..\Admin\Admin.csproj" />
    <ProjectReference Include="..\Shared\Shared.csproj" />
  </ItemGroup>

  <ItemGroup>
    <WCFMetadata Include="Connected Services" />
  </ItemGroup>
</Project>

Did you try that before reporting an exception here?

Actually not as this is first day relase on production to test stability. No other issues than this one. I would stay on same version as before however its not supported by .NET 5. Exception is coming out from RazorLight dll

jzabroski commented 3 years ago

Exception is coming out from RazorLight dll

Yeah, but.... that's not how you report OOM Exceptions, per say... RazorLight could just be throwing that because it's trying to populate the compilation cache with a huge template. That doesn't mean there is a bug in RazorLight- it just means the memory request is too large for .NET to fulfill. To see if that is true, you need to know how segmented your memory pool is at the time the exception is thrown. It doesn't matter if you had 500MB used, you can create an OOM Exception trivially by allocating a long.MaxValue in bytes.

If this is only happening for one user, I'd suggest you add logging and log the template name so that you know more about why this template is bombing out.

Welp.. its not definitely mem leak if I can tell

You can see we don't throw OutOfMemoryException ourselves. https://github.com/toddams/RazorLight/search?q=outofmemoryexception

glararan commented 3 years ago

I just did check for it in source code before you posted it. Truly RazorLight is not throwing it. However there was no issue before upgrade :D So thats super weird. I will check it tomorrow if issue reappears. I restarted application pool since then its okay. (250 MB ram usage now).

I have to wait now if visitors will use features including email generating. I will let you know what I found

jzabroski commented 3 years ago

So, I think the lesson learned here is, if you get an OOM Exception, you may want to try to reproduce it first - my guess is your cached template is quite larger than you think. It's conceivable someone could even exploit the template generation cache and create a denial of service attack. I'm not sure that it's RazorLight's business to defend against that. I mostly use it in applications where that is not a concern, as do others.

glararan commented 3 years ago

Closing for now as error didnt reappear after website restart.