search

dotnet / efcore

EF Core is a modern object-database mapper for .NET. It supports LINQ queries, change tracking, updates, and schema migrations.
https://docs.microsoft.com/ef/
MIT License
13.68k stars 3.17k forks source link

OutOfMemoryException when scaffolding from database since upgrade to EF Core 8 #33406

Closed GetGol closed 6 months ago

GetGol commented 6 months ago

OutOfMemoryException when scaffolding from database

The same database and the same code worked before the upgrade from .net7 to .net8 (including EF Core).

The scaffolding process starts as usual and processes some tables etc.. But after a while the process seems to hang or remain in an endless loop. Then no further output is generated. I can then see in the task manager how the process continues to consume CPU resources, whereby the RAM consumption continues to increase until it is completely used up. A few seconds later, the process then terminates with the OutOfMemoryException (as in the output at the end including the stacktrace).

I have experimented with the parameter "-tables" and have found out that the process runs successfully if I omit one of the two tables "FIFErfBuch" or "FITaKontoplanKto". But I have not yet been able to gain any insight from this.

I have also tried to debug the process. I have attached myself with VisualStudio and paused the process during the hang phase and looked into the "callstack" or "parallel stack" window. It looked like it was hanging trying to determine the data type of a database column. Then I let it run for a few seconds and paused it again. According to the callstack, it was still/again stuck on the same column. But since neither a StackOverflowException appears and more and more RAM is consumed, I assume that it is not a thread blocking problem and also not a simple recursion loop.

I am executing the command in the PackageManagerConsole. Here is the "-verbose" output (truncated for readability and body-character limit in github issue):

PM> Scaffold-DbContext "Server=getsql001;Database=master_contura312;Trusted_Connection=True;Integrated Security=True;Encrypt=False;" Microsoft.EntityFrameworkCore.SqlServer -Project GSDataModelCore -Context GetecoContext -ContextDir DbContexts\Contura -OutputDir Models\Contura\Generated -UseDatabaseNames -NoPluralize -Force -Tables FIKtoDebitorPos, FIKtoKreditorPos, FIKtoSachPos, FIKtoStamm, FITaPKtoDebitor, FITaPKtoKreditor, FITaPKtoPlan, FITnrAllgemein, STAbfragen, STAdressAnsprPartner, STAdressBanken, STAdressBeziehungen, STAdressen, STAdressInfo, STAdressKontakte, STAdressNummern, STAdressTypen, STAdrFreieFelder, STAdrFreieGruppen, STAnreden, STAnsprechpartner, STAnwArcExpFormul, STAnwArcExpKonst, STAnwArcExportSchl, STAnwArcExportSys, STAnwArcFormOrdner, STAnwArcImport, STAnwBeziehungsTypen, STAnwBezugsgruppen, STAnwBezugstexte, STAnwBlob, STAnwEmails, STAnwEmailVersand, STAnwender, STAnwKomponenten, STAnwLokationen, STAnwMailAdressen, STAnwPool, STAnwPoolDaten, STAnwPrjProtSet, STAnwPrjProtSetPos, STAnwPrjVersionen, STAnwProjekte, STAnwProtokoll, STAnwProtokollPar, STAnwProtokollPos, STAnwProtokollSkip, STAnwRptPaket, STAnwRptPaketPos, STAnwTabellen, STAnwTabellenID, STAnwTeam, STAnwTeamMitgl, STAnwTeamPos, STAnwTextBenGrp, STAnwTexte, STAnwTexteInfo, STAnwTextGrpSteu, STAnwTextGruppen, STAppBenutzerConfig, STArcDaten, STArcDatenFkt, STArcFormulare, STArcFormularSchl, STArcFormularVers, STArcImport, STArcInfo, STArcOrdner, STArcPool, STArcPoolDaten, STArcSchlVers, STArcSpeicherplatz, STBanken, STBenGrpProfile, STBenutzer, STBenutzerdaten, STBenutzereinstellungen, STBenutzerFktDaten, STBenutzergruppen, STBenutzerprojekte, STBlz, STBlzBIC, STDialogFunktionen, STDialogGruppen, STDialogTabellen, STDlgTabRelPfad, STEreignisSerie, STFelder, STFreieFelder, STFreieGrpPrgFkt, STFreieGruppen, STFunktionen, STJournal, STJournalDaten, STMeldeLogbuch, STMeldeLogGelesen, STNavigation, STNavigationSubProjekte, STPersAdressdaten, STPlz, STPrjGruppen, STPrjKomponenten, STPrjProtokolle, STPrjSpezialAbfragen, STPrjVersionen, STProgFunkErw, STProgFunkErwGrp, STProgFunkErwLanguage, STProgfunktionen, STProjekte, STRechteGrpFunk, STRechteGruppe, STRechteGruppePos, STSubProjekte, STTabellen, STTabellenPos, STTabFunktionen, STTelerikBenutzer, STTelerikBerichte, STTelerikDateien, STTexte, STAbfPrgFkt, FIFKonsPosBhnr, FIFKonsKnoten, FIUstFormCalcPos, FIUstForm, FIUstFormCalc, FIUstFormTxt, FIUstFormMatrix, FIBHStammAllgemein, FIBHStammPos, FIKtoBuchung, FIFErfAbstKreis, FIBHStatJahr, FIBHStatMonat, FISumSteuer, FIBHSteuAufteilung, FIBHSteuAuftPos, FITaSteuer, FITaSteuSchl, FITaSteuKtoplan, FITaSteuSchlPos, FITaSteuForm, FITaSteuFormPos, FITaSteuKtoPos, FITaSteuKto, FIProfil, FITaKontoplan, FITaKontoplanKto, FIFErfBuch -verbose
Using project 'Standard\Database\GSDataModelCore'.
Using startup project 'Standard\Database\DataModelScaffolding'.
Build started...
Build succeeded.
C:\Program Files\dotnet\dotnet.exe exec --depsfile E:\tfs\Sky\conviva\2.23\Test\Standard\Database\DataModelScaffolding\bin\Debug\net8.0\DataModelScaffolding.deps.json --additionalprobingpath E:\.nuget\packages --additionalprobingpath "C:\Program Files (x86)\Microsoft Visual Studio\Shared\NuGetPackages" --additionalprobingpath "C:\Program Files (x86)\Progress\ToolboxNuGetPackages" --additionalprobingpath "C:\Program Files\dotnet\sdk\NuGetFallbackFolder" --runtimeconfig E:\tfs\Sky\conviva\2.23\Test\Standard\Database\DataModelScaffolding\bin\Debug\net8.0\DataModelScaffolding.runtimeconfig.json E:\.nuget\packages\microsoft.entityframeworkcore.tools\8.0.3\tools\netcoreapp2.0\any\ef.dll dbcontext scaffold "Server=getsql001;Database=master_contura312;Trusted_Connection=True;Integrated Security=True;Encrypt=False;" Microsoft.EntityFrameworkCore.SqlServer --json --output-dir Models\Contura\Generated --context-dir DbContexts\Contura --context GetecoContext --table FIKtoDebitorPos --table FIKtoKreditorPos --table FIKtoSachPos --table FIKtoStamm --table FITaPKtoDebitor --table FITaPKtoKreditor --table FITaPKtoPlan --table FITnrAllgemein --table STAbfragen --table STAdressAnsprPartner --table STAdressBanken --table STAdressBeziehungen --table STAdressen --table STAdressInfo --table STAdressKontakte --table STAdressNummern --table STAdressTypen --table STAdrFreieFelder --table STAdrFreieGruppen --table STAnreden --table STAnsprechpartner --table STAnwArcExpFormul --table STAnwArcExpKonst --table STAnwArcExportSchl --table STAnwArcExportSys --table STAnwArcFormOrdner --table STAnwArcImport --table STAnwBeziehungsTypen --table STAnwBezugsgruppen --table STAnwBezugstexte --table STAnwBlob --table STAnwEmails --table STAnwEmailVersand --table STAnwender --table STAnwKomponenten --table STAnwLokationen --table STAnwMailAdressen --table STAnwPool --table STAnwPoolDaten --table STAnwPrjProtSet --table STAnwPrjProtSetPos --table STAnwPrjVersionen --table STAnwProjekte --table STAnwProtokoll --table STAnwProtokollPar --table STAnwProtokollPos --table STAnwProtokollSkip --table STAnwRptPaket --table STAnwRptPaketPos --table STAnwTabellen --table STAnwTabellenID --table STAnwTeam --table STAnwTeamMitgl --table STAnwTeamPos --table STAnwTextBenGrp --table STAnwTexte --table STAnwTexteInfo --table STAnwTextGrpSteu --table STAnwTextGruppen --table STAppBenutzerConfig --table STArcDaten --table STArcDatenFkt --table STArcFormulare --table STArcFormularSchl --table STArcFormularVers --table STArcImport --table STArcInfo --table STArcOrdner --table STArcPool --table STArcPoolDaten --table STArcSchlVers --table STArcSpeicherplatz --table STBanken --table STBenGrpProfile --table STBenutzer --table STBenutzerdaten --table STBenutzereinstellungen --table STBenutzerFktDaten --table STBenutzergruppen --table STBenutzerprojekte --table STBlz --table STBlzBIC --table STDialogFunktionen --table STDialogGruppen --table STDialogTabellen --table STDlgTabRelPfad --table STEreignisSerie --table STFelder --table STFreieFelder --table STFreieGrpPrgFkt --table STFreieGruppen --table STFunktionen --table STJournal --table STJournalDaten --table STMeldeLogbuch --table STMeldeLogGelesen --table STNavigation --table STNavigationSubProjekte --table STPersAdressdaten --table STPlz --table STPrjGruppen --table STPrjKomponenten --table STPrjProtokolle --table STPrjSpezialAbfragen --table STPrjVersionen --table STProgFunkErw --table STProgFunkErwGrp --table STProgFunkErwLanguage --table STProgfunktionen --table STProjekte --table STRechteGrpFunk --table STRechteGruppe --table STRechteGruppePos --table STSubProjekte --table STTabellen --table STTabellenPos --table STTabFunktionen --table STTelerikBenutzer --table STTelerikBerichte --table STTelerikDateien --table STTexte --table STAbfPrgFkt --table FIFKonsPosBhnr --table FIFKonsKnoten --table FIUstFormCalcPos --table FIUstForm --table FIUstFormCalc --table FIUstFormTxt --table FIUstFormMatrix --table FIBHStammAllgemein --table FIBHStammPos --table FIKtoBuchung --table FIFErfAbstKreis --table FIBHStatJahr --table FIBHStatMonat --table FISumSteuer --table FIBHSteuAufteilung --table FIBHSteuAuftPos --table FITaSteuer --table FITaSteuSchl --table FITaSteuKtoplan --table FITaSteuSchlPos --table FITaSteuForm --table FITaSteuFormPos --table FITaSteuKtoPos --table FITaSteuKto --table FIProfil --table FITaKontoplan --table FITaKontoplanKto --table FIFErfBuch --use-database-names --force --no-pluralize --verbose --no-color --prefix-output --assembly E:\tfs\Sky\conviva\2.23\Test\Standard\Database\DataModelScaffolding\bin\Debug\net8.0\Geteco.Standard.DataModelCore.dll --project E:\tfs\Sky\conviva\2.23\Test\Standard\Database\DataModelCore\GSDataModelCore.csproj --startup-assembly E:\tfs\Sky\conviva\2.23\Test\Standard\Database\DataModelScaffolding\bin\Debug\net8.0\DataModelScaffolding.dll --startup-project E:\tfs\Sky\conviva\2.23\Test\Standard\Database\DataModelScaffolding\DataModelScaffolding.csproj --project-dir E:\tfs\Sky\conviva\2.23\Test\Standard\Database\DataModelCore\ --language C# --configuration Debug --working-dir E:\tfs\Sky\conviva\2.23\Test\App --root-namespace Geteco.Standard.DataModelCore --nullable
Using assembly 'Geteco.Standard.DataModelCore'.
Using startup assembly 'DataModelScaffolding'.
Using application base 'E:\tfs\Sky\conviva\2.23\Test\Standard\Database\DataModelScaffolding\bin\Debug\net8.0'.
Using working directory 'E:\tfs\Sky\conviva\2.23\Test\Standard\Database\DataModelScaffolding'.
Using root namespace 'Geteco.Standard.DataModelCore'.
Using project directory 'E:\tfs\Sky\conviva\2.23\Test\Standard\Database\DataModelCore\'.
Remaining arguments: .
Finding design-time services referenced by assembly 'DataModelScaffolding'...
Finding design-time services referenced by assembly 'Geteco.Standard.DataModelCore'...
No referenced design-time services were found.
Finding design-time services for provider 'Microsoft.EntityFrameworkCore.SqlServer'...
Using design-time services from provider 'Microsoft.EntityFrameworkCore.SqlServer'.
Finding IDesignTimeServices implementations in assembly 'DataModelScaffolding'...
Using design-time services from class 'ScaffoldingDesignTimeServices'.
'DataModelScaffolding.ScaffoldingDesignTimeServices.ConfigureDesignTimeServices' invoked.
'DataModelScaffolding.ScaffoldingDesignTimeServices.ConfigureDesignTimeServiceFor_ScaffoldDbContext' started.
'DataModelScaffolding.ScaffoldingDesignTimeServices.ConfigureDesignTimeServiceFor_ScaffoldDbContext' finished.
To protect potentially sensitive information in your connection string, you should move it out of source code. You can avoid scaffolding the connection string by using the Name= syntax to read it from configuration - see https://go.microsoft.com/fwlink/?linkid=2131148. For more guidance on storing connection strings, see https://go.microsoft.com/fwlink/?LinkId=723263.
Found default schema 'DBUSER'.
Found type alias with name 'sys.sysname' which maps to underlying data type nvarchar(128).
Found table with name 'DBUSER.FIBHStammAllgemein'.
Found table with name 'DBUSER.FIBHStammPos'.
...
Found column with table: DBUSER.FIBHStammAllgemein, column name: AnwenderNr, ordinal: 1, data type: sys.int, maximum length: 4, precision: 10, scale: 0, nullable: False, identity: False, default value: (null), computed value: (null), computed value is stored: False.
Found column with table: DBUSER.FIBHStammAllgemein, column name: TeilnNr, ordinal: 2, data type: sys.int, maximum length: 4, precision: 10, scale: 0, nullable: False, identity: False, default value: (null), computed value: (null), computed value is stored: False.
...
Found primary key on table 'PK__FIBHStam__8D4316A784395C2C' with name 'DBUSER.FIBHStammAllgemein'.
Found primary key on table 'PK__FIBHStam__5E71E8F947B843E8' with name 'DBUSER.FIBHStammPos'.
...
Found foreign key on table 'FIProfil_ProfilNr_R1111_1114' with name 'DBUSER.FIBHStammAllgemein', principal table 'DBUSER.FIProfil', delete action NO_ACTION.
Found foreign key on table 'FITnrAllgemein_TeilnNr_R1111_1100' with name 'DBUSER.FIBHStammAllgemein', principal table 'DBUSER.FITnrAllgemein', delete action NO_ACTION.
...
Found foreign key on table 'STTelerikBerichte_BerichtNr_R155_149' with name 'DBUSER.STTelerikDateien', principal table 'DBUSER.STTelerikBerichte', delete action NO_ACTION.
Found foreign key on table 'STTextgruppen_TEXTGRP_R11_3' with name 'DBUSER.STTexte', principal table 'DBUSER.STTextgruppen', delete action NO_ACTION.
Skipping foreign key 'STTextgruppen_TEXTGRP_R11_3' on table 'DBUSER.STTexte' since principal table 'DBUSER.STTextgruppen' was not found in the model. This usually happens when the principal table was not included in the selection set.
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
   at System.Collections.Generic.Queue`1.SetCapacity(Int32 capacity)
   at Microsoft.EntityFrameworkCore.Metadata.Internal.Property.GetConversion(Boolean throwOnValueConverterConflict, Boolean throwOnProviderClrTypeConflict)
   at Microsoft.EntityFrameworkCore.Metadata.Internal.Property.GetValueConverter()
   at Microsoft.EntityFrameworkCore.Storage.TypeMappingInfo..ctor(IReadOnlyList`1 principals, Nullable`1 fallbackUnicode, Nullable`1 fallbackSize, Nullable`1 fallbackPrecision, Nullable`1 fallbackScale)
   at Microsoft.EntityFrameworkCore.Storage.RelationalTypeMappingInfo..ctor(IReadOnlyList`1 principals, String storeTypeName, String storeTypeNameBase, Nullable`1 fallbackUnicode, Nullable`1 fallbackFixedLength, Nullable`1 fallbackSize, Nullable`1 fallbackPrecision, Nullable`1 fallbackScale)
   at Microsoft.EntityFrameworkCore.Storage.RelationalTypeMappingSource.FindMapping(IProperty property)
   at Microsoft.EntityFrameworkCore.Metadata.Internal.Property.<>c.<get_TypeMapping>b__93_0(IProperty property)
   at Microsoft.EntityFrameworkCore.Internal.NonCapturingLazyInitializer.EnsureInitialized[TParam,TValue](TValue& target, TParam param, Func`2 valueFactory)
   at Microsoft.EntityFrameworkCore.Metadata.Internal.Property.get_TypeMapping()
   at Microsoft.EntityFrameworkCore.Metadata.Internal.Property.Microsoft.EntityFrameworkCore.Metadata.IReadOnlyProperty.FindTypeMapping()
   at Microsoft.EntityFrameworkCore.RelationalPropertyExtensions.FindRelationalTypeMapping(IReadOnlyProperty property)
   at Microsoft.EntityFrameworkCore.RelationalPropertyExtensions.GetColumnType(IReadOnlyProperty property)
   at Microsoft.EntityFrameworkCore.RelationalPropertyExtensions.GetColumnType(IProperty property)
   at Microsoft.EntityFrameworkCore.Metadata.Internal.RelationalModel.CreateDefaultColumnMapping(ITypeBase typeBase, ITypeBase mappedType, TableBase defaultTable, TableMappingBase`1 tableMapping, Boolean isTph, Boolean isTpc)
   at Microsoft.EntityFrameworkCore.Metadata.Internal.RelationalModel.AddDefaultMappings(RelationalModel databaseModel, IEntityType entityType, IRelationalTypeMappingSource relationalTypeMappingSource)
   at Microsoft.EntityFrameworkCore.Metadata.Internal.RelationalModel.Create(IModel model, IRelationalAnnotationProvider relationalAnnotationProvider, IRelationalTypeMappingSource relationalTypeMappingSource, Boolean designTime)
   at Microsoft.EntityFrameworkCore.Metadata.Internal.RelationalModel.Add(IModel model, IRelationalAnnotationProvider relationalAnnotationProvider, IRelationalTypeMappingSource relationalTypeMappingSource, Boolean designTime)
   at Microsoft.EntityFrameworkCore.Infrastructure.RelationalModelRuntimeInitializer.InitializeModel(IModel model, Boolean designTime, Boolean prevalidation)
   at Microsoft.EntityFrameworkCore.Infrastructure.ModelRuntimeInitializer.Initialize(IModel model, Boolean designTime, IDiagnosticsLogger`1 validationLogger)
   at Microsoft.EntityFrameworkCore.Scaffolding.Internal.RelationalScaffoldingModelFactory.Create(DatabaseModel databaseModel, ModelReverseEngineerOptions options)
   at Microsoft.EntityFrameworkCore.Scaffolding.Internal.ReverseEngineerScaffolder.ScaffoldModel(String connectionString, DatabaseModelFactoryOptions databaseOptions, ModelReverseEngineerOptions modelOptions, ModelCodeGenerationOptions codeOptions)
   at Microsoft.EntityFrameworkCore.Design.Internal.DatabaseOperations.ScaffoldContext(String provider, String connectionString, String outputDir, String outputContextDir, String dbContextClassName, IEnumerable`1 schemas, IEnumerable`1 tables, String modelNamespace, String contextNamespace, Boolean useDataAnnotations, Boolean overwriteFiles, Boolean useDatabaseNames, Boolean suppressOnConfiguring, Boolean noPluralize)
   at Microsoft.EntityFrameworkCore.Design.OperationExecutor.ScaffoldContextImpl(String provider, String connectionString, String outputDir, String outputDbContextDir, String dbContextClassName, IEnumerable`1 schemaFilters, IEnumerable`1 tableFilters, String modelNamespace, String contextNamespace, Boolean useDataAnnotations, Boolean overwriteFiles, Boolean useDatabaseNames, Boolean suppressOnConfiguring, Boolean noPluralize)
   at Microsoft.EntityFrameworkCore.Design.OperationExecutor.ScaffoldContext.<>c__DisplayClass0_0.<.ctor>b__0()
   at Microsoft.EntityFrameworkCore.Design.OperationExecutor.OperationBase.<>c__DisplayClass3_0`1.<Execute>b__0()
   at Microsoft.EntityFrameworkCore.Design.OperationExecutor.OperationBase.Execute(Action action)
Exception of type 'System.OutOfMemoryException' was thrown.
PM>

Include provider and version information

EF Core version: 8.0.3 Database provider: Microsoft.EntityFrameworkCore.SqlServer Target framework: .NET 8.0 Operating system: Windows 11 Enterprise IDE: Visual Studio 2022 17.10.0 Preview 2.0

Untruncated output in attachment: full output.txt

roji commented 6 months ago

Duplicate of #33176

roji commented 6 months ago

Can you please try the workaround detailed in https://github.com/dotnet/efcore/issues/33176#issuecomment-1969668284

GetGol commented 6 months ago

Can you please try the workaround detailed in #33176 (comment)

Added the mentioned switch into my DesignTimeService => it executed successfully.

Is this switch needed permanently or can I track a version or an issue to remove it from my code later?

roji commented 6 months ago

@GetGol thanks for confirming. The fix has already been merged for 8.0.4, the tracking issue is #33176.