search

JoshClose / CsvHelper

Library to help reading and writing CSV files
http://joshclose.github.io/CsvHelper/
Other
4.74k stars 1.06k forks source link

MemoryLeak in combination with EF6 (ChangeTracking enabled) #2020

Open Montago opened 2 years ago

Montago commented 2 years ago

I was helping a collegue writing out a 5 GB CSV file from a EF6 entity..

We couldn't simply ToList the whole table so we created a While loop that Skipped and Take'd chunks of 500 records.

The memory allocation of the chunk was never released ! Even when NULL'ing the chunk it didn't help. GC couldn't release the memory.

The only fix was to Disable ChangeTracking

So i seams that CsvHelper is somehow causing a MemoryLeak in combination with EF6 ChangeTracking.

The code was .NET 6 - Console Application (everything newest version pulled from Nuget)

jdaigle commented 2 years ago

This doesn't really sound like a problem with CsvHelper.

Instead, your conclusion about EF6 ChangeTracking is exactly the problem: Entity Framework, by default, keeps track of every entity read from the database within the DbContext. So for every batch of 500 rows read, you're adding another 500 entities into the ChangeTracker without ever clearing or releasing them. It's not really a "memory leak" either, as it is working as designed.

But if the intent is to read to rows "read-only", then disabling ChangeTracking is exactly the right thing to do.