OutOfMemory Exception

[georgebarbu]

Hi, when using the new ChunkDownloader with default 4 threads on a large dataset I get an OutOfMemoryException in GetChunkContentAsync. The underlying JsonSerializer blows up when trying to create a List If I set it to 1 thread no issues. Same request.

[colgreen]

Do you have any more details for this? E.g. a call stack trace? Do you know the approximate size of the data you were trying to retrieve? Thanks.

[colgreen]

Can you remember if your dataset had large binary columns in it? And if so do you an approximate size of the largest binary results? Thanks.

[georgebarbu]

Hi, the issue was the large payload, the result set was above 45M rows no binary data just varchar, timestamp and number. The OOM exception was triggered internally in the JsonSerializer more specifically when trying to new up a List in the ChunkDownloader class. The problem has been made worse by the new parallel chunk downloader as is using multiple threads increasing memory pressure. If I set the chunk downloader to 1 thread then it worked fine but obviously is much slower. I've changed my implementation to pagination to avoid massive datasets.

[fixer-m]

@georgebarbu Hi,

I'm gonna release new version with @colgreen great improvements in memory allocations - this might help.

[georgebarbu]

Sounds great, I'll leave a feedback after I test the new release

[georgebarbu]

The update looks fine