search

JoshClose / CsvHelper

Library to help reading and writing CSV files
http://joshclose.github.io/CsvHelper/
Other
4.65k stars 1.05k forks source link

add max buffer sizes and if should process bad field flag #2188

Open mac477 opened 10 months ago

mac477 commented 10 months ago

It is possible that buffers and field buffers can grow with no limits what leads to big arrays allocations and large memory consumption, what finally leads to OutOfMemory exception. This PR introduces properties to limit max sizes of those buffers. And beside that ProcessBadDataFields flag which makes 'bad field' not processed. Since default is true, current behaviour would not be changed. Otherwise that field will be set to array of empty char because in such flag setting we are not interested in processing bad field. In some cases it can improve performance and memory consumption. WARNING: Limiting max processed field buffer size can lead to field value truncation (if that buffer is too small to fit in field). But it works well with ShouldSkipRecord configured to skip too long fields.

Rob-Hague commented 10 months ago

A similar feature was added in #1278 (version 30) as CsvConfiguration.MaxFieldSize. Is this different?

mac477 commented 10 months ago

A similar feature was added in #1278 (version 30) as CsvConfiguration.MaxFieldSize. Is this different?

That is completely different since it lets skip too long rows/fields. MaxFieldSize would throw whole parsing if set.

BTW: Tomorrow I will push improvements to that pr's branch.

mac477 commented 10 months ago

I've just added mentioned improvement.

mac477 commented 9 months ago

@nicemanman thanks for review but you mostly reviewed things which are not related to this pull request. Please create new pull request regarding changes you propose which are not related to this branch.

nicemanman commented 9 months ago

@nicemanman thanks for review but you mostly reviewed things which are not related to this pull request. Please create new pull request regarding changes you propose which are not related to this branch.

Okay, I will, thanks