Close stream after validation to avoid exhausting memory
Problem
When parsing a large CSV that had errors in header rows, the file stream would remain open after the parser had closed. This could cause node to exhaust available heap memory.
Solution
Close the stream after the validation function returns.
Result
Node no longer runs out of heap memory when validating the large CSV file.
Test Plan
Test using CSV MRF that is sufficiently large: the original problem file was 7.3gb, to give a sense of the size to aim for. Before the change, this caused a memory error to occur after validation completed, but before the CLI exited. After the change, no memory error occurred. You may need a different file size based on your local node memory configuration.
Close stream after validation to avoid exhausting memory
Problem
When parsing a large CSV that had errors in header rows, the file stream would remain open after the parser had closed. This could cause node to exhaust available heap memory.
Solution
Close the stream after the validation function returns.
Result
Node no longer runs out of heap memory when validating the large CSV file.
Test Plan
Test using CSV MRF that is sufficiently large: the original problem file was 7.3gb, to give a sense of the size to aim for. Before the change, this caused a memory error to occur after validation completed, but before the CLI exited. After the change, no memory error occurred. You may need a different file size based on your local node memory configuration.