[BUG] JavaScript heap out of memory

[kreisligaspieler]

Steps to reproduce

Use following pipeline:

pipeline TestPipeline {
Extractor -> TextFileInterpreter->RangeSelector->CSVInterpreter->ColumnDeleter->TableInterpreter->Loader;
block Extractor oftype HttpExtractor {
            url: "https://derwebmaster.eu/wp-content/uploads/2024/02/star0000-1.csv";}
block TextFileInterpreter oftype TextFileInterpreter {encoding: "utf8";}
block RangeSelector oftype TextRangeSelector {lineFrom: 1;}
block CSVInterpreter oftype CSVInterpreter {enclosing: '';delimiter: ",";}
block ColumnDeleter oftype ColumnDeleter {delete: [];}
block TableInterpreter oftype TableInterpreter {
            header: false;
            columns: ["col1" oftype integer,"col2" oftype integer,"col3" oftype integer,"col4" oftype integer,"col5" oftype integer,"col6" oftype decimal,"col7" oftype integer,"col8" oftype integer,"col9" oftype decimal,"col10" oftype integer,"col11" oftype decimal,"col12" oftype integer];}
block Loader oftype SQLiteLoader {table: "tDbNEGvkAEdn"; file: "test.sqlite";}
}

Original source of the csv: https://sdm.lbl.gov/fastbit/data/samples.html

Description

• Expected: Pipeline should download the file and store the content in a database
• Actual: FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory

Debug: [TestPipeline] Overview: Blocks (7 blocks with 1 pipes): -> Extractor (HttpExtractor) -> TextFileInterpreter (TextFileInterpreter) -> RangeSelector (TextRangeSelector) -> CSVInterpreter (CSVInterpreter) -> ColumnDeleter (ColumnDeleter) -> TableInterpreter (TableInterpreter) -> Loader (SQLiteLoader)

[Extractor] Fetching raw data from https://derwebmaster.eu/wp-content/uploads/2024/02/star0000-1.csv

Console output: <--- Last few GCs --->

[99110:0x65186f0] 716409 ms: Mark-Compact (reduce) 2047.2 (2078.4) -> 2047.2 (2079.1) MB, 29.36 / 0.00 ms (+ 602.9 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 999 ms) (average mu = 0.992, current mu = 0.97[99110:0x65186f0] 718079 ms: Mark-Compact (reduce) 2048.2 (2079.1) -> 2048.2 (2080.1) MB, 1666.92 / 0.00 ms (average mu = 0.689, current mu = 0.002) allocation failure; scavenge might not succeed

<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory ----- Native stack trace -----

1: 0xcc08f6 node::OOMErrorHandler(char const, v8::OOMDetails const&) [node] 2: 0x1054130 v8::Utils::ReportOOMFailure(v8::internal::Isolate, char const, v8::OOMDetails const&) [node] 3: 0x1054417 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate, char const, v8::OOMDetails const&) [node] 4: 0x1273655 [node] 5: 0x128a178 [node] 6: 0x126167e v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node] 7: 0x1262964 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node] 8: 0x123f586 v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [node] 9: 0x12310e4 v8::internal::FactoryBase::AllocateRawWithImmortalMap(int, v8::internal::AllocationType, v8::internal::Tagged, v8::internal::AllocationAlignment) [node] 10: 0x12329ae v8::internal::FactoryBase::NewRawOneByteString(int, v8::internal::AllocationType) [node] 11: 0x1232a26 v8::internal::FactoryBase::NewStringFromOneByte(v8::base::Vector, v8::internal::AllocationType) [node] 12: 0x15762fa v8::internal::ValueDeserializer::ReadObjectInternal() [node] 13: 0x15766ef v8::internal::ValueDeserializer::ReadObject() [node] 14: 0x1575d1c v8::internal::ValueDeserializer::ReadDenseJSArray() [node] 15: 0x157618e v8::internal::ValueDeserializer::ReadObjectInternal() [node] 16: 0x15766ef v8::internal::ValueDeserializer::ReadObject() [node] 17: 0x1575d1c v8::internal::ValueDeserializer::ReadDenseJSArray() [node] 18: 0x157618e v8::internal::ValueDeserializer::ReadObjectInternal() [node] 19: 0x15766ef v8::internal::ValueDeserializer::ReadObject() [node] 20: 0x157682b v8::internal::ValueDeserializer::ReadObjectWrapper() [node] 21: 0x10772f9 v8::ValueDeserializer::ReadValue(v8::Local) [node] 22: 0xe4761b node::worker::Message::Deserialize(node::Environment, v8::Local, v8::Local*) [node] 23: 0xe4b327 [node] 24: 0x7ff875dceadd Abgebrochen (Speicherabzug geschrieben)

[rhazn]

(For reference, the file is 185mb large)

Thanks for reporting this :).