jvalue / jayvee

Jayvee is a domain-specific language and runtime for automated processing of data pipelines
https://jvalue.github.io/jayvee/
150 stars 15 forks source link

[BUG] `LocalFileExtractor` emits wrong error messag if the input file is too big #609

Closed TungstnBallon closed 2 months ago

TungstnBallon commented 3 months ago

Steps to reproduce

  1. Execute this model (big_file.csv is larger than 2GB):

    pipeline Pipeline {
    
    Extractor
        -> ToTextFile
        -> ToCSV
        -> ToTable
        -> Loader;
    
    block Extractor oftype LocalFileExtractor {
        filePath: "./big_file.csv";
    }
    
    block ToTextFile oftype TextFileInterpreter { }
    
    block ToCSV oftype CSVInterpreter {
        enclosing: '"';
    }
    
    block ToTable oftype TableInterpreter {
        header: true;
        columns: [
            "C1" oftype integer,
            "C2" oftype text,
            "C3" oftype boolean,
        ];
    }
    
    block Loader oftype SQLiteLoader {
        table: "Data";
        file: "./Data.sqlite";
    }
    }

Description