search

uniVocity / univocity-parsers

uniVocity-parsers is a suite of extremely fast and reliable parsers for Java. It provides a consistent interface for handling different file formats, and a solid framework for the development of new parsers.
918 stars 251 forks source link

Parsing of big CSV file #451

Closed Krzysztof-Lempicki closed 3 years ago

Krzysztof-Lempicki commented 3 years ago

Hi,

Is this lib designed to parse big csv files? (500 - 700MB)

If yes should I use it in some specific way to avoid OutOfMemory?

dmsleptsov commented 3 years ago

Hi @Krzysztof-Lempicki,

it is possible like this:

        CsvParser parser = null;
        try {
            parser = new CsvParser(settings);
            parser.beginParsing(inputStream, StandardCharsets.UTF_8);

            Record record;
            while ((record = parser.parseNextRecord()) != null) {
                //any operations with record
            }
        } finally {
            if (Objects.nonNull(parser)) {
                parser.stopParsing();
            }
        }