uniVocity-parsers is a suite of extremely fast and reliable parsers for Java. It provides a consistent interface for handling different file formats, and a solid framework for the development of new parsers.
I have below code sample. batchProcessed () method will be called after every batch size rows are processed from input. and it repeat untill all row are processed. then when parser.parseAll(inputReader) is called , it returns all rows data (which is processed from all batches) at once. then if l want to so some action of row process row then i have to it on all rows. again it might lead to memory issue.
My requirement is instead of returning all processed rows at once, I want return rows processed in each batch so that I can do some action(may be I want to store in database). Once action (storing in db) on rows returned from batch is complete then i want to continue process next batch. is this possible. if yes how it can be done?
sumanthyss
commented
2 years ago
Hi,
I have below code sample. batchProcessed () method will be called after every batch size rows are processed from input. and it repeat untill all row are processed. then when parser.parseAll(inputReader) is called , it returns all rows data (which is processed from all batches) at once. then if l want to so some action of row process row then i have to it on all rows. again it might lead to memory issue.
My requirement is instead of returning all processed rows at once, I want return rows processed in each batch so that I can do some action(may be I want to store in database). Once action (storing in db) on rows returned from batch is complete then i want to continue process next batch. is this possible. if yes how it can be done?
private static void testbatch() throws Exception { try { Reader inputReader = new InputStreamReader(new FileInputStream(new File("text.txt")), "UTF-8") ; BatchedColumnProcessor rowProcessor = new BatchedColumnProcessor(5) { @Override public void batchProcessed(int rowsInThisBatch) { System.out.println("batchProcessed called"); } }; CsvParserSettings settings = new CsvParserSettings(); settings.setProcessor(rowProcessor); CsvParser parser = new CsvParser(settings); List<String[]> parsedRows = parser.parseAll(inputReader); parsedRows.forEach(e->{System.out.println(Arrays.toString(e));}); } catch (IOException e) { // handle exception } }