osiegmar
commented
2 weeks ago I read this issue as well as your post at https://github.com/osiegmar/FastCSV/discussions/117 multiple times.
A running test case that demonstrates your use case, including the problem you are experiencing, would be very helpful.
As you pass a custom Writer instance to FastCSV and also operate directly on this writer instance, the bare minimum is to disable FastCSV's internal buffering:
CsvWriter.builder()
.bufferSize(0)
.build(yourWriterInstance);Maybe this is already sufficient to solve your problem.
I have to admit that when I developed FastCSV, I did not consider the use case you are describing (continuing to write after an I/O exception occurred).
For demonstration purposes, I created a rolling file example at ExampleCsvWriterWithRollingFile.java that demonstrates how to create CSV files with a maximum number of records or a maximum file size. Although this example uses a very different approach, that may not be suitable for your use case.
Everything else would be speculation without a running test case.
javafanboy
I am using a custom Writer class with a fixed size buffer throwing an exception when the buffer size is reached. This writer also supports mark/reset (as per how it works in i/o streams) and also rewind (reseting the write position of the buffer allowing it to be reused). I am writing to this Writer using FastCSV.
Before each writeRow call to FastCSV I call "mark" on my custom Writer and if an exception is thrown I perform a "reset", read the buffer (and send it over a communication link with a fixed max message size that I want to get as close to as possible), "rewind" the custom writer and re-drive the failed "writeRow" call and then continue writing more CSV records.
What I have observed is that the exception propagates through FastCSV as documented (UncheckedIOException), I will then as mentioned perform a reset, read and rewind on the underlying Writer but when I re-drive the failed writeRecord the CSV line is produced TWICE (once that seem to be buffered in the FastCSV Writer and once for the record I re-drive).
Not sure if this is a feature (assuming that FastCSV always buffers the WHOLE generated CSV record I can stop re-driving) but to me it seems more intuitive that FastCSV instead would clear its internal buffer if an exception is thown entering into a "clean state" allowing the client to decide if the data that caused the exception should be re-driven or for instance just logged and be discarded...
Right now I, as a work-around, I create a new FastCSV Writer each time the exception is thrown but this, even if it only happens every time a buffer is full, results in an unnecessary overhead.