GoogleCodeExporter
commented
8 years ago Could you provide a data sample stub that causes the failure.
From a technical standpoint, even empty lines should contain delimiters to
represent the number of columns in the data.
To quote RFC 4180:
4. Within the header and each record, there may be one or more
fields, separated by commas. Each line should contain the same
number of fields throughout the file. Spaces are considered part
of a field and should not be ignored. The last field in the
record must not be followed by a comma. For example:
aaa,bbb,ccc
Only the last line is allowed to be completely empty in a valid CSV file and
the parser should handle it properly if one exists.
So an empty line with 3 columns would still show up as:
,,
Basically, a CSV file must have the same number of columns in every row to be
considered valid. Even if the trailing values on each row are consistently
empty.
There are two potential solutions to this problem...
First, you could pre-process the line data. A hook called onParseValue() exists
to allow users to insert an arbitrary function to do additional processing on
CSV values. To give users better fine-tuned control over the parser I was
planning on including another called onParseEntry() that does the same but on a
full line of data.
With that in place you will be able to insert a function that adds in the
correct number of delimiters (ex commas).
The second solution works the same as you have already done. Do a full pass on
the data first to remove empty lines. I'll be adding two hooks, onPreParse()
onPostParse() to do the same.
There are two schools of thought on this. Those who want to liberate the data
by making the parser more versatile and those who think the users should be
held responsible for providing 'clean' data.
To see a bad example of the too-liberal approach, just look at pre-HTML 4.01
browsers and 'quirks mode'. Too much variability in the data requires a lot of
'magic' (read 'hacks') to exist below the surface.
CSV already a very liberal format that allows a lot of funky edge cases that
need to be covered in the parser. My main overarching goal is to provide a
parser that specializes in CSV data and not necessarily anything beyond CSV.
There are a LOT of use cases where users want to have the ability to extend the
parser to do more interesting things. I don't want to disregard that request
altogether so my approach is to provide hooks. By doing so users have the
ability insert custom code inline during the different phases of the parsing
process and add their own domain-specific magic where it's needed.
Let me know what you think...Original comment by evanpla...@gmail.com on 10 Oct 2012 at 10:06
Original issue reported on code.google.com by
Chad.R.B...@gmail.comon 10 Oct 2012 at 8:42