Does not parse multiple lines in a column separated with '\n'

[niranjan92]

Consider following input:

"Id","Title","Body","Tags"^M 
"1","How to check if an uploaded file is an image without mime type?","<p>I'd like to check if an uploaded file is an image file (e.g png, jpg, jpeg, gif, bmp) or another file. The problem i    s that I'm using Uploadify to upload the files, which changes the mime type and gives a 'text/octal' or something as the mime type, no matter which file type you upload.</p>

<p>Is there a way to check if the uploaded file is an image apart from checking the file extension using PHP?</p>
","php image-processing file-upload upload mime-types"^M 

Here the Body column consists of HTML text with \n as a separator.The newline is denoted with \r\n or ^M. The attached file highlights occurance of all newline characters in the text. I can mail you the sample file in case there is any confusion.Unable to format it correctly using markdown. The data is actually stackoverflow tag prediction data from kaggle which is available openly.

The parser throws following error:

what():  Escaped string was not closed in line 2 in file "../data/pristine/Train_100.txt".
Aborted (core dumped)

I temporarily fixed this issue by a small hack in next_line() function line number 272.

    while(buffer[line_end] != '\r' && line_end != data _end){
        ++line_end;
    }
    ++line_end;//on assumption that \r is followed by \n

I am searching for \r instead of \n. This works for my case but can we have something more robust that handles such input data.

PS: I am new to github as well and this is my first issue

[niranjan92]

Any updates on this issue?

Regards,

[ben-strasser]

Hi,

this problem already appeared previously and I decided to not fix it at that time. The reasons were:

• CSV files that contain newlines within a line that do not separate lines are defective in my opinion. The first step should consist of fixing the file.
• Splitting up the lines is done in LineReader. Unescaping quotes is done in CSVReader but only if the right policies are set. LineReader knows nothing about quotes and thus can not know whether a newlines is within quotes or not. The whole quoting logic is contained within a CSVReader policy. I therefore see no straightforward way of making this "just work" without major interface breaking changes to the design.
• The most frequent use case consists of parsing CSV-files that only contain numbers and perhaps non-escaped strings. I do not want to add logic to LineReader that makes this use case slower.
• Windows vs Linux newlines, i.e., "\r\n" vs "\n", make this whole problem significantly worse. Note that, these two can appear simultaneously within one file. Sidenote: This whole Linux vs Windows is also the reason why the newline characters are hardcoded within LineReader and not part of a template parameter, as they should in an ideal world.

If you know of a good way to support such ill-formed CSV files, then I can add the change. If not, then this problem will stay.

Best Regards, Ben Strasser

On Sat, 06 Jun 2015 12:06:31 -0700 Niranjan Godbole notifications@github.com wrote:

Any updates on this issue?

Regards,

---

Reply to this email directly or view it on GitHub: https://github.com/ben-strasser/fast-cpp-csv-parser/issues/2#issuecomment-109638105