search

Daniel1of1 / CSwiftV

A csv parser written in swift conforming to rfc4180
BSD 3-Clause "New" or "Revised" License
171 stars 46 forks source link

CSwiftV swallows empty newlines #6

Closed n13 closed 8 years ago

n13 commented 8 years ago

My data looks like this: header1, header2 "the

first line header1",first line header 2

The value of header1 is parsed into: "the first line header1"

So the empty line got removed.

I looked into it briefly, and the problem is that the line of code below kicks out all empty newlines before it goes to merge quoted strings. The code that merges quoted strings and adds newlines back in never even knows about any empty lines that may have been part of the quoted string

        let lines : [String] = includeQuotedStringInFields(Fields:string.splitOnNewLine().filter{(includeElement: String) -> Bool in
            return !includeElement.isEmpty;
        } , quotedString: "\r\n")

I tested removing that line but clearly it's there for a reason - things start breaking without it ;)

As a potential fix maybe the code that removes empty lines could be more careful to not do it when there's quotes...

Side note: The code is really slick. Super short. Kudos.

ghost commented 8 years ago

I am not sure if this should be working, but I believe by your description that it should work. I have a number of CSV files where quoted strings have the delimiter (,) in them. I believe that they should be parsed as a single string, but it appears that a new item is created for each item in the quoted string that is delimited. Here is an example. There should be 6 items, but 8 are generated.

“Fiona Kirwan, vs, Ed Collins”,2,47000000,53000000,6.4,1620

Thanks,

Jim

n13 commented 8 years ago

^ this has nothing to do with my bug report.

Quotes and commas work just fine. Maybe in your case you're using non-standard quote symbols... even from your pasted text it looks like smart quotes. Please file a separate bug on smart quotes not being recognized. I am not sure they should be recognized, to be honest... it's more something that word processors do... but either way, this is a different issue, please open your own issue for it. Thanks.

ghost commented 8 years ago

Daniel,

Sorry about misposting my message. I am a bit new to using github. Anyway, you did solve my problem. You were correct that the editor created smart quotes. I was using the editor as a means of generating a test file. The actual production file does not generate smart quotes.

I apologize again and thank you for your help.

Jim

Jim Collins jim@touchingmemories.com

DNAMatch 4 the iPad dnamatch4ipad.com

On Feb 23, 2016, at 5:57 PM, NIK notifications@github.com wrote:

^ this has nothing to do with my bug report.

Quotes and commas work just fine. Maybe in your case you're using non-standard quote symbols... even from your pasted text it looks like smart quotes. Please file a separate bug on smart quotes not being recognized. I am not sure they should be recognized, to be honest... it's more something that word processors do... but either way, this is a different issue, please open your own issue for it. Thanks.

— Reply to this email directly or view it on GitHub https://github.com/Daniel1of1/CSwiftV/issues/6#issuecomment-188008511.

Daniel1of1 commented 8 years ago

@svjim no worries at all :smile:

@n13 thanks a lot for the report, really useful, was not aware of this behaviour. I'll look into it in a little bit, does seem pretty fixable :+1: I should definitely add a test to for this too.

Daniel1of1 commented 8 years ago

@n13 suuuper late but looks like I got it as of tag 0.0.4 Added test: testThatItParsesFieldswithNewLinesInQuotes I'll close the issue but please let me know if it fails on any of your data set and I'll reopen.