csv2json not parsing commas correctly!

jvetyska commented 5 years ago

Background Information

Module Version: 3.4.1
Node/Browser Version: 8.12

The issue I'm reporting is with:

[ ] json2csv
[x] csv2json

I have...

[ x ] searched to see if an issue has already been reported.
[ x ] verified that my JSON/CSV data is valid (using something like http://jsonlint.com or https://csvlint.io/).
[ x ] tried upgrading to the latest version of json-2-csv (since the issue may already be fixed).

Expected Behavior

csv is parsed correctly and values should be extracted between commas. Thus comma is not a value itself.

Actual Behavior

a comma is actually parsed as a value which renders the result completely useless.

Data Sample

CSV:

Retail_Zone_ID,Department Name,Class Name,Keycode,Product Name,BEL1-STR,MIL1-STR,LYN1-STR,ONLINE-STR,Blacklist Start,Blacklist End 
Z02,KITCHEN,FOOD PREPARATION,42613855,Small Mango Board,X,X,X,X,,
Z05,BED LINEN,COMFORTERS,,,X,X,,,,
Z05,BED LINEN,QUILT COVER SETS,,,X,X,,,,
Z05,BED LINEN,SHEETING,,,X,X,,,,
Z05,BEDROOM ACC,PILLOWS,,,X,X,,,,
Z05,BEDROOM ACC,PROTECTORS,,,X,X,,,3|11|2019,4/11/2019
Z05,BEDROOM ACC,QUILTS,,,X,X,,,,
Z05,BEDROOM ACC,UNDERBLKT|UNDERLAY,,,X,X,,,,
Z05,WHOLESALE GM,,,,X,X,,,,

Which results into the following JSON value (notice the Blacklist Start = "," and Blacklist End = undefined

[ { Retail_Zone_ID: 'Z02',
    'Department Name': 'KITCHEN',
    'Class Name': 'FOOD PREPARATION',
    Keycode: 42613855,
    'Product Name': 'Small Mango Board',
    'BEL1-STR': 'X',
    'MIL1-STR': 'X',
    'LYN1-STR': 'X',
    'ONLINE-STR': 'X',
    'Blacklist Start': ',',
    'Blacklist End ': undefined },
  { Retail_Zone_ID: 'Z05',
    'Department Name': 'BED LINEN',
    'Class Name': 'COMFORTERS',
    Keycode: '',
    'Product Name': '',
    'BEL1-STR': 'X',
    'MIL1-STR': 'X',
    'LYN1-STR': '',
    'ONLINE-STR': '',
    'Blacklist Start': ',',
    'Blacklist End ': undefined },
...]

Code Example

// Please include a simple example to replicate the issue
let converter = require('json-2-csv');
let result = await converter.csv2jsonAsync(input);
console.log(result);

jvetyska commented 5 years ago

Note: I've used other json2csv modules and they work great but some of their performance is really poor. So I have successfully switched to json-2-csv for the json -> csv portion, which converts in 1-2 seconds vs another module took 10 seconds. But unfortunately, the other direction csv -> json is not working!

mrodrig commented 5 years ago

Hi @jvetyska. Thanks for reporting this! That's definitely not what I would expect for JSON output so something is definitely going wrong there. I'm going to take a look at it in a little bit. If what I think is happening is truly the cause, then I think it should be a pretty straightforward fix. Hopefully I'll be able to fix it and release a new version tonight so that it isn't holding you up. I'll keep you posted on the progress that I'm able to make on it tonight.

Happy to hear that it's outperforming out modules out there too!

Thanks again!

mrodrig commented 5 years ago

There ended up being two separate bugs that your example found, so thank you for the great example CSV! Unfortunately I wasn't able to fix the second one tonight, but I understand what the issue is. I'm going to keep working on it after work tomorrow. Once the remaining bug is fixed, I'll add a few tests to cover the areas where the bugs appeared to ensure they don't reappear down the road.

Sorry for not being able to get a fixed version out tonight.

jvetyska commented 5 years ago

No big deal! Thanks for addressing the issues so quickly!

On Wed, Mar 20, 2019 at 8:02 PM Michael Rodrigues notifications@github.com wrote:

There ended up being two separate bugs that your example found, so thank you for the great example CSV! Unfortunately I wasn't able to fix the second one tonight, but I understand what the issue is. I'm going to keep working on it after work tomorrow. Once the remaining bug is fixed, I'll add a few tests to cover the areas where the bugs appeared to ensure they don't reappear down the road.

Sorry for not being able to get a fixed version out tonight.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mrodrig/json-2-csv/issues/109#issuecomment-475099569, or mute the thread https://github.com/notifications/unsubscribe-auth/ABr0QBsTxp6SOHNxTG1lYFg-KPr9iIdJks5vYvZjgaJpZM4cABFR .

--

Jiri.

mrodrig commented 5 years ago

No problem! I was able to fix the second bug tonight, but would like to add a few tests before merging the fixes in to make sure that they don't reappear at some point. I should be able to get those added in tomorrow and then I'll publish a new release to NPM. I'll let you know once the new version with the fixes is published.

Sorry for the delay!

mrodrig commented 5 years ago

The bug fixes are now published to NPM in 3.4.2. Please feel free to let me know if you find any other issues.

Thanks again for reporting this!

mrodrig / json-2-csv