search

adaltas / node-csv

Full featured CSV parser with simple api and tested against large datasets.
https://csv.js.org
MIT License
4.05k stars 267 forks source link

Column can not contain '.' #352

Closed levarberry closed 2 years ago

levarberry commented 2 years ago

Using csv-parse/sync Getting Error when a column has a '.' in the name.

This issue comes and goes with each release. In v4 ... 4.6.5 it broke and stop allowing '.' in column names. In 4.16.1 it worked again. Now I am on 5.2.2 it doesn't work anymore.

Is there a way to allow '.' in column names? The source file is out of my control , so I would like to keep it intact.

wdavidw commented 2 years ago

Provide a unit test or a minimalist sample as requested in the bug and feature templates.

levarberry commented 2 years ago

const parseSync = require('csv-parse/sync'); const input = 'Ref,Sample No,Analysis Date,Legionella pneumophila (total),L. pneumophila - Serogroup 1,L. pneumophila - Serogroup 2-14,Legionella L (total),Legionella species,Detection limit,Unit,COA KQL-25551,6248776,06/20/2022,<5000,<5000,<5000,<5000,<5000,<5000,CFU/L,3303861' const recordss = parseSync.parse(input, { columns: true, to_line: 2, skip_lines_with_error: true, delimiter:",", skip_empty_lines: true }); console.log(recordss)

levarberry commented 2 years ago 
(node:12612) UnhandledPromiseRejectionWarning: Error: key L. pneumophila - Serogroup 1 must not contain '.'
at serializeInto (/Volumes/edrivenhd/dubois/aqua/aqua-analytics/node_modules/mongodb/node_modules/bson/lib/bson/parser/serializer.js:913:19)
at serializeObject (/Volumes/edrivenhd/dubois/aqua/aqua-analytics/node_modules/mongodb/node_modules/bson/lib/bson/parser/serializer.js:347:18)
at serializeInto (/Volumes/edrivenhd/dubois/aqua/aqua-analytics/node_modules/mongodb/node_modules/bson/lib/bson/parser/serializer.js:727:17)
at serializeObject (/Volumes/edrivenhd/dubois/aqua/aqua-analytics/node_modules/mongodb/node_modules/bson/lib/bson/parser/serializer.js:347:18)
at serializeInto (/Volumes/edrivenhd/dubois/aqua/aqua-analytics/node_modules/mongodb/node_modules/bson/lib/bson/parser/serializer.js:937:17)
at serializeObject (/Volumes/edrivenhd/dubois/aqua/aqua-analytics/node_modules/mongodb/node_modules/bson/lib/bson/parser/serializer.js:347:18)
at serializeInto (/Volumes/edrivenhd/dubois/aqua/aqua-analytics/node_modules/mongodb/node_modules/bson/lib/bson/parser/serializer.js:727:17)
at serializeObject (/Volumes/edrivenhd/dubois/aqua/aqua-analytics/node_modules/mongodb/node_modules/bson/lib/bson/parser/serializer.js:347:18)
at serializeInto (/Volumes/edrivenhd/dubois/aqua/aqua-analytics/node_modules/mongodb/node_modules/bson/lib/bson/parser/serializer.js:937:17)
at BSON.serialize (/Volumes/edrivenhd/dubois/aqua/aqua-analytics/node_modules/mongodb/node_modules/bson/lib/bson/bson.js:64:28)
wdavidw commented 2 years ago

Here is your code, formatted properly. Running node lib/352.js from the "csv/demo/issues-cjs" with csv-parse version 5.2.2 prints:

[
  {
    Ref: 'KQL-25551',
    'Sample No': '6248776',
    'Analysis Date': '06/20/2022',
    'Legionella pneumophila (total)': '<5000',
    'L. pneumophila - Serogroup 1': '<5000',
    'L. pneumophila - Serogroup 2-14': '<5000',
    'Legionella L (total)': '<5000',
    'Legionella species': '<5000',
    'Detection limit': '<5000',
    Unit: 'CFU/L',
    COA: '3303861'
  }
]
levarberry commented 2 years ago

My apologies. After digging more into this I've found the error is actually not coming from csv-parser. I immediately store the result in Mongo (as is). The error is actually coming from Mongo. It had been working for 7 years, and the only thing I did was npm i csv-parser@5 ... so I made a bad assumption.

Atlas did a force upgrade of our cloud versions back in April and now the column names are a factor.