mafintosh / csv-parser

Streaming csv parser inspired by binary-csv that aims to be faster than everyone else
MIT License
1.41k stars 134 forks source link

latin1 supplement separator not working #195

Open sushovannits opened 3 years ago

sushovannits commented 3 years ago

Expected Behavior

Using "¬" as a separator in csv file should work. Output should look like:

[ { a: '1', 'b': '2' } ]

Actual Behavior

Parsing seems not correct. Producing something like this:

[ { 'a�b': '1�2' } ]

How Do We Reproduce?

Test file:

a¬b
1¬2

Code:

const csv = require('csv-parser');
const fs = require('fs');
var iconv = require('iconv-lite');
const fileName = ('/tmp/test.csv')
const results = []
fs.createReadStream(fileName)
// .pipe(iconv.decodeStream("utf-8"))
// .pipe(iconv.decodeStream("latin1"))
// .pipe(iconv.encodeStream("utf-8"))
.pipe(csv({
    separator: '¬'
}))
.on('data', (row) => {
    results.push(row)
})
.on('end', () => {
    // console.log(JSON.stringify(results, null, 4))
    console.log(results)
    console.log('CSV file successfully processed');
});