mafintosh / csv-parser

Streaming csv parser inspired by binary-csv that aims to be faster than everyone else
MIT License
1.41k stars 134 forks source link

Encoded characters when reading utf16 #184

Open scolobey opened 3 years ago

scolobey commented 3 years ago

Expected Behavior

Output decoded text when reading utf16

Actual Behavior

Outputs encoded characters when reading utf16

How Do We Reproduce?

Referencing test/fixtures/bench/utf16.csv If you attempt to process this file, the result is an object containing encoded characters, while test/fixtures/bench/utf8.csv returns decoded characters.

Is it intentional for data to be returned in different forms like this? My ideal solution would be to identify and decode utf16. But a blurb in the readme to explain this or how to handle it would work.