zemirco / json2csv

Convert json to csv with column titles
http://zemirco.github.io/json2csv
MIT License
2.72k stars 362 forks source link

Special characters are broken on output #528

Open MichaelA59 opened 3 years ago

MichaelA59 commented 3 years ago

json2csv - 5.0.5 Node - 10.16.0

Running into an issue where special characters are being outputted incorrectly.

Input string: Postică Outputted as: Postică

I need my file to be UTF-8 encoded (apparently doing this correctly) but also needs to account for special characters. If I export the file as a TXT file, the issue goes away, but that is not a viable solution for me. I need this exported as a CSV file.

Here's the function I built which uses the package. It accepts a JSON payload of data to compile the report.

function buildCSV(fields, data, file_name, extention = '', folder = 'Feeds') {
  const opts   = { fields };
  const parser = new Parser(opts, {withBOM: true});
  const csv    = parser.parse(data);

  let path = `./../../${extention}${folder}/Dated Backups/${file_name}.csv`;

  fs.writeFile(path, csv, function(err) {
    if (err) {
      throw err;
    }
  });
}

This is being used as a module, not CLI.

juanjoDiaz commented 3 years ago

Hi @MichaelA59 ,

What do you mean that saving as txt works but saving as csv, doesn't? Aren't you using fs.writeFile exactly the same way for both formats?

MichaelA59 commented 3 years ago

In the above code snippet, I have the path variable. I'm saying, that if I modify that so that the extension reads .txt rather than .csv like you see above, the special characters will be outputted in the txt file as they should (like this: Postică) rather than the jumbled up broken way they are outputted in the csv version. I don't know why that makes any difference though

juanjoDiaz commented 3 years ago

That's my point. json2csv produces the correct content as seen in the TXT. And I expect the fs.writeFile to also write the content equally using the default utf-8 enconding.

so the only think that I can think of is that you are opening the documents differently? How are you opening the txt? And the CSV?

mvrikxix commented 3 years ago

@juanjoDiaz I have been facing the similar issue. I am opening the txt using Notepad and the csv using Microsoft Excel. The text file is showing the special characters properly but the csv file opened in Excel is giving out a cryptic output.

juanjoDiaz commented 2 years ago

Any news on this?

I haven't been able to replicate the issue. It sounds like an issue with Excel and the character encoding selected when importing the file.

wesleysmitthe commented 2 years ago

This solution works to me: https://blog.theodo.com/2017/04/csv-excel-escape-from-the-encoding-hell-in-nodejs/

iconv.encode(csvData, "iso-8859-1")