CsvDataReader cannot parse the characters with accents correctly

[jkiller295]

Describe the bug Given this data Côte d'Ivoire,São Tomé and Príncipe

After loading it to a DataTable object, the values become C�te d'Ivoire,S�o Tom� and Pr�ncipe

To Reproduce

using (var reader = new StreamReader("Test.csv")) //data is in this Test.csv file
   using (var csv = new CsvReader(reader, CultureInfo.InvariantCulture))
   {
       using (var dr = new CsvDataReader(csv))
       {
           var dt = new DataTable();
           dt.Load(dr);
           foreach (DataRow row in dt.Rows)
           {
               Console.WriteLine(row[1]);
           }
       }
   }

Expected behavior The text with accents should be parsed as it comes from the input file

[JoshClose]

I believe you need to set the Encoding on your StreamReader. CsvHelper knows nothing about that.

[jkiller295]

I believe you need to set the Encoding on your StreamReader. CsvHelper knows nothing about that.

Hi Josh, thanks for the prompt reply. What if the file encoding is not Unicode, does CSV Helper have a function to convert the file's Encoding (maybe a FR if it does not currently have)?

[AltruCoder]

You would need to figure out the encoding being used and set it on the StreamReader.

For example, you could try iso-8859-1.

using (var reader = new StreamReader("Test.csv", Encoding.GetEncoding("iso-8859-1")))

[jkiller295]

 var encodings = Encoding.GetEncodings()
                                    .Select(e => e.GetEncoding())
                                    .Select(e => new
                                    {
                                        Encoding = e,
                                        Preamble = e.GetPreamble()
                                    })
                                    .Where(e => e.Preamble.Any())
                                    .ToArray();

            int maxPrembleLength = encodings.Max(e => e.Preamble.Length);
            byte[] buffer = new byte[maxPrembleLength];

            using (FileStream stream = File.OpenRead(filePath))
            {
                stream.Read(buffer, 0, (int)Math.Min(maxPrembleLength, stream.Length));
            }

            return encodings
                   .Where(enc => enc.Preamble.SequenceEqual(buffer.Take(enc.Preamble.Length)))
                   .Select(enc => enc.Encoding)
                   .FirstOrDefault() ?? Encoding.Default;

I found this piece of code that can get the Encoding of a file. Would be nice if CsvHelper has a built-int function to get the CSV's encoding

[JoshClose]

It looks like there are already libraries that do this. https://github.com/errepi/ude

[babisque]

Try using this: using (var reader = new StreamReader("Test.csv", new UTF8Encoding(true)))

I had the same problem when writing to a CSV file, so I tried using encoding in the StreamWriter like this:

string filePath = $"{config.FileName}.csv";
using (var writer = new StreamWriter(filePath, false, new UTF8Encoding(true)))
using (var csv = new CsvWriter(writer, new CsvConfiguration(CultureInfo.InvariantCulture)))
{
    // Write header
    var header = CsvServices.GenerateCsvHeader(config);
    csv.WriteField(header);
    csv.NextRecord();

    // Write each line of the CSV
    foreach (var line in CsvServices.GenerateCsvLines(config))
    {
        csv.WriteField(line);
        csv.NextRecord();
    }
}

[jkiller295]

@babisque That will only work if the input file is unicode encoded. Anw, I'm closing this issue since it's not a bug on CsvHelper side