phatcher / CsvReader

Extended version of Sebastian Lorien's fast CSV Reader
MIT License
300 stars 102 forks source link

Problems without headers and Version 3.9.0 #20

Open Jubilein666 opened 8 years ago

Jubilein666 commented 8 years ago

I have a problem with the version 3.9.0.

When i read a CSV like:

CSV File:

12345678;Hello 78945612;World

C# Code:

using (CachedCsvReader csv = new CachedCsvReader(new StreamReader(csvFile, Encoding.Default), false, ';')) myDataGridView.DataSource = csv;

The following error message appears:

Field index must be included in [0, FieldCount[. Specified field index was : '0'.

But in the version 3.8.3 it works perfectly fine.

phatcher commented 8 years ago

I've attempted to reproduce this with a test in CsvReaderTests but it passes with the current code base

    [Test]
    public void CachedNoHeader()
    {
        var csv = new CachedCsvReader(new StringReader("12345678;Hello\r\n78945612;World"), false, ';');

        var dgv = new DataGridView();
        dgv.DataSource = csv;

        dgv.Refresh();
    }

Can you make it fail still?

aeallord commented 7 years ago

I'm currently seeing this issue quite frequently.

`using (var csv = new CsvReader(new StreamReader(importFile), false)) { csv.MissingFieldAction = MissingFieldAction.ReplaceByNull;

            using (var bulkCopy = new SqlBulkCopy(conClient))
            {
                bulkCopy.DestinationTableName = "[" + strDB + "_swag].[dbo].[" + targetTable + "]";
                bulkCopy.BatchSize = 5000;
                bulkCopy.BulkCopyTimeout = 1800;

                string strErrCheck = "";
                foreach (KeyValuePair<int, int> kvp in ColumnMapping)
                {
                    // Make sure that the columns are actually mapped to something
                    if (kvp.Key >= 0 && kvp.Value >= 0)
                    {
                        bulkCopy.ColumnMappings.Add(kvp.Key, kvp.Value);
                        strErrCheck += "<" + kvp.Key + ":" + kvp.Value + ">";
                    }
                }
                Console.WriteLine(strErrCheck);

                try
                {
                    Console.WriteLine("Attempted Bulk Insert");
                    bulkCopy.WriteToServer(csv); 
                    ColumnMapping.Clear();
                    return "true";
                }
                catch (Exception ee)
                {
                    ColumnMapping.Clear();
                    //return ee.Message;
                    string retMessage = ee.Message + "<br /><br />";
                    return retMessage + ee.StackTrace;
                }
            }
        }`

When the StreamReader returns the "Columns" value is equal to 0 (zero) while the field count is 23. Being columns is 0, this then throws the error

bulkcopy error an item with the same key has already been added

Of course the error is due to the zero column count. Currently this occurs intermittently, but most frequently the error has been thrown.

My work around was to revert back to the older version of Sebastien Lorion and the same function here ran without an error. I was also using version 3.9.1

phatcher commented 6 years ago

@aeallord Can you try now, I've merged a pull-request that might address this

setagana commented 6 years ago

This issue appears to still exist in verion 4.0.0, the latest available version from NuGet.

To reproduce:

[TestClass]
    public class CsvReaderTests
    {
        [TestMethod]
        public void CsvReader_WithoutHeader_ReturnsZeroColumns()
        {
            var stringReader = new StringReader("Column1Value;Column2Value;Column3Value");

            var csvReader = new CsvReader(stringReader, hasHeaders: false, delimiter: ';');

            // This test always succeeds (in contrast to CsvReader_WithHeader_ReturnsNonZeroColumns())
            Assert.AreEqual(0, csvReader.Columns.Count);
        }

        [TestMethod]
        public void CsvReader_WithoutHeader_ReturnsNonZeroFields()
        {
            var stringReader = new StringReader("Column1Value;Column2Value;Column3Value");

            var csvReader = new CsvReader(stringReader, hasHeaders: false, delimiter: ';');

            Assert.AreEqual(3, csvReader.FieldCount);
        }

        [TestMethod]
        public void CsvReader_WithHeader_ReturnsNonZeroColumns()
        {
            var stringReader = new StringReader("Column1Header;Column2Header;Column3Header\r\nColumn1Value;Column2Value;Column3Value");

            var csvReader = new CsvReader(stringReader, hasHeaders: true, delimiter: ';');
            var thing = csvReader[1];

            // Run this test and it will fail because Columns.Count == 0
            // Debug this test and inspect the csvReader before continuing and the test succeeds - Columns.Count == 3
            Assert.AreNotEqual(0, csvReader.Columns.Count);
        }

        [TestMethod]
        public void CsvReader_WithHeader_ReturnsNonZeroFields()
        {
            var stringReader = new StringReader("Column1Header;Column2Header;Column3Header\r\nColumn1Value;Column2Value;Column3Value");

            var csvReader = new CsvReader(stringReader, hasHeaders: true, delimiter: ';');

            Assert.AreEqual(3, csvReader.FieldCount);
        }
    }
phatcher commented 6 years ago

@setagana Thanks for the repo - I'll take a look