File size limit?

[danspencer]

Is there a file size limit when trying to seed from a CSV file? I'm trying to seed data for various models from multiple CSV files of various size some that are 3+MB. The data from the smaller CSV files seed the database properly. However, the larger CSV files cause an error to be thrown. If I cut down the file size, the data gets seeded in the database.

Running Seed method.
System.Runtime.Serialization.SerializationException: Type 'CsvHelper.CsvReaderException' in assembly 'CsvHelper, Version=2.0.0.0, Culture=neutral, PublicKeyToken=8c4959082be5c823' is not marked as serializable.
   at System.AppDomain.DoCallBack(CrossAppDomainDelegate callBackDelegate)
   at System.Data.Entity.Migrations.Design.ToolingFacade.Run(BaseRunner runner)
   at System.Data.Entity.Migrations.Design.ToolingFacade.Update(String targetMigration, Boolean force)
   at System.Data.Entity.Migrations.UpdateDatabaseCommand.<>c__DisplayClass2.<.ctor>b__0()
   at System.Data.Entity.Migrations.MigrationsDomainCommand.Execute(Action command)
Type 'CsvHelper.CsvReaderException' in assembly 'CsvHelper, Version=2.0.0.0, Culture=neutral, PublicKeyToken=8c4959082be5c823' is not marked as serializable.

Here's where I try seeding the data:

protected override void Seed(DbContext context)
{
    string path;
    path = AppDomain.CurrentDomain.GetData("DataDirectory") + "/Seeds/Address.csv";
    context.Addresses.SeedFromFile(path, address => address.ID);
    path = AppDomain.CurrentDomain.GetData("DataDirectory") + "/Seeds/Appearance.csv";
    context.Appearances.SeedFromFile(path, appearance => appearance.ID);
}

Has anyone else had this issue before? Is there any way to increase the file size that Seeder can use?

[dpaquette]

I am not aware of any file size limits. 3MB isn't really that big so we should be able to handle that. I will run some tests to see if there is a limit. Any chance you can share your data set or is the data private?

It looks like there error is being thrown by the underlying CsvHelper and we are not getting the full message. I will create a separate issue to improve the error messaging so it is easier to diagnose problems when they do happen.

btw, I like your profile picture. I'm pretty sure I own that hat.

[danspencer]

Hi Dave, what email address would you like me to send you a copy of one the data sets?

Thanks for the compliment on the profile pic. However, I can't take credit for it, as it came from a Jays blog that I used to read all the time. I always found the picture funny. ;-)

[dpaquette]

contactme@davepaquette.com

[danspencer]

CSV file sent via email.

[dpaquette]

I tested the file you sent me and it worked on the first try so I don't think the error is related to file size. It did take a very long time to complete though. It might have taken as long as an hour on my laptop. I eventually left to work on something else and came back so I'm not sure how long it took exactly.

I just made a simple class that matched exactly the CSV that you sent:

    public class Address
    {
        [Key]        
        [DatabaseGenerated(DatabaseGeneratedOption.None)]
        public int ADDRESS_ID { get; set; }

        public int? STREET_NO { get; set; }

        public string STREET_NO_SUFFIX  { get; set; }
        //etc...
    }

My next step will be to add some better error logging so you can get a better idea of why it is crashing for you. I will let you know once that is available.

[danspencer]

Thanks for the update @dpaquette!

[dpaquette]

Okay, give the new v0.3.2 a try. This new version should give you a full stacktrace and proper error message when the seeding fails.

[danspencer]

Upgrading to 0.3.2 gives me the full stacktrace and error message. Thanks! That should help me troubleshoot the error.