NorthernLight1 / N.EntityFramework.Extensions

Bulk data support for the EntityFramework 6.5.0+
MIT License
35 stars 20 forks source link

NullReferenceException when calling BulkInsertAsync #92

Closed JasonRodman closed 1 year ago

JasonRodman commented 2 years ago

I am trying your library for the first time but I cannot seem to get it to work. I am simply trying to bulk insert a bunch of rows while ignoring a timestamp column that is used for optimistic concurrency. I cant seem to get any detail out of the error that leads me to what could be causing the error. There may be something specific to the setup of my datacontext that your library does not handle but I have no way to find out what that is. I am trying to go step by step based on your code to see if I can reproduce what your code does to find the issue, but so far I have not found the issue. Any ideas how I can troubleshoot this?

await context.BulkInsertAsync(recordsToAdd, options => { options.UsePermanentTable = true; options.IgnoreColumns = o => new { o.RowVersion }; });

NorthernLight1 commented 2 years ago

JasonRodman,

I have tried several times to replicate your issue but I have not been able to get it to error at all.

Please provide more details about the error that is thrown like the stack trace and any other relevant data I could use to trace it.

You may want to download the source code and use it instead of the nuget library to step through the code to figure out why it is erroring.

JasonRodman commented 2 years ago

I spent a day and tried to pinpoint where the code was failing too. The stack trace gives me no insight. So, resorted to using reflection to recreate what your BulkInsertAsync call does internally, step by step. I was trying to pinpoint where it was failing and why to give you some direction since I can't give you a repro project for this. I got all the way to the step where you generate the merge sql statement. After struggling for many hours with a looming deadline I had to give up on it and changed course. What I suspect is the cause is that I am storing enums as strings iso I have dual properties: an enum and a string with the enum ignored but the string property is named the same as the enum property in the schema when I map it. We did this since EF6 can't store enums as string yet, and we needed a way to query using linq.

[JsonConverter(typeof(StringEnumConverter))]
 public JobStatus Status { get; private set; }

[JsonIgnore]
public string StatusValue
{
    get => Status.ToString();
    set => Status = value.ToEnum<JobStatus>();
}

Another cause may be the accessibility of my properties. Most of them have private setters, and the identity is protected internal. This is on purpose due to using DDD as our architectural pattern for these entities, only allowing change thru methods. EF handles settings these properties just fine though. This entity also uses an integer identity column as well as a timestamp column marked as [RowVersion] for optimistic concurrency. I am not sure which of these were the cause, but hopefully this will help you find which one is.

NorthernLight1 commented 2 years ago

JasonRodman,

I tried the following without any errors:

  1. Added enumeration code that you posted above
  2. Added a row version property with a TimeStamp property

This is what the Product class looks like in my test code:

`public class Product {

    [Key]
    public int Id { get; set; }
    public string Name { get; set; }
    public double? Price { get; set; }

   [JsonConverter(typeof(StringEnumConverter))]
    public ProductStatus Status { get; private set; }

    [JsonIgnore]
    public string StatusValue
    {
        get => Status.ToString();
        set => Status = (ProductStatus)Enum.Parse(typeof(ProductStatus), value);
    }
    public DateTime AddedDate { get; set; }
    [Timestamp]
    public byte[] RowVersion { get; set; }
}`

When I set the Id to be protected internal, the framework complains that no key exists on the class and this is a validation error generated by EF.

A couple of other things you could try

  1. Set the AutoMapOutput to false so that auto generated columns from the database are not set after the Insert happens
  2. Use the IncludeColumns to only include required fields. Identity fields will be automatically included though as those are required by the framework.
NorthernLight1 commented 2 years ago

JasonRodman,

I just pushed a new version v1.7.2 to nuget that takes care of properties that use the Timestamp attribute so you will no longer need to explicitly omit that property using the IgnoreColumns option.

NorthernLight1 commented 1 year ago

Issue can't be reproduced.