zzzprojects / Bulk-Operations

C# SQL Bulk Operations | High-performance C# bulk insert, update, delete and merge for SQL Server, SQL Azure, SQL Compact, MySQL, and SQLite.
https://bulk-operations.net
142 stars 36 forks source link

BulkInsert many-to-many with implicit link table #34

Closed riley-van-hengstum closed 5 years ago

riley-van-hengstum commented 5 years ago

I have this entity class which has a many-to-many relation with an implicit link table:

public class DiscountRule
{
    public virtual ISet<CostType> ExcludedCostTypes { get; set; }
              = new HashSet<CostType>();
}

I want to bulk insert a set of DiscountRules and include the implicit link table but not the CostType target table.

If I don't specifiy a IncludeGraphOperationBuilder option, like this:

await context.BulkInsertAsync(discountRules, options =>
{                    
    options.IncludeGraph = true;
    options.UnsafeMode = true;
});

I get the error "Cannot insert duplicate key row in object 'dbo.CostTypes' with unique index 'IX_CostType_Value'. The duplicate key value is (0)."

So I try to add a IncludeGraphOperationBuilder option to exclude the CostType table:

await context.BulkInsertAsync(discountRules, options =>
{    
    options.IncludeGraph = true;
    options.IncludeGraphOperationBuilder = operation =>
    {
        if (operation is BulkOperation<CostType>)
        {
            var bulk = (BulkOperation<CostType>) operation;
            bulk.IsReadOnly = true;
        }
    };
    options.UnsafeMode = true;
});

But then I get the exception:

"internalexception": {
  "message": "Exception has been thrown by the target of an invocation.",
  "type": "System.Reflection.TargetInvocationException",
  "stacktrace": "   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)\r\n   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)\r\n   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)\r\n   at Z.EntityFramework.Extensions.InternalBulkOperationManager.\u0001[\u0001](BulkOperation`1 \u0002, DbContext \u0003, List`1 \u0004, Boolean \u0005, List`1 \u0006, Type \u0007, String \b)\r\n   at Z.EntityFramework.Extensions.InternalBulkOperationManager.\b.\u0001(SchemaEntityType )\r\n   at System.Collections.Generic.List`1.ForEach(Action`1 action)\r\n   at Z.EntityFramework.Extensions.InternalBulkOperationManager.\u0001[\u0001](DbContext \u0002, BulkOperation`1 \u0003, IEnumerable`1 \u0004, List`1 \u0005)\r\n   at Z.EntityFramework.Extensions.InternalBulkOperationManager.\u0001[\u0001](BulkOperation`1 \u0002, DbContext \u0003, IEnumerable`1 \u0004, List`1 \u0005)\r\n   at DbContextExtensions.\u0001[\u0001](DbContext , IEnumerable`1 , Action`1 , List`1 )\r\n   at Z.EntityFramework.Extensions.BulkSaveChanges.\u0001(DbContext , List`1 , List`1 , Action`1 )\r\n   at DbContextExtensions.\u0001(DbContext , Boolean , Action`1 , Boolean )",
  "internalexception": {
    "message": "Incorrect syntax near ';'.",
    "type": "System.Data.SqlClient.SqlException",
    "stacktrace": "   at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)\r\n   at System.Data.SqlClient.SqlInternalConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)\r\n   at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)\r\n   at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)\r\n   at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()\r\n   at System.Data.SqlClient.SqlDataReader.get_MetaData()\r\n   at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)\r\n   at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)\r\n   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)\r\n   at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)\r\n   at System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method)\r\n   at System.Data.SqlClient.SqlCommand.ExecuteDbDataReader(CommandBehavior behavior)\r\n   at System.Data.Common.DbCommand.System.Data.IDbCommand.ExecuteReader(CommandBehavior behavior)\r\n   at System.Data.Common.DbDataAdapter.FillInternal(DataSet dataset, DataTable[] datatables, Int32 startRecord, Int32 maxRecords, String srcTable, IDbCommand command, CommandBehavior behavior)\r\n   at System.Data.Common.DbDataAdapter.Fill(DataSet dataSet, Int32 startRecord, Int32 maxRecords, String srcTable, IDbCommand command, CommandBehavior behavior)\r\n   at System.Data.Common.DbDataAdapter.Fill(DataSet dataSet)\r\n   at Z.BulkOperations.SqlActionExecutor.\u0001(SqlCommand \u0002, BulkOperation \u0003, Int32 \u0004)\r\n   at Z.BulkOperations.SqlActionExecutor.\u0001(DbCommand \u0002, BulkOperation \u0003, Int32 \u0004)\r\n   at Z.BulkOperations.SqlActionExecutor.\u0006(ExecuteAction , DbCommand )\r\n   at Z.BulkOperations.DbActionExecutor.\u0001(List`1 )\r\n   at Z.BulkOperations.SqlProvider.\u0001(List`1 )\r\n   at \u0007.\u0003.\u0001(BulkOperation )\r\n   at Z.BulkOperations.BulkOperation.BulkInsert()\r\n   at Z.EntityFramework.Extensions.InternalBulkOperationManager.\u0001[\u0001](BulkOperation`1 \u0002, DbContext \u0003, List`1 \u0004, Boolean \u0005, List`1 \u0006, Type \u0007, String \b)\r\n   at Z.EntityFramework.Extensions.PublicInternalBulkOperationManager.BulkInsertCast[T](BulkOperation`1 this, DbContext context, List`1 list, Boolean isManager, List`1 entitiesToUpdate, Type type, String typeName)"
  }
}

Which I don't know what to do with.

Can BulkInsert handle a many-to-many relation and include the implicit link table but not the target table?

JonathanMagnan commented 5 years ago

Hello @Jaap-van-Hengstum ,

We tried on our side and everything works as expected.

However, the exception message gives us some hint. We have some idea to reproduce it but we want to make sure we will fix your case.

Do you think you could provide us the SQL generated by our library by using SQL Profiler?

Here is the code we tried:

using System.Collections.Generic;
using System.ComponentModel.DataAnnotations.Schema;
using System.Data.Entity;
using System.Linq;
using System.Threading.Tasks;
using System.Windows.Forms;
using Z.BulkOperations;

namespace Z.EntityFramework.Extensions.Lab
{
    public partial class Form_Request_ManyToMany_ReadOnly : Form
    {
        public Form_Request_ManyToMany_ReadOnly()
        {
            InitializeComponent();

            // CLEAR
            {
                using (var context = new CurrentContext())
                {
                    context.DiscountRules.RemoveRange(context.DiscountRules);
                    context.CostTypes.RemoveRange(context.CostTypes);
                    context.SaveChanges();
                }
            }

            // SEED
            {
                using (var context = new CurrentContext())
                {
                    for (int i = 0; i < 11; i++)
                    {
                        context.CostTypes.Add(new CostType() {ColumnInt = i});
                    }

                    context.SaveChanges();
                }
            }

            // TEST
            {
                using (var context = new CurrentContext())
                {
                    var costTypes = context.CostTypes.ToList();

                    var discountRules = new List<DiscountRule>();

                    {
                        // Create 11 rules

                        for (int i = 0; i < 11; i++)
                        {
                            discountRules.Add(new DiscountRule() { ColumnInt = 1, CostTypes = new List<CostType>()});
                        }
                    }

                    // Populate many to many list
                    {
                        for (int i = 0; i < costTypes.Count; i++)
                        {
                            var costType = costTypes[i];
                            var discountRuleToAdd = discountRules.Take(i).ToList();

                            costType.DiscountRules = discountRuleToAdd;
                            discountRuleToAdd.ForEach(x => x.CostTypes.Add(costType));
                        }
                    }

                    //{
                    //    // MUST throw unique index exception
                    //    context.BulkInsert(discountRules, options =>
                    //    {
                    //        options.IncludeGraph = true;
                    //    });
                    //}

                    {
                        // WORK with and without proxy
                        var task = context.BulkInsertAsync(discountRules, options =>
                        {
                            options.IncludeGraph = true;
                            options.UnsafeMode = true;
                            options.IncludeGraphOperationBuilder = operation =>
                            {
                                if (operation is BulkOperation<CostType>)
                                {
                                    var bulk = (BulkOperation<CostType>)operation;
                                    bulk.IsReadOnly = true;
                                }
                            };
                        });

                        Task.WaitAll(task);
                    }
                }
            }
        }

        public class CurrentContext : DbContext
        {
            public CurrentContext()
                : base("CodeFirstEntities")
            {
            }

            public virtual DbSet<DiscountRule> DiscountRules { get; set; }
            public virtual DbSet<CostType> CostTypes { get; set; }

            protected override void OnModelCreating(DbModelBuilder modelBuilder)
            {
                modelBuilder.Entity<DiscountRule>()
                    .HasMany(x => x.CostTypes)
                    .WithMany(x => x.DiscountRules)
                    .Map(c =>
                    {
                        c.MapLeftKey("DiscountRule");
                        c.MapRightKey("CostType");
                        c.ToTable("ImplicitLinkTable");
                    });

                base.OnModelCreating(modelBuilder);
            }
        }

        public class DiscountRule
        {
            public int Id { get; set; }
            public int ColumnInt { get; set; }
            public virtual List<CostType> CostTypes { get; set; }
        }

        public class CostType
        {
            public int Id { get; set; }
            [Index(IsUnique = true)]
            public int ColumnInt { get; set; }
            public virtual List<DiscountRule> DiscountRules { get; set; }
        }
    }
}
riley-van-hengstum commented 5 years ago

Here is a trace: trace.zip

This seems like the offending query: exec sp_executesql N' SELECT ''INSERT'' AS [$action], StagingTable.ZZZ_Index, DestinationTable.[Id] AS [Id_zzzinserted] FROM [dbo].[CostTypes] AS DestinationTable INNER JOIN (SELECT @0_0 AS [Id], @0_1 AS [Value], @0_2 AS [Name], @0_3 AS [Description], @0_4 AS ZZZ_Index UNION ALL SELECT @1_0 AS [Id], @1_1 AS [Value], @1_2 AS [Name], @1_3 AS [Description], @1_4 AS ZZZ_Index ) AS StagingTable ON ;',N'@0_0 bigint,@0_1 int,@0_2 nvarchar(max) ,@0_3 nvarchar(max) ,@0_4 int,@1_0 bigint,@1_1 int,@1_2 nvarchar(max) ,@1_3 nvarchar(max) ,@1_4 int',@0_0=1,@0_1=0,@0_2=N'Certificate',@0_3=N'certificaat',@0_4=0,@1_0=3,@1_1=2,@1_2=N'Examination',@1_3=N'examen',@1_4=1

JonathanMagnan commented 5 years ago

Hello @Jaap-van-Hengstum ,

Thank you for the additional information,

It gives us a big hint to look at the primary key join.

We are currently investigating it to try to reproduce it.

Best Regards,

Jonathan

JonathanMagnan commented 5 years ago

Hello @Jaap-van-Hengstum ,

We tried several scenarios to reproduce it but unfortunately, they all worked.

Your trace tells us that the key is not mapped correctly or is not found.

We have some additional question that might help us:

Obviously providing us a basic project sample with the issue will be the best for us since we will be able to fix it instead to find a way to reproduce it.

Best Regards,

Jonathan

JonathanMagnan commented 5 years ago

Hello @Jaap-van-Hengstum ,

Do you think it will be possible on your side to provide the requesting information?

Best Regards,

Jonathan

riley-van-hengstum commented 5 years ago

Sorry for the delay.

Hello @Jaap-van-Hengstum ,

We tried several scenarios to reproduce it but unfortunately, they all worked.

Your trace tells us that the key is not mapped correctly or is not found.

We have some additional question that might help us:

* Which version of EFE are you using?

6.1.3

* Is the `Id` an identity column?

Yes (see below)

* Do you use some other options? Such as specifying ColumnInputExpression

No, just IncludeGraph, IncludeGraphOperationBuilder and UnsafeMode.

* What's the `ISet` class?

The standard System.Collections.Generic.ISet<T> interface.

Obviously providing us a basic project sample with the issue will be the best for us since we will be able to fix it instead to find a way to reproduce it.

Best Regards,

Jonathan

We use the following model classes:

DiscountRule.cs

    public class DiscountRule : IIdentifiable, IExtendedAuditable
    {
        public static readonly DiscountRule Default = new DiscountRule
        {
            Id = 0,
            EmployerId = null,
            CalculationMethod = DiscountCalculationMethod.DiscountFirst,
            DiscountFactor = decimal.Zero,
            ResellerFeeFactor = decimal.Zero,
            EmployerFeeFactor = decimal.Zero,
            ProcessingFeeFactor = decimal.Zero,
            EffectiveFromDate = DateTime.MinValue,
            Tag = "Default"
        };

        public long Id { get; set; }

        public string Tag { get; set; }

        public virtual Employer Employer { get; set; }

        public long? EmployerId { get; set; }

        public virtual Catalog Catalog { get; set; }

        public long CatalogId { get; set; }

        [Column(TypeName = "date")]
        public DateTime EffectiveFromDate { get; set; }

        public decimal DiscountFactor { get; set; }

        public decimal ResellerFeeFactor { get; set; }

        public decimal EmployerFeeFactor { get; set; }

        public decimal ProcessingFeeFactor { get; set; }

        public DiscountCalculationMethod CalculationMethod { get; set; }

        public virtual ISet<CostType> ExcludedCostTypes { get; set; }
            = new HashSet<CostType>();
    }

DiscountRule has the following derived types:

    public class CatalogDiscountRule : DiscountRule
    {
    }

    public class ProductDiscountRule : DiscountRule
    {
        public virtual Product Product { get; set; }

        public long ProductId { get; set; }
    }

    public class ProgramFormDiscountRule : DiscountRule
    {
        public virtual ProgramForm ProgramForm { get; set; }

        public long ProgramFormId { get; set; }
    }

CostType.cs

    public class CostType : IIdentifiable
    {
        private CostType(CostTypeValue value)
        {
            Value = value;
            Name = value.ToString();
            Description = value.GetMember().Attributes.Get<DescriptionAttribute>().Description;
        }

        protected CostType()
        {
        }

        public long Id { get; set; }

        [Index("IX_CostType_Value", 0, IsUnique = true)]
        public CostTypeValue Value { get; set; }

        public string Name { get; set; }

        public string Description { get; set; }

        public static implicit operator CostType(CostTypeValue value) => new CostType(value);

        public static implicit operator CostTypeValue(CostType costType) => costType.Value;

        public virtual ISet<DiscountRule> DiscountRules { get; set; }
            = new HashSet<DiscountRule>();
    }

We also use the following EF configuration for DiscountRule and CostType:


        protected override void OnModelCreating(DbModelBuilder modelBuilder)
        {
             ...

            modelBuilder.Entity<DiscountRule>()
                .HasMany(e => e.ExcludedCostTypes)
                .WithMany(e => e.DiscountRules)
                .Map(m => m.ToTable("DiscountRules_CostTypes"));

            modelBuilder.Entity<DiscountRule>()
                .Property(_ => _.DiscountFactor)
                .HasPrecision(18, 4);

            modelBuilder.Entity<DiscountRule>()
                .Property(_ => _.ResellerFeeFactor)
                .HasPrecision(18, 4);

            modelBuilder.Entity<DiscountRule>()
                .Property(_ => _.EmployerFeeFactor)
                .HasPrecision(18, 4);

            modelBuilder.Entity<DiscountRule>()
                .Property(_ => _.ProcessingFeeFactor)
                .HasPrecision(18, 4);
            ...
        }
JonathanMagnan commented 5 years ago

Thank you for the additional information,

We will try it again on our side.

JonathanMagnan commented 5 years ago

Hello @Jaap-van-Hengstum ,

We are still not able to reproduce the issue. We tried hard but everything seems to work for us.

For an unknown reason, our library cannot find the Id key property. So the WHERE part is wrongly created which lead to the issue.

We used your class to create our example but it still works for us ;(

You will find in attachment one project that we created. Could you try it and see if everything work as expected? If you can point us at what we are missing, that will greatly help us.

Attachment: Z.Lab.Issue_0034.zip

JonathanMagnan commented 5 years ago

Hello @Jaap-van-Hengstum ,

This issue will be closed since we haven't heard back from you.

Feel free to reopen if you can provide further information.

Best Regards,

Jonathan