zzzprojects / Dapper-Plus

Dapper Plus - High-Efficient Bulk Actions (Insert, Update, Delete, and Merge) for .NET
https://dapper-plus.net/
383 stars 85 forks source link

IndexOutOfRangeException when inserting ~700k rows #72

Closed pedromfonsecapt closed 3 years ago

pedromfonsecapt commented 3 years ago

I have the following mapping:

DapperPlusManager.Entity<Iccid>().Table("iccid")
                .Identity(x => x.Id, "id")
                .Key(x => x.Identifier, "identifier")
                .Map(x => x.IccidValue, "iccid_value")
                .Map(x => x.Status, "status")
                .Map(x => x.IdBatch, "id_batch")
                .Map(x => x.IdProduct, "id_product")
                .Map(x => x.IdManufacturer, "id_manufacturer")
                .Map(x => x.IdConsumer, "id_consumer")
                .Map(x => x.IdSimType, "id_sim_type")
                .Map(x => x.IdSimFormat, "id_sim_format")
                .Map(x => x.Pin1, "pin1")
                .Map(x => x.Puk1, "puk1")
                .Map(x => x.Pin2, "pin2")
                .Map(x => x.Puk2, "puk2")
                .Map(x => x.Comments, "comments")
                .Map(x => x.CreatedBy, "created_by")
                .Map(x => x.CreationDate, "creation_date")
                .Map(x => x.LastUpdatedBy, "last_updated_by")
                .Map(x => x.LastUpdatedDate, "last_updated_date")
                .InsertIfNotExists();

For the following entity:

public class Iccid : Resource
{
     public decimal IccidValue { get; set; }

     public int IdBatch { get; set; }
     public int IdProduct { get; set; }
     public int IdManufacturer { get; set; }
     public int IdConsumer { get; set; }
     public int IdSimType { get; set; }
     public int IdSimFormat { get; set; }
     public string Puk1 { get; set; }
     public string Pin1 { get; set; }
     public string Puk2 { get; set; }
     public string Pin2 { get; set; }

     public List<Msisdn> MsisdnList { get; set; } = new List<Msisdn>();
 }

public abstract class Resource : BaseAudit
{
     public long Id { get; set; }
     public string Identifier { get; set; }
     public string Comments { get; set; }
     public ResourceStatus Status { get; set; }
}

public abstract class BaseAudit
{
     public string CreatedBy { get; set; }
     public DateTime CreationDate { get; set; }
     public string LastUpdatedBy { get; set; }
     public DateTime LastUpdatedDate { get; set; }
}

After I do the following insert

transaction.BulkInsert(iccids);

I get the following exception:

System.IndexOutOfRangeException: 'There is no row at position 9804.'

System.IndexOutOfRangeException
  HResult=0x80131508
  Message=There is no row at position 9804.
  Source=System.Data.Common
  StackTrace:
   at System.Data.RBTree`1.GetNodeByIndex(Int32 userIndex)
   at .( , DbCommand )
   at .Execute(List`1 actions)
   at .(List`1 )
   at Z.BulkOperations.BulkOperation.Execute()
   at Z.BulkOperations.BulkOperation.BulkInsert()
   at Z.Dapper.Plus.DapperPlusAction.Execute()
   at Z.Dapper.Plus.DapperPlusActionSet`1.AddAction(String mapperKey, DapperPlusActionKind actionKind, TEntity item)
   at Z.Dapper.Plus.DapperPlusActionSet`1.DapperPlusActionSetBuilder(DapperPlusContext context, IDbConnection connection, IDbTransaction transaction, String mapperKey, DapperPlusActionKind actionKind, TEntity item, Func`2[] selectors)
   at Z.Dapper.Plus.DapperPlusActionSet`1..ctor(DapperPlusContext context, IDbTransaction transaction, String mapperKey, DapperPlusActionKind actionKind, TEntity item, Func`2[] selectors)
   at Z.Dapper.Plus.DapperPlusExtensions.BulkInsert[T](IDbTransaction transaction, String mapperKey, T item, Func`2[] selectors)
   at Z.Dapper.Plus.DapperPlusExtensions.BulkInsert[T](IDbTransaction transaction, T item, Func`2[] selectors)
   at RMMigration.CerillionMigration.Run() in C:\dev\crm\tools\rm-migration\RMMigration\RMMigration\CerillionMigration.cs:line 182
   at RMMigration.Program.Main(Boolean cleanDb, Boolean runCerillionMigration, Boolean runSimManagerMigration) in C:\dev\crm\tools\rm-migration\RMMigration\RMMigration\Program.cs:line 42

  This exception was originally thrown at this call stack:
    [External Code]
    RMMigration.CerillionMigration.Run() in CerillionMigration.cs
    RMMigration.Program.Main(bool, bool, bool) in Program.cs

I'm trying to insert ~700k entities in a Postgres database. Everything seems fine for item from the iccids list at index 9804. If I insert only some part of the data (~600k) it seems to be working though, which is odd.If I insert the remaining part of the data (~100k) I have the same error System.IndexOutOfRangeException: 'There is no row at position 7318.'

JonathanMagnan commented 3 years ago

Hello @pedromfonsecapt ,

That's indeed weird.

At this moment, I'm not sure if the issue is on your side or in our library.

Here is a working test we made. Can you reproduce the issue starting from this example or have some hint that will help us to try to reproduce it?

Is it possible to see your Main and Run code? Perhaps we will find something that could help us to reproduce it. You can send the code in private here: info@zzzprojects.com

using System;
using System.Collections.Generic;
using System.Windows.Forms;
using Dapper;
using Z.Dapper.Plus;

namespace Z.Lab
{
    public partial class Form_Request_2941 : Form
    {
        /*
CREATE TABLE [dbo].[iccid](
    [id] [INT] IDENTITY(1,1) NOT NULL,
    [identifier] [NVARCHAR](50) NOT NULL,
    [iccid_value] [INT] NOT NULL,
    [status] [INT] NOT NULL,
    [id_batch] [INT] NOT NULL,
    [id_product] [INT] NOT NULL,
    [id_manufacturer] [INT] NOT NULL,
    [id_consumer] [INT] NOT NULL,
    [id_sim_type] [INT] NOT NULL,
    [id_sim_format] [INT] NOT NULL,
    [pin1] [NVARCHAR](50) NOT NULL,
    [puk1] [NVARCHAR](50) NOT NULL,
    [pin2] [NVARCHAR](50) NOT NULL,
    [puk2] [NVARCHAR](50) NOT NULL,
    [comments] [NVARCHAR](50) NOT NULL,
    [created_by] [NVARCHAR](50) NOT NULL,
    [creation_date] [DATETIME2](7) NOT NULL,
    [last_updated_by] [NVARCHAR](50) NOT NULL,
    [last_updated_date] [DATETIME2](7) NOT NULL,
 CONSTRAINT [PK_iccid] PRIMARY KEY CLUSTERED 
(
    [identifier] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
GO

         */
        public Form_Request_2941()
        {
            InitializeComponent();

            DapperPlusManager.Entity<Iccid>().Table("iccid")
                .Identity(x => x.Id, "id")
                .Key(x => x.Identifier, "identifier")
                .Map(x => x.IccidValue, "iccid_value")
                .Map(x => x.Status, "status")
                .Map(x => x.IdBatch, "id_batch")
                .Map(x => x.IdProduct, "id_product")
                .Map(x => x.IdManufacturer, "id_manufacturer")
                .Map(x => x.IdConsumer, "id_consumer")
                .Map(x => x.IdSimType, "id_sim_type")
                .Map(x => x.IdSimFormat, "id_sim_format")
                .Map(x => x.Pin1, "pin1")
                .Map(x => x.Puk1, "puk1")
                .Map(x => x.Pin2, "pin2")
                .Map(x => x.Puk2, "puk2")
                .Map(x => x.Comments, "comments")
                .Map(x => x.CreatedBy, "created_by")
                .Map(x => x.CreationDate, "creation_date")
                .Map(x => x.LastUpdatedBy, "last_updated_by")
                .Map(x => x.LastUpdatedDate, "last_updated_date")
                .InsertIfNotExists();

            var connection = My.ConnectionFactory();
            connection.Execute("TRUNCATE TABLE iccid");

            // INSERT some data initially for the InsertIfNotExists() part
            Insert(100000);

            // INSERT 700k
            Insert(700000);
        }

        public static void Insert(int count)
        {
            var iccids = new List<Iccid>();

            for (int i = 0; i < count; i++)
            {
                iccids.Add(new Iccid()
                {
                    Identifier = i.ToString(),
                    IccidValue = i,
                    Status = ResourceStatus.Active,
                    IdBatch = i,
                    IdProduct = i,
                    IdManufacturer = i,
                    IdConsumer = i,
                    IdSimType = i,
                    IdSimFormat = i,
                    Pin1 = i.ToString(),
                    Puk1 = i.ToString(),
                    Pin2 = i.ToString(),
                    Puk2 = i.ToString(),
                    Comments = i.ToString(),
                    CreatedBy = i.ToString(),
                    CreationDate = DateTime.Now,
                    LastUpdatedBy = i.ToString(),
                    LastUpdatedDate = DateTime.Now
                });
            }

            var connection = My.ConnectionFactory();
            connection.Open();
            var trans = connection.BeginTransaction();
            trans.BulkInsert(iccids);
            trans.Commit();
        }

        public class Iccid : Resource
        {
            public decimal IccidValue { get; set; }

            public int IdBatch { get; set; }
            public int IdProduct { get; set; }
            public int IdManufacturer { get; set; }
            public int IdConsumer { get; set; }
            public int IdSimType { get; set; }
            public int IdSimFormat { get; set; }
            public string Puk1 { get; set; }
            public string Pin1 { get; set; }
            public string Puk2 { get; set; }
            public string Pin2 { get; set; }
        }

        public abstract class Resource : BaseAudit
        {
            public long Id { get; set; }
            public string Identifier { get; set; }
            public string Comments { get; set; }
            public ResourceStatus Status { get; set; }
        }

        public abstract class BaseAudit
        {
            public string CreatedBy { get; set; }
            public DateTime CreationDate { get; set; }
            public string LastUpdatedBy { get; set; }
            public DateTime LastUpdatedDate { get; set; }
        }

        public enum ResourceStatus
        {
            None,
            Active,
        }
    }
}
JonathanMagnan commented 3 years ago

Hello @pedromfonsecapt,

We haven't heard from you since our last conversation.

Are you able to show us your Main and Run code?

Please let us know if you need further assistance.

Best regards,

Jon

pedromfonsecapt commented 3 years ago

Hello @JonathanMagnan ,

I've been able to solve this using a different BatchSize, meaning:

transaction.UseBulkOptions(x => x.BatchSize = batchSize)

Even though it feels odd that changing batch sizes, solves it, its enough for me since I was just doing some tests for a one time execution.

Thanks for the help! No further assistance required.

JonathanMagnan commented 3 years ago

Great to hear!

Don't hesitate to contact us if needed.

Best regards,

Jon