borisdj / EFCore.BulkExtensions

Entity Framework EF Core efcore Bulk Batch Extensions with BulkCopy in .Net for Insert Update Delete Read (CRUD), Truncate and SaveChanges operations on SQL Server, PostgreSQL, MySQL, SQLite
https://codis.tech/efcorebulk
Other
3.65k stars 582 forks source link

Issues with conversion and strongly typed Ids #1343

Closed RMarjanovic closed 2 months ago

RMarjanovic commented 10 months ago

Hey and thank you for the work you've put into this library! This is my first time using it and trying to get it set up in my current applications, please bare with me.

TDLR; when performing a bulk insert/update with only BulkSaveChanges I'm getting an invalid cast exception.

 System.InvalidCastException: Invalid cast from 'System.String' to 'Domain.Features.Players.ValueObjects.PlayerId'.

Long version: I'm following domain driven design principles, using strongly typed ids and abstract factory methods to handle conversions. However when the BulkSaveChanges method runs it throws. Original EfCore SaveChanges is working fine. In this case "ModId", "PlayerId" are strings under the hood. Any help with this is highly appreciated.

Full exception:

Titan.SentinelManagement     | [12:14:31 INF] Executed DbCommand (1ms) [Parameters=[], CommandType='Text', CommandTimeout='300']
Titan.SentinelManagement     | CREATE TABLE "public"."ModsTempdc6417df" AS TABLE "public"."Mods" WITH NO DATA;
Titan.SentinelManagement     | [12:14:31 INF] Executed DbCommand (7ms) [Parameters=[], CommandType='Text', CommandTimeout='300']
Titan.SentinelManagement     | INSERT INTO "public"."Mods" ("Id", "PlayerId", "Level", "ModDefinitionId", "ModTier", "Rarity", "RerolledCount", "Slot", "Type") (SELECT "Id", "PlayerId", "Level", "ModDefinitionId", "ModTier", "Rarity", "RerolledCount", "Slot", "Type" FROM "public"."ModsTempdc6417df") ON CONFLICT ("Id", "PlayerId") DO UPDATE SET "Level" = EXCLUDED."Level", "ModDefinitionId" = EXCLUDED."ModDefinitionId", "ModTier" = EXCLUDED."ModTier", "Rarity" = EXCLUDED."Rarity", "RerolledCount" = EXCLUDED."RerolledCount", "Slot" = EXCLUDED."Slot", "Type" = EXCLUDED."Type" RETURNING "public"."Mods"."Id", "public"."Mods"."PlayerId", "public"."Mods"."Level", "public"."Mods"."ModDefinitionId", "public"."Mods"."ModTier", "public"."Mods"."Rarity", "public"."Mods"."RerolledCount", "public"."Mods"."Slot", "public"."Mods"."Type"
Titan.SentinelManagement     | [12:14:31 INF] Executed DbCommand (1ms) [Parameters=[], CommandType='Text', CommandTimeout='300']
Titan.SentinelManagement     | DROP TABLE IF EXISTS "public"."ModsTempdc6417df"
Titan.SentinelManagement     | [12:14:31 INF] Executed DbCommand (1ms) [Parameters=[], CommandType='Text', CommandTimeout='300']
Titan.SentinelManagement     | CREATE TABLE "public"."ModStatTemp15fd849a" AS TABLE "public"."ModStat" WITH NO DATA;
SentinelDb                   | 2023-12-07 12:14:31.700 UTC [34] ERROR:  COPY from stdin failed:
SentinelDb                   | 2023-12-07 12:14:31.700 UTC [34] STATEMENT:  COPY "public"."ModStatTemp15fd849a" ("ModId", "PlayerId", "Position", "IsPercentage", "ModPlayerId", "StatRolls", "StatType", "Value") FROM STDIN (FORMAT BINARY);
SentinelDb                   | 2023-12-07 12:14:31.701 UTC [34] ERROR:  current transaction is aborted, commands ignored until end of transaction block
SentinelDb                   | 2023-12-07 12:14:31.701 UTC [34] STATEMENT:  DROP TABLE IF EXISTS "public"."ModStatTemp15fd849a"
Titan.SentinelManagement     | [12:14:31 ERR] Failed executing DbCommand (1ms) [Parameters=[], CommandType='Text', CommandTimeout='300']
Titan.SentinelManagement     | DROP TABLE IF EXISTS "public"."ModStatTemp15fd849a"
Titan.SentinelManagement     | [12:14:31 ERR] Job DEFAULT.UpdateGuildMemberTrackedPlayersJob threw an unhandled Exception:
Titan.SentinelManagement     | System.InvalidCastException: Invalid cast from 'System.String' to 'Titan.SentinelManagement.Domain.Features.Players.ValueObjects.PlayerId'.
Titan.SentinelManagement     |    at System.Convert.DefaultToType(IConvertible value, Type targetType, IFormatProvider provider)
Titan.SentinelManagement     |    at System.Convert.ChangeType(Object value, Type conversionType, IFormatProvider provider)
Titan.SentinelManagement     |    at Microsoft.EntityFrameworkCore.Storage.ValueConversion.ValueConverter`2.Sanitize[T](Object value)
Titan.SentinelManagement     |    at Microsoft.EntityFrameworkCore.Storage.ValueConversion.ValueConverter`2.<>c__DisplayClass6_0`2.<SanitizeConverter>b__1(Object v)
Titan.SentinelManagement     |    at EFCore.BulkExtensions.SqlAdapters.PostgreSql.PostgreSqlAdapter.InsertAsync[T](DbContext context, IEnumerable`1 entities, TableInfo tableInfo, Action`1 progress, Boolean isAsync, CancellationToken cancellationToken)
Titan.SentinelManagement     |    at EFCore.BulkExtensions.SqlAdapters.PostgreSql.PostgreSqlAdapter.InsertAsync[T](DbContext context, IEnumerable`1 entities, TableInfo tableInfo, Action`1 progress, Boolean isAsync, CancellationToken cancellationToken)
Titan.SentinelManagement     |    at EFCore.BulkExtensions.SqlAdapters.PostgreSql.PostgreSqlAdapter.InsertAsync[T](DbContext context, Type type, IEnumerable`1 entities, TableInfo tableInfo, Action`1 progress, CancellationToken cancellationToken)
Titan.SentinelManagement     |    at EFCore.BulkExtensions.SqlAdapters.PostgreSql.PostgreSqlAdapter.MergeAsync[T](DbContext context, Type type, IEnumerable`1 entities, TableInfo tableInfo, OperationType operationType, Action`1 progress, Boolean isAsync, CancellationToken cancellationToken)
Titan.SentinelManagement     |    at EFCore.BulkExtensions.SqlAdapters.PostgreSql.PostgreSqlAdapter.MergeAsync[T](DbContext context, Type type, IEnumerable`1 entities, TableInfo tableInfo, OperationType operationType, Action`1 progress, Boolean isAsync, CancellationToken cancellationToken)
Titan.SentinelManagement     |    at EFCore.BulkExtensions.SqlAdapters.PostgreSql.PostgreSqlAdapter.MergeAsync[T](DbContext context, Type type, IEnumerable`1 entities, TableInfo tableInfo, OperationType operationType, Action`1 progress, CancellationToken cancellationToken)
Titan.SentinelManagement     |    at EFCore.BulkExtensions.SqlBulkOperation.MergeAsync[T](DbContext context, Type type, IEnumerable`1 entities, TableInfo tableInfo, OperationType operationType, Action`1 progress, CancellationToken cancellationToken)
Titan.SentinelManagement     |    at EFCore.BulkExtensions.DbContextBulkTransaction.ExecuteAsync[T](DbContext context, Type type, IEnumerable`1 entities, OperationType operationType, BulkConfig bulkConfig, Action`1 progress, CancellationToken cancellationToken)
Titan.SentinelManagement     |    at EFCore.BulkExtensions.DbContextBulkTransactionSaveChanges.InvokeBulkMethod(DbContext context, List`1 entities, Type entityType, String methodName, BulkConfig bulkConfig, Action`1 progress, Boolean isAsync, CancellationToken cancellationToken)
Titan.SentinelManagement     |    at EFCore.BulkExtensions.DbContextBulkTransactionSaveChanges.SaveChangesAsync(DbContext context, BulkConfig bulkConfig, Action`1 progress, Boolean isAsync, CancellationToken cancellationToken)
Titan.SentinelManagement     |    at EFCore.BulkExtensions.DbContextBulkTransactionSaveChanges.SaveChangesAsync(DbContext context, BulkConfig bulkConfig, Action`1 progress, Boolean isAsync, CancellationToken cancellationToken)
Titan.SentinelManagement     |    at EFCore.BulkExtensions.DbContextBulkTransactionSaveChanges.SaveChangesAsync(DbContext context, BulkConfig bulkConfig, Action`1 progress, CancellationToken cancellationToken)
Titan.SentinelManagement     |    at EFCore.BulkExtensions.DbContextBulkTransaction.ExecuteAsync[T](DbContext context, Type type, IEnumerable`1 entities, OperationType operationType, BulkConfig bulkConfig, Action`1 progress, CancellationToken cancellationToken)
Titan.SentinelManagement     |    at Titan.SentinelManagement.Persistence.Repositories.UnitOfWork.CompleteAsync(CancellationToken cancellationToken) in /src/Infrastructure/Titan.SentinelManagement.Persistence/Repositories/UnitOfWork.cs:line 62
Titan.SentinelManagement     |    at Titan.SentinelManagement.Application.Behaviors.UnitOfWorkPipelineBehavior`2.Handle(TRequest request, RequestHandlerDelegate`1 next, CancellationToken cancellationToken) in /src/Core/Titan.SentinelManagement.Application/Behaviors/UnitOfWorkPipelineBehavior.cs:line 26
Titan.SentinelManagement     |    at Titan.SentinelManagement.Application.Behaviors.LoggingPipelineBehavior`2.Handle(TRequest request, RequestHandlerDelegate`1 next, CancellationToken cancellationToken)
Titan.SentinelManagement     |    at Titan.SentinelManagement.Application.Behaviors.ValidationPipelineBehavior`2.Handle(TRequest request, RequestHandlerDelegate`1 next, CancellationToken cancellationToken)
Titan.SentinelManagement     |    at Titan.SentinelManagement.Infrastructure.BackgroundJobs.UpdateGuildMemberTrackedPlayersJob.Execute(IJobExecutionContext context) in /src/Infrastructure/Titan.SentinelManagement.Infrastructure/BackgroundJobs/UpdateGuildMemberTrackedPlayersJob.cs:line 39
Titan.SentinelManagement     |    at Quartz.Core.JobRunShell.Run(CancellationToken cancellationToken)

The domain models has been slimmed down for brevity.

public sealed class Mod : Entity<ModId>
{
    public PlayerId PlayerId { get; private set; }
    public ModDefinitionId ModDefinitionId { get; private set; }
    public int Level { get; private set; }
    public IReadOnlyCollection<ModStat> Stats => _stats;
    private readonly List<ModStat> _stats = [];
    public ModTier ModTier { get; private set; }
    public ModSlot Slot { get; private set; }
    public ModType Type { get; private set; }
    public ModRarity Rarity { get; private set; }
    public int RerolledCount { get; private set; }
}

public sealed class ModStat : ValueObject
{
    public ModId ModId { get; init; }
    public PlayerId PlayerId { get; init; }
    public StatType StatType { get; init; }
    public int Position { get; init; }
    public float Value { get; init; }
    public int StatRolls { get; init; }
    public bool IsPercentage { get; init; }
}

Db configurations

public class ModConfiguration : IEntityTypeConfiguration<Mod>
    {
        public void Configure(EntityTypeBuilder<Mod> builder)
        {
            builder
                .ToTable("Mods");

            builder
                .HasKey(x => new { x.Id, x.PlayerId });

            builder
                .Property(x => x.Id)
                .HasConversion(x => x.Value, v => ModId.Create(v).Value)
                .HasMaxLength(ModId.MaxLength)
                .ValueGeneratedNever()
                .IsRequired();

            builder
                .Property(x => x.PlayerId)
                .HasConversion(x => x.Value, v => PlayerId.Create(v).Value)
                .HasMaxLength(PlayerId.MaxLength)
                .ValueGeneratedNever()
                .IsRequired();

            builder
                .Property(x => x.ModDefinitionId)
                .HasConversion(x => x.Value, v => ModDefinitionId.Create(v).Value)
                .HasMaxLength(ModDefinitionId.MaxLength)
                .ValueGeneratedNever()
                .IsRequired();

            builder
                .Property(x => x.Slot)
                .HasConversion<int>()
                .IsRequired();

            builder
                .Property(x => x.Type)
                .HasConversion<int>()
                .IsRequired();

            builder
                .Property(x => x.Rarity)
                .HasConversion<int>()
                .IsRequired();

            builder
                .Property(x => x.ModTier)
                .HasConversion<int>()
                .IsRequired();

            builder
                .Property(x => x.Level)
                .IsRequired();

            builder
                .Property(x => x.RerolledCount)
                .IsRequired();

            builder
                .HasMany(x => x.Stats)
                .WithOne();

            builder
                .Metadata
                .FindNavigation(nameof(Mod.Stats))!
                    .SetPropertyAccessMode(PropertyAccessMode.Field);
        }
    }
   public class ModStatConfiguration : IEntityTypeConfiguration<ModStat>
    {
        public void Configure(EntityTypeBuilder<ModStat> builder)
        {
            builder
                .ToTable("ModStat");

            builder
                .HasKey(x => new { x.ModId, x.PlayerId, x.Position });

            builder
                .Property(x => x.ModId)
                .HasConversion(x => x.Value, v => ModId.Create(v).Value)
                .HasMaxLength(ModId.MaxLength)
                .ValueGeneratedNever()
                .IsRequired();

            builder
                .Property(x => x.PlayerId)
                .HasConversion(x => x.Value, v => PlayerId.Create(v).Value)
                .HasMaxLength(PlayerId.MaxLength)
                .ValueGeneratedNever()
                .IsRequired();

            builder
                .Property(x => x.Position)
                .IsRequired();

            builder
                .Property(x => x.StatType)
                .HasConversion<int>()
                .IsRequired();

            builder
                .Property(x => x.IsPercentage)
                .IsRequired();

            builder
                .Property(x => x.Value)
                .HasPrecision(8, 2)
                .IsRequired();

            builder
                .Property(x => x.StatRolls)
                .IsRequired();
        }
    }

Use case (slimmed down for brevity):

    public async Task<Result> Handle(UpdateModsCommand request, CancellationToken cancellationToken)
    {
        var modDict = await _unitOfWork.Mods
            .GetAggregateDictionary(request.ExpandedPlayerProfiles
                .Select(x => x.Key)
                .ToList(), cancellationToken);

        var newMods = new List<Mod>();
        foreach (var profile in request.ExpandedPlayerProfiles.Values)
        {
            var addedMods = GetOrUpdateMods(modDict, profile);
            newMods.AddRange(addedMods);
        }

        await _unitOfWork.Mods.AddRangeAsync(newMods, cancellationToken);
        return Result.Success();
    }

Save changes is run through a pipeline behavior (CompleteAsync), and if changes are above 1000 it uses BulkSaveChanges (without any Config provided) instead of ordinary SaveChanges

    public async Task<TResponse> Handle(TRequest request, RequestHandlerDelegate<TResponse> next, CancellationToken cancellationToken)
    {
        await _unitOfWork.BeginTransaction(IsolationLevel.ReadCommitted, cancellationToken);
        var response = await next();
        if (response is Result { IsSuccess: true })
        {
            await _unitOfWork.CompleteAsync(cancellationToken);
            await _unitOfWork.CommitTransaction(cancellationToken);
            return response;
        }

        await _unitOfWork.RollbackTransaction(cancellationToken);
        return response;
    }
borisdj commented 10 months ago

Can you also add a definition of base class : Entity And even better could you write a test for this example.

RMarjanovic commented 10 months ago

Thank you for looking into this and your quick reply. I managed to reproduce the issue in this repo.

RMarjanovic commented 10 months ago

Base class entity added. Fully working test example demonstrating the error in the repo link above.

public abstract class Entity<TId> : IEquatable<Entity<TId>>
    where TId : notnull
{
    public TId Id { get; private init; }

    protected Entity(TId id)
    {
        Id = id;
    }

    public static bool operator ==(Entity<TId> first, Entity<TId> second)
    {
        return first is not null && second is not null && first.Equals(second);
    }

    public static bool operator !=(Entity<TId> first, Entity<TId> second)
    {
        return !(first == second);
    }

    public override bool Equals(object? obj)
    {
        if (obj is null) return false;
        if (obj.GetType() != GetType()) return false;
        if (obj is not Entity<TId> entity) return false;
        return Id.Equals(entity.Id);
    }

    public override int GetHashCode()
    {
        return Id.GetHashCode() * 41;
    }

    public bool Equals(Entity<TId>? other)
    {
        if (other is null) return false;
        if (other.GetType() != GetType()) return false;
        return Id.Equals(other.Id);
    }
}
borisdj commented 10 months ago

I have copied all files from your project into the source of the library, but getting: Column, parameter, or variable #5: Cannot specify a column width on data type real.

Could you Fork the library, and add those files there. Then make a like this: (where the issue would be reproducible)

[Fact]
public static void RunTestTemp()
{
  using var context = new TestContext(ContextUtil.GetOptions());

  var playerId = PlayerId.Create("--wvkCLQSFiG9xpBbTTXBg");
  var modStat1 = ModStat.Create("000NeMAdR1Cx6FVye4UV6Q", playerId.Value, StatType.Unitstatmaxhealth, 1, 2, 2, false);
  var modStat2 = ModStat.Create("000NeMAdR1Cx6FVye4UV6Q", playerId.Value, StatType.Unitstatmaxhealth, 2, 2, 2, false);
  var modStatLis = new List<ModStat>
  {
   modStat1.Value,
   modStat2.Value
  };

  var mod = Mod.Create("000NeMAdR1Cx6FVye4UV6Q", playerId.Value, "163", ModType.POTENCY, ModRarity.MODRARITY3, ModSlot.ARROW, 1, ModTier.A, 2, modStatLis);

  context.Mods.Add(mod.Value);
  context.BulkSaveChanges();
}
RMarjanovic commented 10 months ago

Could you please advice which file and / or directory you want me to add the test to?

The error you're getting is from a migration done using sql server (I was using postgres) which caused real types to add a width. Ie real(8). I had to manually remove these in the migration to get it to work in sql server.

borisdj commented 10 months ago

It can be added into EFCoreBulkTestAtypical.cs

borisdj commented 10 months ago

So are using this with Postres or with SqlServer when having the exception? Also if you can strip the example into minimal component that would still reproduce the issue. Currently it has multiple files and classes, probably most of those are no required for the issue.

RMarjanovic commented 10 months ago

I have slimmed it down into one file, and removed all unnecessary classes and concepts. It looks like the error occurs when trying to add the one to many with mod->modstat as I had no issue with adding just the mod.

You have a different context than me so I used my own created context in place of your using var context = new TestContext(ContextUtil.GetOptions());

I was using postgres when I first noticed the error, but I have tried different approaches and databases to get this to work, hence why I know why sql server has issues with float and precision.

Added a pull request for the changes.

    #region issue1343
    [Fact]
    public async static void RunTestTemp()
    {
        using var context = new MyDbContext(ContextUtil.GetOptions());
        var playerId = PlayerId.Create("--wvkCLQSFiG9xpBbTTXBg");
        var modStat1 = ModStat.Create("000NeMAdR1Cx6FVye4UV6Q", playerId);
        var modStatLis = new List<ModStat>
        {
            modStat1
        };
        var mod = Mod.Create("000NeMAdR1Cx6FVye4UV6Q", playerId, modStatLis);

        await context.Mods.AddAsync(mod);
        await context.BulkSaveChangesAsync();
    }
    public sealed class Mod
    {
        public ModId Id { get; private set; }
        public PlayerId PlayerId { get; private set; }
        public IReadOnlyCollection<ModStat> Stats => _stats;
        private readonly List<ModStat> _stats = [];

        private Mod(ModId id, PlayerId playerId)
        {
            Id = id;
            PlayerId = playerId;
        }

        public static Mod Create(string id, PlayerId playerId, IEnumerable<ModStat> modStat)
        {
            var modId = ModId.Create(id);
            var mod = new Mod(modId, playerId);

            mod.AddStats(modStat);
            return mod;
        }
        private void AddStats(IEnumerable<ModStat> modStats)
        {
            _stats.AddRange(modStats);
        }
    }

    public sealed class ModStat
    {
        public ModId ModId { get; init; }
        public PlayerId PlayerId { get; init; }

        private ModStat(ModId modId, PlayerId playerId)
        {
            ModId = modId;
            PlayerId = playerId;
        }

        public static ModStat Create(string modId, PlayerId playerId)
        {
            var mId = ModId.Create(modId);
            return new ModStat(mId, playerId);
        }
    }
    public sealed class PlayerId
    {
        public string Value;
        private PlayerId(string value)
            => Value = value;

        public static PlayerId Create(string value)
            => new PlayerId(value);
    }

    public sealed class ModId
    {
        public string Value;
        private ModId(string value)
            => Value = value;

        public static ModId Create(string value)
            => new ModId(value);
    }

    public class MyDbContext(DbContextOptions options) : DbContext(options)
    {
        public virtual DbSet<Mod> Mods { get; set; } = default!;

        protected override void OnModelCreating(ModelBuilder modelBuilder)
        {
            modelBuilder.ApplyConfigurationsFromAssembly(typeof(MyDbContext).Assembly);
            modelBuilder.Entity<Mod>()
                .ToTable("Mods");

            modelBuilder.Entity<Mod>()
               .HasKey(x => new { x.Id, x.PlayerId });

            modelBuilder.Entity<Mod>()
               .Property(x => x.Id)
               .HasConversion(x => x.Value, v => ModId.Create(v))
               .ValueGeneratedNever()
               .IsRequired();

            modelBuilder.Entity<Mod>()
               .Property(x => x.PlayerId)
               .HasConversion(x => x.Value, v => PlayerId.Create(v))
               .ValueGeneratedNever()
               .IsRequired();

            modelBuilder.Entity<Mod>()
               .HasMany(x => x.Stats)
               .WithOne();

            modelBuilder.Entity<Mod>()
               .Metadata
               .FindNavigation(nameof(Mod.Stats))!
                   .SetPropertyAccessMode(PropertyAccessMode.Field);

            modelBuilder.Entity<ModStat>()
                .ToTable("ModStat");

            modelBuilder.Entity<ModStat>()
                .HasKey(x => new { x.ModId, x.PlayerId });

            modelBuilder.Entity<ModStat>()
                .Property(x => x.ModId)
                .HasConversion(x => x.Value, v => ModId.Create(v))
                .ValueGeneratedNever()
                .IsRequired();

            modelBuilder.Entity<ModStat>()
                .Property(x => x.PlayerId)
                .HasConversion(x => x.Value, v => PlayerId.Create(v))
                .ValueGeneratedNever()
                .IsRequired();
        }
    }
    #endregion
borisdj commented 2 months ago

Fixed with latest commit, new nugget will be published in a few days.