Exception while reading from stream System.TimeoutException: Timeout during reading attempt @BulkMerge

[dbenjimanraj]

Npgsql.NpgsqlException (0x80004005): Exception while reading from stream ---> System.TimeoutException: Timeout during reading attempt at Npgsql.NpgsqlConnector.gReadMessageLong|194_0(NpgsqlConnector connector, Boolean async, DataRowLoadingMode dataRowLoadingMode, Boolean readingNotifications, Boolean isReadingPrependedMessage) at Npgsql.NpgsqlDataReader.NextResult(Boolean async, Boolean isConsuming, CancellationToken cancellationToken) at Npgsql.NpgsqlDataReader.NextResult() at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken) at Npgsql.NpgsqlCommand.ExecuteReader(CommandBehavior behavior, Boolean async, CancellationToken cancellationToken) at Npgsql.NpgsqlCommand.ExecuteNonQuery(Boolean async, CancellationToken cancellationToken) at Npgsql.NpgsqlCommand.ExecuteNonQuery() at .(DbCommand , BulkOperation , Int32 ) at .( , DbCommand ) at .Execute(List1 actions) at .(List1 ) at Z.BulkOperations.BulkOperation.Execute() at Z.BulkOperations.BulkOperation.BulkMerge() at Z.Dapper.Plus.DapperPlusAction.Execute() at Z.Dapper.Plus.DapperPlusActionSet1..ctor(BaseDapperPlusActionSet oldActionSet, String mapperKey, DapperPlusActionKind actionKind, IEnumerable1 items) at Z.Dapper.Plus.DapperPlusActionSet1.BulkMerge[T](String mapperKey, T[] items) at Z.Dapper.Plus.DapperPlusActionSet1.BulkMerge[T](T[] items) at innRoad.Reports.Data.Repository.Replication.DapperPlusGenericRepository`1.<>c__DisplayClass3_1.b2(DapperPlusActionSet1 x) in /home/jenkins/agent/workspace/bernetes_PMS_Reports-API_develop/src/innRoad.Reports.Data/Repository/Replication/DapperPlusGenericRepository.cs:line 60 at Z.Dapper.Plus.DapperPlusExtensions..() at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state) --- End of stack trace from previous location --- at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) --- End of stack trace from previous location --- at innRoad.Reports.Data.Repository.Replication.DapperPlusGenericRepository1.SaveAll(IEnumerable`1 data, Boolean IsNewDataOnly) in /home/jenkins/agent/workspace/bernetes_PMS_Reports-API_develop/src/innRoad.Reports.Data/Repository/Replication/DapperPlusGenericRepository.cs:line 60

   public async Task SaveAll(IEnumerable<T> data, bool IsNewDataOnly)
    {
        if (!data.Any())
        {
            return;
        }

        using (var connection = _connectionFactory.GetPostgresServerConnection(true))
        {
            if (IsNewDataOnly)
            {
                await connection.BulkActionAsync(x => x.BulkInsert(data));
            }
            else
            {
                connection.UseBulkOptions(options => options.InsertIfNotExists = true);

                foreach (var batch in data.Batch(_replicationSettings.BulkMergeBatchSize))
                {
                    var batchArray = batch.ToArray();
                    await connection.BulkActionAsync(x => x.BulkMerge(batchArray));
                }
            }
        }
    }

[JonathanMagnan]

Hello @dbenjimanraj ,

Thank you for reporting,

Tomorrow we are supposed to release a new version with new options (for testing). Another person is supposed to test it to see if that fixes the issue.

At this moment, it looks more like an issue inside Npgsql but we could be wrong on this.

I will give you an update at the beginning of next week.

Best Regards,

Jon

[dbenjimanraj]

Thank You Jon

[JonathanMagnan]

Hello @dbenjimanraj ,

A new version has been released today to better handle BatchTimeout.

I'm not sure if your issue is different than the other person but could you try to change the default timeout?

You can do it globally this way:

DapperPlusManager.MapperFactory = mapper => mapper.BatchTimeout(600);

Let me know if that worked or you still have the same issue.

Best Regards,

Jon

[JonathanMagnan]

Hello @dbenjimanraj ,

Since our last conversation, we haven't heard from you!

Did you get the chance to change the default timeout?

Let me know if that worked or you still have the same issue.

Looking forward to hearing from you,

Jon

[JonathanMagnan]

Hello again!

A simple reminder that we are here to assist you!

Feel free to contact us once you change the default timeout.

Best regards,

Jon

[xavidram]

When trying the BulkActionAsync I also recieve the Exception while reading from stream error after the action times out. I am not able to catch any sql errors. Any ETA on the shiping of BulkInsertAsync? Possibly also AlsoBulkInsertAsync

[JonathanMagnan]

Hello @xavidram ,

Have you modified the timeout value?

For async method, this is currently possible via the BulkActionAsync such as connection.BulkActionAsync(x => x.BulkInsert(invoices));

There is no plan, for now, to add BulkInsertAsync due to how our chaining works but it might eventually come.

[xavidram]

I have now. I have set both a connection timeout and a command timeout. The image in my initial post runs until the timeout occurs. Then after a few minutes I refresh the table and the data is inserted.

Am I able to retrieve the identity from the bulkinsert within the async call?

[JonathanMagnan]

Timeout

So you still have a timeout issue even when increasing the BatchTimeout such as:

DapperPlusManager.MapperFactory = mapper => mapper.BatchTimeout(600);

Modifying it in the connection or command will not change it. You need to increase the default BatchTimeout our library uses.

Identity

If you want to retrieve the identity value, you should add it to your mapping and it will be automatically populated in your attendent entity: https://dapper-plus.net/map#mapper-identity

DapperPlusManager.Entity<Attendant>()
                 .Identity(x => x.AttendantID);

[dbenjimanraj]

Hello again!

A simple reminder that we are here to assist you!

Feel free to contact us once you change the default timeout.

Best regards,

Jon

Hi Jon

BatchTimeout resolved my issue.

Thanks, Ben