Deadlocks when creating jobs

[MiloszKrajewski]

Unfortunately we didn't find a reliable way to reproduce this problem, but it seems to happen when of the servers is terminated but the others are still working. It might be related to https://github.com/arnoldasgudas/Hangfire.MySqlStorage/issues/56.

It starts with operations failing due to deadlocks (I sanitized stack trace a little bit):

Hangfire.BackgroundJobClientException: Background job creation failed. See inner exception for details. 
MySql.Data.MySqlClient.MySqlException: XA_RBDEADLOCK: Transaction branch was rolled back: deadlock was detected
at MySqlConnector.Core.ResultSet.ReadResultSetHeaderAsync(IOBehavior ioBehavior)
at MySql.Data.MySqlClient.MySqlDataReader.ActivateResultSet(ResultSet resultSet)
at MySql.Data.MySqlClient.MySqlDataReader.ReadFirstResultSetAsync(IOBehavior ioBehavior)
at MySql.Data.MySqlClient.MySqlDataReader.CreateAsync(MySqlCommand command, CommandBehavior behavior, ResultSetProtocol resultSetProtocol, IOBehavior ioBehavior)
at MySqlConnector.Core.TextCommandExecutor.ExecuteReaderAsync(String commandText, MySqlParameterCollection parameterCollection, CommandBehavior behavior, IOBehavior ioBehavior, CancellationToken cancellationToken)
at MySql.Data.MySqlClient.MySqlCommand.ExecuteNonQueryAsync(IOBehavior ioBehavior, CancellationToken cancellationToken)
at MySql.Data.MySqlClient.MySqlCommand.ExecuteNonQuery()
at MySqlConnector.Core.MySqlXaTransaction.ExecuteXaCommand(String statement)
at MySqlConnector.Core.MySqlXaTransaction.Rollback(Enlistment enlistment)
at System.Transactions.VolatileEnlistmentAborting.EnterState(InternalEnlistment enlistment)
at System.Transactions.VolatileEnlistmentActive.InternalAborted(InternalEnlistment enlistment)
at System.Transactions.TransactionStateAborted.EnterState(InternalTransaction tx)
at System.Transactions.TransactionStateActive.Rollback(InternalTransaction tx, Exception e)
at System.Transactions.Transaction.Rollback()
at System.Transactions.TransactionScope.InternalDispose()
at System.Transactions.TransactionScope.Dispose()
at Hangfire.MySql.MySqlStorage.UseTransaction[T](Func`2 func, Nullable`1 isolationLevel)
at Hangfire.MySql.MySqlStorage.UseTransaction(Action`1 action)
at Hangfire.Client.CoreBackgroundJobFactory.Create(CreateContext context)
at Hangfire.Client.BackgroundJobFactory.<>c__DisplayClass7_0.<CreateWithFilters>b__0()
at Hangfire.Client.BackgroundJobFactory.InvokeClientFilter(IClientFilter filter, CreatingContext preContext, Func`1 continuation)
at Hangfire.Client.BackgroundJobFactory.Create(CreateContext context)
at Hangfire.BackgroundJobClient.Create(Job job, IState state)
at Hangfire.BackgroundJobClient.Create(Job job, IState state)
at Hangfire.BackgroundJobClientExtensions.Schedule[T](IBackgroundJobClient client, Expression`1 methodCall, DateTimeOffset enqueueAt)

Later on, it finished with Too many connections (that's why I'm invoking issue #56) but I understand it is just a consequence of more and connections being deadlocked.

Deadlocked queries seems to be:

insert into `State` (JobId, Name, Reason, CreatedAt, Data) values ('8222071', 'Processing', NULL, timestamp('2019-01-28 14:14:18.845045'), '{"StartedAt":"2019-01-28T14:09:01.7185889Z","ServerId":"0cd4c9800987:d612e05a-ada5-4fb6-ae6a-feaf3fb9f89c","WorkerId":"9aea6139-8033-460c-afd2-ba18ad601f85"}')

[Recodify]

Having the exact same problem. I have a single server and single dashboard running and seeing this on small work loads. Nothing else is writing or reading to the db other than hangfire.

[yuzd]

does this project is over??