Concurrency Implementation with Hangfire

[vpatil007]

We are having problems with accessing the LiteDB database with Hangfire implementation. We are using Hangfire.LiteDB (0.3.0) and looks like for concurrent access database is getting locked, which results in crashing the Hangfire jobs.

I am referring your documentation here and thread safe implementation is something what i am looking for.

Could you share an example of how to implement concurrency with hangfire jobs using ReaderWriterLockSlim?

[diegolbrito]

I have the same problem

[mbdavid]

Hi guys, there are 2 types of concurrency control in LiteDB v4:

• Across multiple threads: lock are controlled by ReaderWriterLockSlim class (from .NET Threading)
• Across multiple process: using FileStream.Lock and Unlock

But I don't how Hangfire implements LiteDB access. Do you have any error log to check?

[diegolbrito]

@mbdavid yes, it follows: 2020-01-04 07:20:55 -03:00 ERR] Unable to clean up locks on the resource 'HangFire:locks:schedulepoller'. LiteDB.LiteException: Timeout. Database is locked for more than 00:10:00. em LiteDB.LockService.Write() em LiteDB.LiteEngine.Transaction[T](String collection, Boolean addIfNotExists, Func2 action) em LiteDB.LiteEngine.Delete(String collection, Query query) em LiteDB.LiteCollection1.Delete(Query query) em Hangfire.LiteDB.LiteDbDistributedLock.Cleanup() Hangfire.LiteDB.LiteDbDistributedLock 2020-01-04 07:30:54 -03:00 ERR] Execution Worker is in the Failed state now due to an exception, execution will be retried no more than in 00:00:04 Hangfire.Processing.BackgroundExecution LiteDB.LiteException: Timeout. Database is locked for more than 00:10:00. em LiteDB.LockService.Read() em LiteDB.LiteEngine.<Find>d__9.MoveNext() em LiteDB.LiteEngine.<Find>d__13.MoveNext() em LiteDB.LiteCollection1.d__17.MoveNext() em System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable1 source) em Hangfire.LiteDB.LiteDbJobQueue.Dequeue(String[] queues, CancellationToken cancellationToken) em Hangfire.LiteDB.LiteDbConnection.FetchNextJob(String[] queues, CancellationToken cancellationToken) em Hangfire.Server.Worker.Execute(BackgroundProcessContext context) em Hangfire.Server.BackgroundProcessDispatcherBuilder.ExecuteProcess(Guid executionId, Object state) em Hangfire.Processing.BackgroundExecution.Run(Action2 callback, Object state) 2020-01-04 07:31:43 -03:00 ERR] Server dsp2029_101:6880:3f6bc590 will be restarted due to server time out Hangfire.Server.ServerHeartbeatProcess 2020-01-04 07:40:56 -03:00 ERR] Unable to clean up locks on the resource 'HangFire:recurring-jobs:lock'. LiteDB.LiteException: Timeout. Database is locked for more than 00:10:00. em LiteDB.LockService.Write() em LiteDB.LiteEngine.Transaction[T](String collection, Boolean addIfNotExists, Func2 action) em LiteDB.LiteEngine.Delete(String collection, Query query) em LiteDB.LiteCollection1.Delete(Query query) em Hangfire.LiteDB.LiteDbDistributedLock.Cleanup() Hangfire.LiteDB.LiteDbDistributedLock`

[mbdavid]

LiteDB timeout occurs when you have a thread that writes on database and keeps open more than 1 minute... while another thread are trying to read (or write too) database. Looks an IEnumerable fetch that are taking too long time or are not disposing correctly

[diegolbrito]

Okay, thanks for the help. For now I solved the problem by isolating hangFire server in another database.