search

HangfireIO / Hangfire

An easy way to perform background job processing in .NET and .NET Core applications. No Windows Service or separate process required
https://www.hangfire.io
Other
9.42k stars 1.7k forks source link

Multiple workers executing the same job SQLite #1674

Open RogueCookie opened 4 years ago

RogueCookie commented 4 years ago

Hello i have issue that all 20 workers execute the same 1 job. Ok I can not to pay attention on console but it also insert in table Job this job sometimes twice sometimes once. I have no idea how to restrict it.Could you help me with it? I use Hangfire 1.7.11 image image image why I have this result image in Counter table I have new records from 10-17 from each execution

pieceofsummer commented 4 years ago

Looks like a very typical race condition in SQLite distributed lock implementation. Try using a different storage if possible.

bsoulier commented 4 years ago

@pieceofsummer I Just noticed the exact same behavior on running jobs that gets started multiple times (using SQL Server storage)

Here is a screenshot that shows Hangfire.Console showing a starting job Writeline that is only found once in code:

image

RogueCookie commented 4 years ago

@pieceofsummer I Just noticed the exact same behavior on running jobs that gets started multiple times (using SQL Server storage)

Here is a screenshot that shows Hangfire.Console showing a starting job Writeline that is only found once in code:

image

I check my app with sql sqrver and it's work great but my manager wont's sqlite and I don't know how to avoid it. image uf i added it everything is ok. and work as I want but I loose parallelism

RogueCookie commented 4 years ago

Looks like a very typical race condition in SQLite distributed lock implementation. Try using a different storage if possible.

Theres a lot of discussion going on it from 2017 and i suppose someone found way hoe to avoid

pieceofsummer commented 4 years ago

I guess you can just patch the SQLite storage provider and wrap distributed lock acquiring (the part where it checks for existing row and updates/inserts it) into a lock statement to avoid race conditions. It is not a very good solution concurrency-wise, but it would work with minimal changes.

pieceofsummer commented 4 years ago

@pieceofsummer I Just noticed the exact same behavior on running jobs that gets started multiple times (using SQL Server storage)

Here is a screenshot that shows Hangfire.Console showing a starting job Writeline that is only found once in code:

image

It is not started multiple times, as there’s only one Processing state. Each job start would generate a new state, and all Console writes within a Processing state are created by one running job.

As of why there’s multiple lines, it is hard to tell without seeing your code.

RogueCookie commented 4 years ago

lock acquiring

Sorry I'm a new in this area could you give more details or referens/exemples how to do it?