Many Hangfire servers on Cosmos DB, problems with processing jobs

imranmomin / Hangfire.AzureCosmosDb

Azure Cosmos DB storage provider for Hangfire

https://www.hangfire.io/

MIT License

18 stars 17 forks source link

Many Hangfire servers on Cosmos DB, problems with processing jobs #16

Closed robertgm closed 2 years ago

robertgm commented 3 years ago

When we have many Hangfire servers and add a lot of jobs for example 100, there is a problem with processing these jobs. When we have the only one Hangfire server there is no problem. When we change Hangfire server on SQL storage and add a lot of jobs there is no problem with processing.

Jobs aren't enqueued somethimes after retry status:

Counters are wrong:

Retry Screen -jobs aren't processed: job that has retry status is taken and processed by many Hangfire servers

MikeCardiff commented 3 years ago

I'm also experiencing this issue. Scheduled retry jobs remain scheduled forever until enqueued manually.

Some jobs are also being processed by multiple servers causing an exception when they both try to save to the same Azure table causing a Etag conflict:

imranmomin commented 3 years ago

@MikeCardiff - You will a Cosmos DB using SQL API. I'm not sure if it will work on Azure Table