search

marcoCasamento / Hangfire.Redis.StackExchange

HangFire Redis storage based on original (and now unsupported) Hangfire.Redis but using lovely StackExchange.Redis client
Other
460 stars 109 forks source link

How to improve performance? #104

Closed int2e closed 3 years ago

int2e commented 3 years ago

redis server cpu: i3 9100 mem size: 32GB ip: 192.168.1.2 / 10gb port

hangfire server cpu: i9 10900k mem size: 64GB ip: 192.168.1.3 / 10gb port workcount: 20-100 no difference

runtime .net 5.0

The current processing speed does not exceed 2000 / s image

marcoCasamento commented 3 years ago

how many workers on that instance ? (you can see that the number on the server page)

int2e commented 3 years ago

how many workers on that instance ? (you can see that the number on the server page)

workcount = 20 Can't reach 2000/s workcount = 100 Can't reach 2000/s

image

marcoCasamento commented 3 years ago

the number "12" you're seeing here is not an useful indicator for such quick jobs. It represents the number of current running jobs at any given moment. Hangfire statistics, though, are a valid point. I've never dealt with such numbers as my job usually take at least 100ms so the server would cap way early than 2000 job/sec. I can suggest to look at the method FetchNextJob that along with the redis pub/sub in the RedisScubscription is responsible of Fetching jobs.

As a mean of having a grasp of redis performance, it COULD be useful to fire up a couple of instances on the same hangfire server

However, with such a great number of jobs, it seems that you're using hangfire as a messagebus for which purpose I'd look elsewhere.

int2e commented 3 years ago

the number "12" you're seeing here is not an useful indicator for such quick jobs. It represents the number of current running jobs at any given moment. Hangfire statistics, though, are a valid point. I've never dealt with such numbers as my job usually take at least 100ms so the server would cap way early than 2000 job/sec. I can suggest to look at the method FetchNextJob that along with the redis pub/sub in the RedisScubscription is responsible of Fetching jobs.

As a mean of having a grasp of redis performance, it COULD be useful to fire up a couple of instances on the same hangfire server

However, with such a great number of jobs, it seems that you're using hangfire as a messagebus for which purpose I'd look elsewhere.

hi，I want to achieve 20k-200k tasks per second, I don’t think I can do it! :)

marcoCasamento commented 3 years ago

I don't think either. However, there's a paid option for redis storage directly from hangfire site, maybe that version is able to achieve better performance

int2e commented 3 years ago

ok, thanks!