Most string value could efficiently be compressed to save valuable memory on Redis.
The problem arise especially when Hangfire uses batches and so it's particulraly exposed to process literally millions of very small jobs.
Not everything of course worth the compression time, let's agree on which to compress and how:
The Jobh HashValue i.e. hangfire:job: since many arguments and/or custom properties could be set
The history list i.e. hangfire:job:{jobid}:history since retries and output from the job could significantly takes a lot of memory
I'm not a compression expert so I'd like to receive comments from someone who is, my opinion is that compressing within the context of a single string isn't a viable solution.
Let's make an example. I use to decorate my job classes with an attribute that capture current principal info (name and claims) and store them as JobParameter in each job in order to restore security info during job execution.
Each job hashset so usually contains at least the claims of the enqueing user:
along with other security relevants ones.
Giving that a single user could literally enqueue tens of thousands jobs, I think that compressing the string "one by one" wouldn't be as efficient as having a sort of "shared dictionary" for all jobs.
However, I didn't do a single test till now, neither I know if this is reasonable at all.
Most string value could efficiently be compressed to save valuable memory on Redis. The problem arise especially when Hangfire uses batches and so it's particulraly exposed to process literally millions of very small jobs. Not everything of course worth the compression time, let's agree on which to compress and how:
I'm not a compression expert so I'd like to receive comments from someone who is, my opinion is that compressing within the context of a single string isn't a viable solution. Let's make an example. I use to decorate my job classes with an attribute that capture current principal info (name and claims) and store them as JobParameter in each job in order to restore security info during job execution. Each job hashset so usually contains at least the claims of the enqueing user:
along with other security relevants ones. Giving that a single user could literally enqueue tens of thousands jobs, I think that compressing the string "one by one" wouldn't be as efficient as having a sort of "shared dictionary" for all jobs. However, I didn't do a single test till now, neither I know if this is reasonable at all.