Succesful jobs are not cleaned from storage

[xumix]

I have 900 000 Succeeded jobs, the list is limited to 10 000 but actual storage is not cleaned This actually prevent my from using GUI tools for redis, since the number of keys and data is so large

[marcoCasamento]

the list limit doesn't govern the amount of jobs that you keep on the storage. it simply set how many jobs are visible from the dashboard. Jobs are cleaned up by expiring them, and the default retention time is 1 day. You can refer to https://discuss.hangfire.io/t/how-to-configure-the-retention-time-of-job/34 in order to change it.

I actually keep jobs for 7 days and that means near to 5mln keys in Redis. Yep, this make tools like redisdesktop almost useless, you may want to take a look at https://github.com/RedisLabsModules/RediSearch

[xumix]

Well, hangfire:*:succeeded list is being cleaned, but not the job keys themself

[marcoCasamento]

exact, this is the expected behaviour. Use JobExpriationTime to cleanup jobs.

[xumix]

@marcoCasamento thanks for fast answer! But I have default expiration time set, which means 1 day

[marcoCasamento]

It'a TimeSpan, set it to whatever make sense for you. I recommend to don't go lower than 1 hour since it would clash with the JobInvisibility. Anything more than 1 hour would be fine.

But... you really want to cleanup jobs this fast ?

[marcoCasamento]

one more thing. The JobExpirationTimeOut is set on job creation, so the jobs that you already have in the storage wouldn't be affected. You can use EXPIRE or DEL command along with SCAN in a LUA script to expire your hangfire:job:* keys (or whatever your namespace is)

[xumix]

@marcoCasamento It is ok for me to clean successful jobs in 1 day, but is looks like they are not expired at all. The number of jobs (and job keys in redis) constantly increases over time, which indicates no retention is made. I can see, that you get the succeeded list, parse it and then perform retention, but if the list is limited to 10 000 (via config), it means that if I have 100 000 jobs executed daily then 90 000 of then will never be cleaned up

[marcoCasamento]

No, I see a couple of misconceptions here:

• The hangfire succeded list is governed by the list size of the storage, that you set to 10.000 and I can see is actaully correct with 10.001 (well... off by one actually 😄 )
• The Succeeded number that you see in the dashboard (823252 in red square in your last image) is simply a statistic number. Is stored in hangfire:stats:succeeded and incremented every time a new job succeed.
• The lists and hashes in namespace hangfire:job:* (last image) are the actual jobs that are stored in your database and they will expire according to your JobExpirationTimeOut. Just take a look at the TTL of one of them (simply TTL hangfire:PpProd:Job:xxxxx....) it express how many seconds the key will still be in your redis db

[xumix]

@marcoCasamento Thanks, I see now! You actually manage expiration via redis TTL, that is great 👍

[chrismcv]

@marcoCasamento - on a similar thread - is it intentional that failed/retrying jobs don't have a TTL? Is there a way these get cleaned up? (or is it a bug?)

[marcoCasamento]

@chrismcv yes, it's intentional. In the author of Hangfire view, only jobs that reach a "final state" can be cleaned up, and by default only "Succeeded" and "Deleted" are considered final, so no TTL have to be put on jobs that are not in that state. I believe that you can customize states and maybe you could consider as "final" the "failed" jobs, as far as I know you can customize the state machine and even add further states to it, that however is well beyond the scope of this repo, I suggest you to file a question on the hangfire repo

[FixRM]

@xumix, hi! Can you please share the solution? I'm pretty new with Redis, so will be grateful for direct guidance or documentation links.

[xumix]

@FixRM There is no solution, because the problem does not exist, Job keys expire with Redis TTL mechanics Use proper JobExpriationTime and in will work as expected