search

marcoCasamento / Hangfire.Redis.StackExchange

HangFire Redis storage based on original (and now unsupported) Hangfire.Redis but using lovely StackExchange.Redis client
Other
456 stars 109 forks source link

Move job related data to disk before they expires #10

Closed marcoCasamento closed 8 years ago

marcoCasamento commented 9 years ago

Assuming you have millions of job a days that runs on your hangfire instances (and that quickly become true when you start using batches) retention time longer than a day could take significant amount of space on your Redis instance.

Moving jobs executions details on slow and cheaper storages like RDBMS or plain filesystem would free retention policy to fill up expensive RAM on Redis.

The RedisWriteOnlyTransaction.ExpireJob method seems to be a good extension point for that approach.

public override void ExpireJob(string jobId, TimeSpan expireIn)
    {
         _transaction.KeyExpireAsync(
            string.Format(RedisStorage.Prefix + "job:{0}", jobId),
            expireIn);

         _transaction.KeyExpireAsync(
            string.Format(RedisStorage.Prefix + "job:{0}:history", jobId),
            expireIn);

         _transaction.KeyExpireAsync(
            string.Format(RedisStorage.Prefix + "job:{0}:state", jobId),
            expireIn);
    }