search

HangfireIO / Hangfire

An easy way to perform background job processing in .NET and .NET Core applications. No Windows Service or separate process required
https://www.hangfire.io
Other
9.35k stars 1.69k forks source link

Hangfire.Redis job queue is huge #896

Open aidanmorgan opened 7 years ago

aidanmorgan commented 7 years ago

I am using the latest Hangfire.Redis package and in my test environment the queue size for the Job queue is at 1596020 entries.

Is there a way to truncate this queue to reduce the Redis memory usage? I've already had to move up a tier of Azure Redis to have my app work correctly.

I have set the queue length properties in the settings, however it appears to have had no impact.

aidanmorgan commented 7 years ago

I have the following settings:

            MaxDeletedListLength = 1000,
            MaxSucceededListLength = 1000,
aidanmorgan commented 7 years ago

Does anyone have a LUA script or some source code that might be able to clean out the older entries in the queues?

odinserj commented 7 years ago

Oh, sorry for the delay. What is the number of background jobs do you have? 1.5M enqueued jobs will cause 1.5M entries in job queue, and that's fine. If the number of background jobs is less than that, then there is a problem, because Hangfire contain no logic that will grow your memory usage in Redis. Have you performed manual cleanup of your Redis instance? For example, by removing jobs by using Redis client, instead of Hangfire's API?

aidanmorgan commented 7 years ago

@odinserj do you have a script that can be used to remove completed/expired jobs from the queue?

odinserj commented 7 years ago

Could you send me the output of the Redis' INFO command?

aidanmorgan commented 7 years ago

This is from my test envionment (production is significantly larger and takes forever):

INFO (db1 is my Hangfire db):

db0:keys=894,expires=894,avg_ttl=11445864167 db1:keys=113287,expires=33786,avg_ttl=42789842 db2:keys=1,expires=0,avg_ttl=0

I used the memory analyser from here (https://github.com/gamenet/redis-memory-analyzer/) and it shows my usage as:

image

aidanmorgan commented 7 years ago

I think it's the history that is killing my memory usage. I do have some job arguments that contain quite a lot of serialized json data (id's for batch processing), so whilst it's good to have the job history (success etc.) available, I need to get rid of the argument data to reduce the memory pressure.

odinserj commented 7 years ago

268 MB for 6K of the job:*:history keys is too much. Expected value is currently about 5 MB. Could you tell me whether your background jobs contain a return statement with a complex result? It will be serialized, and if are returning a huge list, or huge strings, this will cause memory issues. Job arguments are serialized to an another key, job:*, and it's a hash.

Nevertheless, I don't understand yet what the problem is: high memory usage or long queue. I can turn on my telepathy skills, but they require the output of the INFO command from your production server ☺️ It doesn't iterate through all the keys and is safe enough to be used even when you have a lot of keys (unlike rma).

aidanmorgan commented 7 years ago

Sorry - it's the memory usage that is my fundamental problem.

It is related to the queue length because I have jobs that take in a large amount of serialised Json data (can be up to 1Mb of Job argument data), I was expecting the queue length settings to remove the history once the queue exceeded the maximum length provided.

The INFO from my production server (again db1) is:

Keyspace db0:keys=9,expires=0,avg_ttl=0 db1:keys=6151124,expires=254075,avg_ttl=76066991 db2:keys=1,expires=0,avg_ttl=0

odinserj commented 7 years ago

Not a problem, so let's examine the reasons. Hangfire is about background jobs, not about queues. Queues contain only background job identifiers, and nothing more. So large arguments does not affect queue size at all. Could you also show me the Memory block of the INFO command's output (again, from production, please see the example below):

# Memory
used_memory:17278176
used_memory_human:16.48M
used_memory_rss:17240264
used_memory_rss_human:16.44M
used_memory_peak:17358120
used_memory_peak_human:16.55M
total_system_memory:0
total_system_memory_human:0B
used_memory_lua:39936
used_memory_lua_human:39.00K
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
mem_fragmentation_ratio:1.00
mem_allocator:jemalloc-3.6.0

Also, I need to see your serialized background jobs. Information from your testing environment is enough. Please run the following commands against a background job identifier, whose contents are suspected to be large enough to cause the high memory issue, with the corresponding prefix:

HGETALL <prefix>:job:<jobid>
HGETALL <prefix>:job:state:<jobid>
LRANGE <prefix>:job:history:<jobid> 0 -1
odinserj commented 7 years ago

Sorry, the correct commands are:

HGETALL <prefix>:job:<jobid>
HGETALL <prefix>:job:<jobid>:state
LRANGE <prefix>:job:<jobid>:history 0 -1
aidanmorgan commented 7 years ago 
# Memory
used_memory:5219148136
used_memory_human:4.86G
used_memory_rss:6124490752
used_memory_rss_human:5.70G
used_memory_peak:5995279904
used_memory_peak_human:5.58G
used_memory_lua:56320
maxmemory:6100000000
maxmemory_reservation:4000000
maxfragmentationmemory_reservation:0
maxmemory_human:5.68G
maxmemory_policy:volatile-lru
mem_allocator:jemalloc-3.6.0

1 (with some censoring):

HGETALL XXX.job:7fb36d33-773d-4f59-bcb2-41cfef2366fe

 1)  "ParameterTypes"
 2)  [XXXArgs, XXX.Application, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"]"
 3)  "Method"
 4)  "Execute"
 5)  "Type"
 6)  "XXXBatchJob, XXX.Application, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null"
 7)  "State"
 8)  "Processing"
 9)  "Arguments"
 10)  "["{\"ids\":[417614,416839,416842,416804,416807,416809,416811,416813,416815,416828,26111,26085,26083,340038,417615,333392,120123,26279,26384,26359,26356,116780,333393,417618,340077,31012,31021,31022,381295,28732,28731,28722,340087,381298,417619,333446,29171,29180,29183,29208,30302,30301,30291,333474,417621,333481,30571,30580,30583,30608,30702,30701,30692,340135,128582,417624,333503,31331,31341,31342,117754,31382,31380,340155,123723,417626,333507,31491,31501,31502,32248,32221,32212,333522,417628,340176,32251,32260,32262,301599,32302,32301,32292,340177,417629,333529,117048,32421,32423,32448,33262,33253,340203,417631,333590,301680,34732,34741,34769,34889,34863,34862,34853,131337,333593,417634,340248,35056,35062,35063,35089,35024,35021,35012,340249,417635,333657,127079,37256,37265,37268,37453,37427,37426,37417,333661,417638,340314,37497,37506,37507,37533,37653,37627,37626,37621,340317,417639,333674,184265,37946,37947,37973,38013,37985,37976,145724,333675,417642,340328,38060,38066,38067,38253,38225,38217,340332,417643,333685,76785,38377,38413,38668,38666,38657,340344,417645,333695,38740,38737,38746,38773,39028,39026,39017,340353,417647,333711,51817,41142,41148,41151,41153,41179,39413,39387,39385,39376,340363,417649,333716,123657,39496,39506,39508,39573,39547,39545,39544,39536,39541,340368,417651,333723,39780,39786,41831,41805,41803,41795,333778,417653,43426,340467,43386,43396,43398,44315,44289,44287,340488,44318,417656,333931,48363,48374,48663,48637,48635,48627,340588,48666,417658,333940,48758,48767,48769,48795,49760,49758,49749,340613,49788,417660,333983,50748,50757,50760,51807,51781,51779,51770,334004,417661,56560,340753,56521,56530,56532,59954,59928,59926,59918,340834,59957,417664,334187,60105,60109,60115,67079,67052,67046,67043,67048,334338,417665,68915,341032,68875,68879,68885,68912,69686,69685,69683,69679,69680,334398,417667,70567,341067,70527,70537,70564,70539,76876,76849,76843,76839,76845,334559,417669,77916,341234,77883,77877,77881,77885,77913,82729,82728,82726,82722,82718,341342,82757,417672,334716,83893,83897,83902,93987,93960,93954,93951,93957,334937,417673,97703,341676,97671,97664,97668,97673,101131,101104,101098,101094,101100,101101,335104,417675,102262,341778,102230,102222,102226,102232,102259,102783,102756,102750,102746,102752,102753,335140,417677,107212,341888,107178,107172,107176,107182,107209,417679,417681,335294,111652,111651,111649,111645,111641,111648,112211,112210,112204,112208,112214,112241,112244,341959,417683,335330,113109,113108,113106,113102,113098,113105,118646,118645,118639,118643,118649,118650,118678,342009,417685,335413,116134,116133,116131,116127,116123,116130,116680,116679,116673,116677,116683,116709,116712,342084,417687,335436,116852,116851,116849,116845,116841,116848,120565,120564,120558,120562,120568,120597,342152,417689,335504,120744,120720,120717,120713,120709,120716,122150,122149,122143,122147,122153,122179,335530,417690,342182,122090,122063,122060,122056,122052,122059,122339,122338,122332,122336,122342,122343,122371,342187,417693,335540,122597,122573,122570,122566,122562,122569,123026,123025,123019,123023,123055,335549,417694,342239,127380,127352,127351,127345,127341,127347,127348,131064,131057,131061,131065,131068,131082,131094,335662,417696,342336,132164,132161,132137,132134,132130,132126,132133],\"processingTimeSinceEpoch\":1496135703,\"unreplied\":[28905,28906,28962],\"notificationCounts\":{\"6289_2017-05-30\":{\"UserId\":6289,\"year\":2017,\"month\":5,\"day\":30,\"count\":2,\"key\":\"6289_2017-05-30\"},\"20065_2017-05-30\":{\"UserId\":20065,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"20065_2017-05-30\"},\"20189_2017-05-30\":{\"UserId\":20189,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"20189_2017-05-30\"},\"20263_2017-05-30\":{\"UserId\":20263,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"20263_2017-05-30\"},\"20292_2017-05-30\":{\"UserId\":20292,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"20292_2017-05-30\"},\"20788_2017-05-30\":{\"UserId\":20788,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"20788_2017-05-30\"},\"20940_2017-05-30\":{\"UserId\":20940,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"20940_2017-05-30\"},\"21000_2017-05-30\":{\"UserId\":21000,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"21000_2017-05-30\"},\"21185_2017-05-30\":{\"UserId\":21185,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"21185_2017-05-30\"},\"21293_2017-05-30\":{\"UserId\":21293,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"21293_2017-05-30\"},\"21729_2017-05-30\":{\"UserId\":21729,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"21729_2017-05-30\"},\"21839_2017-05-30\":{\"UserId\":21839,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"21839_2017-05-30\"},\"21896_2017-05-30\":{\"UserId\":21896,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"21896_2017-05-30\"},\"22103_2017-05-30\":{\"UserId\":22103,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"22103_2017-05-30\"},\"22133_2017-05-30\":{\"UserId\":22133,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"22133_2017-05-30\"},\"22138_2017-05-30\":{\"UserId\":22138,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"22138_2017-05-30\"},\"22288_2017-05-30\":{\"UserId\":22288,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"22288_2017-05-30\"},\"22290_2017-05-30\":{\"UserId\":22290,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"22290_2017-05-30\"},\"22444_2017-05-30\":{\"UserId\":22444,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"22444_2017-05-30\"},\"22456_2017-05-30\":{\"UserId\":22456,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"22456_2017-05-30\"},\"22745_2017-05-30\":{\"UserId\":22745,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"22745_2017-05-30\"},\"22760_2017-05-30\":{\"UserId\":22760,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"22760_2017-05-30\"},\"22773_2017-05-30\":{\"UserId\":22773,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"22773_2017-05-30\"},\"22813_2017-05-30\":{\"UserId\":22813,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"22813_2017-05-30\"},\"22868_2017-05-30\":{\"UserId\":22868,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"22868_2017-05-30\"},\"22871_2017-05-30\":{\"UserId\":22871,\"year\":2017,\"month\":5,\"day\":30,\"count\":2,\"key\":\"22871_2017-05-30\"},\"22925_2017-05-30\":{\"UserId\":22925,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"22925_2017-05-30\"},\"23021_2017-05-30\":{\"UserId\":23021,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"23021_2017-05-30\"},\"23109_2017-05-30\":{\"UserId\":23109,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"23109_2017-05-30\"},\"23138_2017-05-30\":{\"UserId\":23138,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"23138_2017-05-30\"},\"23139_2017-05-30\":{\"UserId\":23139,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"23139_2017-05-30\"},\"23174_2017-05-30\":{\"UserId\":23174,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"23174_2017-05-30\"},\"23211_2017-05-30\":{\"UserId\":23211,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"23211_2017-05-30\"},\"23271_2017-05-30\":{\"UserId\":23271,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"23271_2017-05-30\"},\"23352_2017-05-30\":{\"UserId\":23352,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"23352_2017-05-30\"},\"23525_2017-05-30\":{\"UserId\":23525,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"23525_2017-05-30\"},\"23731_2017-05-30\":{\"UserId\":23731,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"23731_2017-05-30\"},\"23938_2017-05-30\":{\"UserId\":23938,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"23938_2017-05-30\"},\"23971_2017-05-30\":{\"UserId\":23971,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"23971_2017-05-30\"},\"23974_2017-05-30\":{\"UserId\":23974,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"23974_2017-05-30\"},\"24059_2017-05-30\":{\"UserId\":24059,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"24059_2017-05-30\"},\"24246_2017-05-30\":{\"UserId\":24246,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"24246_2017-05-30\"},\"24384_2017-05-30\":{\"UserId\":24384,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"24384_2017-05-30\"},\"24602_2017-05-30\":{\"UserId\":24602,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"24602_2017-05-30\"},\"24854_2017-05-30\":{\"UserId\":24854,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"24854_2017-05-30\"},\"25116_2017-05-30\":{\"UserId\":25116,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"25116_2017-05-30\"},\"26329_2017-05-30\":{\"UserId\":26329,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"26329_2017-05-30\"},\"26492_2017-05-30\":{\"UserId\":26492,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"26492_2017-05-30\"},\"27346_2017-05-30\":{\"UserId\":27346,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"27346_2017-05-30\"},\"28105_2017-05-30\":{\"UserId\":28105,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28105_2017-05-30\"},\"28626_2017-05-30\":{\"UserId\":28626,\"year\":2017,\"month\":5,\"day\":30,\"count\":3,\"key\":\"28626_2017-05-30\"},\"28649_2017-05-30\":{\"UserId\":28649,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28649_2017-05-30\"},\"28650_2017-05-30\":{\"UserId\":28650,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28650_2017-05-30\"},\"28653_2017-05-30\":{\"UserId\":28653,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28653_2017-05-30\"},\"28664_2017-05-30\":{\"UserId\":28664,\"year\":2017,\"month\":5,\"day\":30,\"count\":2,\"key\":\"28664_2017-05-30\"},\"28674_2017-05-30\":{\"UserId\":28674,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28674_2017-05-30\"},\"28764_2017-05-30\":{\"UserId\":28764,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28764_2017-05-30\"},\"28767_2017-05-30\":{\"UserId\":28767,\"year\":2017,\"month\":5,\"day\":30,\"count\":2,\"key\":\"28767_2017-05-30\"},\"28771_2017-05-30\":{\"UserId\":28771,\"year\":2017,\"month\":5,\"day\":30,\"count\":2,\"key\":\"28771_2017-05-30\"},\"28777_2017-05-30\":{\"UserId\":28777,\"year\":2017,\"month\":5,\"day\":30,\"count\":2,\"key\":\"28777_2017-05-30\"},\"28778_2017-05-30\":{\"UserId\":28778,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28778_2017-05-30\"},\"28780_2017-05-30\":{\"UserId\":28780,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28780_2017-05-30\"},\"28785_2017-05-30\":{\"UserId\":28785,\"year\":2017,\"month\":5,\"day\":30,\"count\":2,\"key\":\"28785_2017-05-30\"},\"28787_2017-05-30\":{\"UserId\":28787,\"year\":2017,\"month\":5,\"day\":30,\"count\":2,\"key\":\"28787_2017-05-30\"},\"28792_2017-05-30\":{\"UserId\":28792,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28792_2017-05-30\"},\"28794_2017-05-30\":{\"UserId\":28794,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28794_2017-05-30\"},\"28799_2017-05-30\":{\"UserId\":28799,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28799_2017-05-30\"},\"28801_2017-05-30\":{\"UserId\":28801,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28801_2017-05-30\"},\"28806_2017-05-30\":{\"UserId\":28806,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28806_2017-05-30\"},\"28811_2017-05-30\":{\"UserId\":28811,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28811_2017-05-30\"},\"28813_2017-05-30\":{\"UserId\":28813,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28813_2017-05-30\"},\"28815_2017-05-30\":{\"UserId\":28815,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28815_2017-05-30\"},\"28816_2017-05-30\":{\"UserId\":28816,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28816_2017-05-30\"},\"28821_2017-05-30\":{\"UserId\":28821,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28821_2017-05-30\"},\"28826_2017-05-30\":{\"UserId\":28826,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28826_2017-05-30\"},\"28828_2017-05-30\":{\"UserId\":28828,\"year\":2017,\"month\":5,\"day\":30,\"count\":2,\"key\":\"28828_2017-05-30\"},\"28831_2017-05-30\":{\"UserId\":28831,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28831_2017-05-30\"},\"28835_2017-05-30\":{\"UserId\":28835,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28835_2017-05-30\"},\"28839_2017-05-30\":{\"UserId\":28839,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28839_2017-05-30\"},\"28842_2017-05-30\":{\"UserId\":28842,\"year\":2017,\"month\":5,\"day\":30,\"count\":2,\"key\":\"28842_2017-05-30\"},\"28844_2017-05-30\":{\"UserId\":28844,\"year\":2017,\"month\":5,\"day\":30,\"count\":2,\"key\":\"28844_2017-05-30\"},\"28849_2017-05-30\":{\"UserId\":28849,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28849_2017-05-30\"},\"28853_2017-05-30\":{\"UserId\":28853,\"year\":2017,\"month\":5,\"day\":30,\"count\":2,\"key\":\"28853_2017-05-30\"},\"28856_2017-05-30\":{\"UserId\":28856,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28856_2017-05-30\"},\"28858_2017-05-30\":{\"UserId\":28858,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28858_2017-05-30\"},\"28859_2017-05-30\":{\"UserId\":28859,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28859_2017-05-30\"},\"28864_2017-05-30\":{\"UserId\":28864,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28864_2017-05-30\"},\"28865_2017-05-30\":{\"UserId\":28865,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28865_2017-05-30\"},\"28870_2017-05-30\":{\"UserId\":28870,\"year\":2017,\"month\":5,\"day\":30,\"count\":2,\"key\":\"28870_2017-05-30\"},\"28872_2017-05-30\":{\"UserId\":28872,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28872_2017-05-30\"},\"28879_2017-05-30\":{\"UserId\":28879,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28879_2017-05-30\"},\"28880_2017-05-30\":{\"UserId\":28880,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28880_2017-05-30\"},\"28885_2017-05-30\":{\"UserId\":28885,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28885_2017-05-30\"},\"28887_2017-05-30\":{\"UserId\":28887,\"year\":2017,\"month\":5,\"day\":30,\"count\":2,\"key\":\"28887_2017-05-30\"},\"28889_2017-05-30\":{\"UserId\":28889,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28889_2017-05-30\"},\"28894_2017-05-30\":{\"UserId\":28894,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28894_2017-05-30\"},\"28895_2017-05-30\":{\"UserId\":28895,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28895_2017-05-30\"},\"28899_2017-05-30\":{\"UserId\":28899,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28899_2017-05-30\"},\"28902_2017-05-30\":{\"UserId\":28902,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28902_2017-05-30\"},\"28906_2017-05-30\":{\"UserId\":28906,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28906_2017-05-30\"},\"28924_2017-05-30\":{\"UserId\":28924,\"year\":2017,\"month\":5,\"day\":30,\"count\":2,\"key\":\"28924_2017-05-30\"},\"28928_2017-05-30\":{\"UserId\":28928,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28928_2017-05-30\"},\"28930_2017-05-30\":{\"UserId\":28930,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28930_2017-05-30\"},\"28931_2017-05-30\":{\"UserId\":28931,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28931_2017-05-30\"},\"28932_2017-05-30\":{\"UserId\":28932,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28932_2017-05-30\"},\"28933_2017-05-30\":{\"UserId\":28933,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28933_2017-05-30\"},\"28934_2017-05-30\":{\"UserId\":28934,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28934_2017-05-30\"},\"28936_2017-05-30\":{\"UserId\":28936,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28936_2017-05-30\"},\"28937_2017-05-30\":{\"UserId\":28937,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28937_2017-05-30\"},\"28938_2017-05-30\":{\"UserId\":28938,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28938_2017-05-30\"},\"28939_2017-05-30\":{\"UserId\":28939,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28939_2017-05-30\"},\"28941_2017-05-30\":{\"UserId\":28941,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28941_2017-05-30\"},\"28944_2017-05-30\":{\"UserId\":28944,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28944_2017-05-30\"},\"28947_2017-05-30\":{\"UserId\":28947,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28947_2017-05-30\"},\"28953_2017-05-30\":{\"UserId\":28953,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28953_2017-05-30\"},\"28954_2017-05-30\":{\"UserId\":28954,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28954_2017-05-30\"},\"28956_2017-05-30\":{\"UserId\":28956,\"year\":2017,\"month\":5,\"day\":30,\"count\":5,\"key\":\"28956_2017-05-30\"},\"28957_2017-05-30\":{\"UserId\":28957,\"year\":2017,\"month\":5,\"day\":30,\"count\":2,\"key\":\"28957_2017-05-30\"},\"28958_2017-05-30\":{\"UserId\":28958,\"year\":2017,\"month\":5,\"day\":30,\"count\":2,\"key\":\"28958_2017-05-30\"},\"28961_2017-05-30\":{\"UserId\":28961,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28961_2017-05-30\"},\"28962_2017-05-30\":{\"UserId\":28962,\"year\":2017,\"month\":5,\"day\":30,\"count\":5,\"key\":\"28962_2017-05-30\"},\"28963_2017-05-30\":{\"UserId\":28963,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28963_2017-05-30\"},\"28964_2017-05-30\":{\"UserId\":28964,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28964_2017-05-30\"},\"28965_2017-05-30\":{\"UserId\":28965,\"year\":2017,\"month\":5,\"day\":30,\"count\":4,\"key\":\"28965_2017-05-30\"},\"28966_2017-05-30\":{\"UserId\":28966,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28966_2017-05-30\"},\"28967_2017-05-30\":{\"UserId\":28967,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28967_2017-05-30\"},\"28968_2017-05-30\":{\"UserId\":28968,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"28968_2017-05-30\"}},\"deliverMessage\":true}"]"
 11)  "CreatedAt"
 12)  "2017-05-30T09:15:04.5483279Z"
 13)  "CurrentUICulture"
 14)  ""en-US""
 15)  "CurrentCulture"
 16)  ""en-US""

2)

HGETALL XXX.job:7fb36d33-773d-4f59-bcb2-41cfef2366fe:state
 1)  "StartedAt"
 2)  "2017-05-30T09:15:04.5796245Z"
 3)  "ServerId"
 4)  "YYY:4200:4e79b3aa-ca22-4550-8ca1-e96c818b899b"
 5)  "WorkerId"
 6)  "1a93d8f2-1132-4960-882e-8ccdf7f57b6b"
 7)  "State"
 8)  "Processing"

3)

LRANGE XXX.job:7fb36d33-773d-4f59-bcb2-41cfef2366fe:history 0 -1
 1)  "{"EnqueuedAt":"2017-05-30T09:45:29.1455099Z","Queue":"default","State":"Enqueued","Reason":"Background job was re-queued, because InvisibilityTimeout exceeded","CreatedAt":"2017-05-30T09:45:29.1455099Z"}"
 2)  "{"StartedAt":"2017-05-30T09:15:04.5796245Z","ServerId":"YYYY:4200:4e79b3aa-ca22-4550-8ca1-e96c818b899b","WorkerId":"1a93d8f2-1132-4960-882e-8ccdf7f57b6b","State":"Processing","Reason":null,"CreatedAt":"2017-05-30T09:15:04.6161749Z"}"
 3)  "{"EnqueuedAt":"2017-05-30T09:15:04.5483279Z","Queue":"default","State":"Enqueued","Reason":null,"CreatedAt":"2017-05-30T09:15:04.5664979Z"}"
odinserj commented 7 years ago

So you was right, payload size is huge, and 260K of jobs with the same argument size may cause memory consumption of 5GB (all calculations are naive, but more or less correct). You can reduce the memory consumption in one of the following ways:

  1. Configure the retention time of each job. By default it is 24 hours, but you can set it to 1 hour.
  2. Reduce the size of serialized payload. At least the following data can be minified:
{\"UserId\":20065,\"year\":2017,\"month\":5,\"day\":30,\"count\":1,\"key\":\"20065_2017-05-30\"}

To save 30%:

{\"u\":20065,\"d\":\"20170530\",\"c\":1,\"k\":\"20065_2017-05-30\"}

Or at least to save 20%:

{\"u\":20065,\"y\":2017,\"m\":5,\"d\":30,\"c\":1,\"k\":\"20065_2017-05-30\"}
  1. Save that information in another storage, and pass to Hangfire only identifiers.
aidanmorgan commented 7 years ago

Will setting the retention period cause the bulky data to be deleted from Redis after a period of time?

odinserj commented 7 years ago

Yes, that expiration time configures retention of a background job after it is succeeded or deleted. Hangfire.Pro.Redis calls the EXPIRE command, and Redis will delete those jobs just fine.

aidanmorgan commented 7 years ago

Is there a script or something that I can use to empty out the old values in the Redis cache once I've done this?

Should I just run a loop and run the EXPIRE command on every entry? I could mash something together, but I am concerned that because I don't know how Hangfire is using the cache that I will inadvertently break something.

odinserj commented 7 years ago

We couldn't perform this without iterating all the keys, and you already know it's a very expensive operation. We can get the list of succeeded jobs from the succeeded key, but it will contain only 1000 jobs, since MaxSucceededListLength setting is set. It's better to temporarily increase the amount of memory available to Redis to handle the workload, or, much better give Redis much more memory on a persistent basis (16GB or 32GB), since your workload is huge.

aidanmorgan commented 7 years ago

@odinserj is it possible that I have data hanging around from before I set the MaxSucceededListLength property? I see some large data payloads in my cache instance that are more than 24 hours old. It looks like my cache is not growing at all, so I believe that all my jobs created since I set the MaxSucceededListLength property are correctly expiring after 24 hours.

Is it safe to manually EXPIRE those specific keys if I can identify them without an expensive SCAN?