WebUI not loading Queues

jirevwe commented 1 year ago

Hi 👋🏿,

I'm on running asynq: 0.23.0 and asynqmon: 0.7.1 in my production environment and I have been unable to access the Web UI, when this happened in the past running flushall in redis fixed it (got the idea from #149), but I can't afford to do that now as I am storing other things in redis.

What do you think caused this (so I can prevent this from happening in the future) and how can I get it to work without clearing all the keys?

I attached a screenshot of the error. Thanks in advance.

Screenshot from 2023-03-22 17-24-38

netixx commented 9 months ago

I have the same issue, I ran flushall, but it didn't solve my issue either.

MrRobo-t commented 9 months ago

Facing the same issue.. It momentarily shows the active task.. That's all I get.. There is no archive task coming in.. @hibiken Can you please help on this? I have build using the docker image .. I am getting graph for task processed though.. The db is deployed on aws elasticache redis..

netixx commented 9 months ago

I looked at the call graph for the /queues route, and given the error is lua running in redis, my guess is that the issue is in this script:

var memoryUsageCmd = redis.NewScript(`
local sample_size = tonumber(ARGV[2])
if sample_size <= 0 then
    return redis.error_reply("sample size must be a positive number")
end
local memusg = 0
for i=1,2 do
    local ids = redis.call("LRANGE", KEYS[i], 0, sample_size - 1)
    local sample_total = 0
    if (table.getn(ids) > 0) then
        for _, id in ipairs(ids) do
            local bytes = redis.call("MEMORY", "USAGE", ARGV[1] .. id)
            sample_total = sample_total + bytes
        end
        local n = redis.call("LLEN", KEYS[i])
        local avg = sample_total / table.getn(ids)
        memusg = memusg + (avg * n)
    end
    local m = redis.call("MEMORY", "USAGE", KEYS[i])
    if (m) then
        memusg = memusg + m
    end
end
for i=3,6 do
    local ids = redis.call("ZRANGE", KEYS[i], 0, sample_size - 1)
    local sample_total = 0
    if (table.getn(ids) > 0) then
        for _, id in ipairs(ids) do
            local bytes = redis.call("MEMORY", "USAGE", ARGV[1] .. id)
            sample_total = sample_total + bytes
        end
        local n = redis.call("ZCARD", KEYS[i])
        local avg = sample_total / table.getn(ids)
        memusg = memusg + (avg * n)
    end
    local m = redis.call("MEMORY", "USAGE", KEYS[i])
    if (m) then
        memusg = memusg + m
    end
end
local groups = redis.call("SMEMBERS", KEYS[7])
if table.getn(groups) > 0 then
    local agg_task_count = 0
    local agg_task_sample_total = 0
    local agg_task_sample_size = 0
    for i, gname in ipairs(groups) do
        local group_key = ARGV[4] .. gname
        agg_task_count = agg_task_count + redis.call("ZCARD", group_key)
        if i <= tonumber(ARGV[3]) then
            local ids = redis.call("ZRANGE", group_key, 0, sample_size - 1)
            for _, id in ipairs(ids) do
                local bytes = redis.call("MEMORY", "USAGE", ARGV[1] .. id)
                agg_task_sample_total = agg_task_sample_total + bytes
                agg_task_sample_size = agg_task_sample_size + 1
            end
        end
    end
    local avg = agg_task_sample_total / agg_task_sample_size
    memusg = memusg + (avg * agg_task_count)
end
return memusg
`)

Since the issue is the "bytes", then I guess more specifically here:

local bytes = redis.call("MEMORY", "USAGE", ARGV[1] .. id)
            sample_total = sample_total + bytes

running asynqmon v0.7.2 and asynq v0.24.1

This part of the code is asynq code (internal/rdb/inspect.go). All calls to CurrentStats should be impacted as well.