unresponsive client await redis.scan() never returns

[cokeeffekt]

ioredis version 4.16 redis version 5.0.8

using a means of recursion i'm looking for keys that match a pattern. Nothing out of the ordinary the code works endlessly in dev but after about an hour or so of running in production the follow stops returning, once it's stopped nothing on that connection returns anymore. Im really only taking a guess that this it the command that triggers the issue by wrapping it in a timeout, this line seems to be the one that hits the timeout first.

var keys = await redis.scan('0', 'MATCH', pattern, 'COUNT', '500');
console.log('never runs');

unable to get anything out of ioredis via the normal means, from redis.status prior it is 'ready', wrapped in a try catch no errors are thrown. using events i cant see any errors, disconnects OR reconnects.

I have tried to used promise and callback implementations with same outcome.

This issue has me on my knees, i have tried to eliminate everything, redis version latest stable, ioredis version latest, even upped the specs on the production redis server. Im 90% sure the issue is in the client... i can run similar commands against production from the cli and never hit this issue.

Edit: Side note. other instances of the server (pointing at same redis instance) continue working even if one instance begins to fail, but they will all fail to the same never returning state if left long enough (few hours)

[alavers]

I'd suggest maybe running your code with ioredis debug logging turned on.

[cokeeffekt]

thanks @alavers this lead me down another rabbit hole... i can see that ioredis is trying to reconnect, but something that is curious that redis.status never changes from 'ready' status.

Im still stuck on this issue in the meantime i've broken my app onto 2 redis instances (one dedicated to pub/sub incase that has an impact) and restarting the app every 30minutes to avoid hitting the issue.

in the process of debugging all this i've changed pretty much every redis config there is related, other than having a much deeper understanding of redis i've made no real progress on finding a solution.

[alavers]

Wait a sec,

i've broken my app onto 2 redis instances (one dedicated to pub/sub incase that has an impact)

You already have to do this. Once a redis client subscribes it can't issue any command other than (P)SUBSCRIBE / (P)UNSUBSCRIBE / PING / QUIT.

Is it possible you're running that .scan command on a subscriber client? ioredis would produce an error in that case, and since you invoked the scan with an await the console.log('never runs'); below it would never run if an error went unhandled. Do you see any unhandled promise rejection warnings?

[cokeeffekt]

@alavers 2 redis server instances. the clients are always separate subscriber OR store, you cant do anything on a subscriber client i understand this.

The problem is over time based on load the issue happens, everything works until it doesn't, i doubt implementation is wrong, it's been working for years, but the added load of late has cause the problem to show extremely frequently. and yes everything when possible is wrapped in try catch with error logs, nothing ever shows up once the client goes unresponsive.

This issue still plagues me. With increased load due to isolation my app services are being restarted every 20 minutes to avoid the issue.

Further debugging has be led down the path of removing .scan completely from the code, replaced with storing keys on a list. Clients still eventually become unresponsive. When unresponsive nothing returns and iosredis client status is always ready... when i have the time next week i'm planing on reimplementing some of my services using another redis client, i'm at a loss on what else to do.

[deadcoder0904]

faced this now.

i have a simple script that i run with tsx which gets all redis keys & values.

im doing this for debugging purposes. dont like ssh'ing into vps to get values if i can do it with 1 command.

but it never stops. i get stuck. idk why?

import consola from 'consola'

import { redis } from '@/app/lib/redis/ioredis'

async function main() {
  const keys = await redis.keys('*')

  consola.info(`Total Keys: ${keys.length}`)

  for await (const key of keys) {
    consola.success('Key', key)
    const value = await redis.get(key)
    consola.success('Value', value)
  }
}

main()

the code should exit at least.

i tried redis.disconnect() but that closes it before i get value so that's a no go.

pretty sure this is a ioredis bug. am i wrong?