[redis] upstash shows error when accessing too many keys

unjs / unstorage

💾 Unstorage provides an async Key-Value storage API with conventional features like multi driver mounting, watching and working with metadata, dozens of built-in drivers and a tiny core.

https://unstorage.unjs.io

MIT License

1.65k stars 122 forks source link

[redis] upstash shows error when accessing too many keys #418

Open Rigo-m opened 3 months ago

Rigo-m commented 3 months ago

Describe the feature

Working with large redis db, you might encounter an error if you try to fetch too many keys altogether. There should be a scan option available (or getKeys should be implemented via scan)

Additional information

[ ] Would you be willing to help implement this feature?

pi0 commented 3 months ago

Would you please share runnable reproduction?

Rigo-m commented 3 months ago

Hi pi0, using upstash as a redis provider https://stackblitz.com/edit/nuxt-starter-ako5kk?file=server%2Fapi%2Ferror-redis.ts this simple script triggers the error when using getKeys.

Actually, as a second thought, I'm not sure whether the error is from upstash which is prohibiting reading that many keys altogether (since it is a blocking operation https://redis.io/commands/keys/) or whether it is a redis limitation. Can I do anything else?

pi0 commented 2 months ago

Hi @chronark do you know if it is a known limitation? And what would you suggest to scan too many keys?