felixmosh / bull-board

🎯 Queue background jobs inspector
MIT License
2.31k stars 362 forks source link

API endpoint returning 500 #457

Closed n10000k closed 1 year ago

n10000k commented 2 years ago

API Uri - /api/queues?activeQueue=&page=1&jobsPerPage=10

Memory usage
8.13 MB of 10.5 GB0.08%
Peak memory usage
8.57 MB
Fragmentation ratio
6.18
Connected clients
28
Blocked clients
0
Version
6.2.5
Mode
cluster
OS
Amazon ElastiCache
Up time
24 days, 8 hours, 21 minutes

Error:

{
   "error":"Internal server error",
   "message":"EXECABORT Transaction discarded because of previous errors.",
   "details":"ReplyError: EXECABORT Transaction discarded because of previous errors.\n    at parseError (/node_modules/redis-parser/lib/parser.js:179:12)\n    at parseType (/node_modules/redis-parser/lib/parser.js:302:14)"
}

Local setup runs Redis cluster and works fine, same queue setup using key prefixes. Has anyone had this issue when deploying to an instance on AWS? Are there any better ways to debug this? thanks in advance

felixmosh commented 2 years ago

Sorry I'm not familiar with this setup. Maybe anyone else can help you...

felixmosh commented 2 years ago

What is the server error that you get?

minlite commented 2 years ago

Seeing the same issue on my end after upgrading from 3.9.2 to 4.6.0.

{"error":"Internal server error","message":"EXECABORT Transaction discarded because of previous errors.","details":"ReplyError: EXECABORT Transaction discarded because of previous errors.\n    at parseError (/var/app/current/node_modules/redis-parser/lib/parser.js:179:12)\n    at parseType (/var/app/current/node_modules/redis-parser/lib/parser.js:302:14)"}

Details:

Memory usage
1.1 GB of 1.1 GB100.00%
Peak memory usage
1.11 GB
Fragmentation ratio
0.82
Connected clients
407
Blocked clients
51
Version
6.0.5
Mode
standalone
OS
Amazon ElastiCache
Up time
30 days, 2 hours, 26 minutes

Interesting to note that this doesn't always happen. It seems to happen in waves where 3-4 requests would succeed and the UI will load but then 10 next requests would fail with Internal Server Error and then back to 3-4 successful requests.

n10000k commented 2 years ago

I can confirm the same happens still to me, just randomly (not sure on the amount of requests). It brings up the queues then randomly has some form of timeout or something going on where it throws the above message. Maybe some form of issue with ElastiCache?

minlite commented 2 years ago

Disregard my previous message. As it's evident from the details I posted, the issue comes from the server being out of memory. I just noticed the OOM issues in my server logs.

Once I upgraded the cache to a bigger instance, the errors went away.

felixmosh commented 2 years ago

@n10000k ?

n10000k commented 2 years ago

We run it in cluster mode, no memory issues. I feel like after x amount of polling within a cluster mode it drops off. I can share an example to replicate locally later on.

felixmosh commented 2 years ago

I'll happy to get a replication example

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.