search

taskforcesh / bullmq

BullMQ - Message Queue and Batch processing for NodeJS and Python based on Redis
https://bullmq.io
MIT License
5.88k stars 380 forks source link

[Bug]: Many connections to Redis #2243

Open venticello opened 10 months ago

venticello commented 10 months ago

Version

1.15.2

Platform

Python

What happened?

I tried to get data from completed tasks from 2 queues, and in the end I got 34 additional connections to Redis. Well, we also lack the ability to reuse an existing connection like in NodeJS.

How to reproduce.

import asyncio
from bullmq import Queue

elo_queue = Queue('elo', "redis://default:redispw@localhost:6379")
dna_queue = Queue('dna', "redis://default:redispw@localhost:6379")

async def get_data():
    jobs = await elo_queue.getCompleted(0, 20)
    jobs2 = await dna_queue.getCompleted(0, 20)
    print(jobs, jobs2)
    await asyncio.sleep(10)

if __name__ == "__main__":
    asyncio.run(get_data())

Relevant log output

127.0.0.1:6379> CLIENT LIST
id=20886 addr=172.24.0.1:47940 laddr=172.24.0.2:6379 fd=56 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20887 addr=172.24.0.1:47942 laddr=172.24.0.2:6379 fd=57 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20850 addr=172.24.0.1:47868 laddr=172.24.0.2:6379 fd=20 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20851 addr=172.24.0.1:47870 laddr=172.24.0.2:6379 fd=21 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20852 addr=172.24.0.1:47872 laddr=172.24.0.2:6379 fd=22 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20853 addr=172.24.0.1:47874 laddr=172.24.0.2:6379 fd=23 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20854 addr=172.24.0.1:47876 laddr=172.24.0.2:6379 fd=24 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20855 addr=172.24.0.1:47878 laddr=172.24.0.2:6379 fd=25 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20856 addr=172.24.0.1:47880 laddr=172.24.0.2:6379 fd=26 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20857 addr=172.24.0.1:47882 laddr=172.24.0.2:6379 fd=27 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20858 addr=172.24.0.1:47884 laddr=172.24.0.2:6379 fd=28 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20859 addr=172.24.0.1:47886 laddr=172.24.0.2:6379 fd=29 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20860 addr=172.24.0.1:47888 laddr=172.24.0.2:6379 fd=30 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20861 addr=172.24.0.1:47890 laddr=172.24.0.2:6379 fd=31 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20862 addr=172.24.0.1:47892 laddr=172.24.0.2:6379 fd=32 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20863 addr=172.24.0.1:47894 laddr=172.24.0.2:6379 fd=33 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20864 addr=172.24.0.1:47896 laddr=172.24.0.2:6379 fd=34 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20865 addr=172.24.0.1:47898 laddr=172.24.0.2:6379 fd=35 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20866 addr=172.24.0.1:47900 laddr=172.24.0.2:6379 fd=36 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=46 addr=127.0.0.1:35958 laddr=127.0.0.1:6379 fd=16 name= age=7372 idle=0 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=26 qbuf-free=20448 argv-mem=10 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=22426 events=r cmd=client|list user=default redir=-1 resp=2 lib-name= lib-ver=
id=20846 addr=172.24.0.1:47860 laddr=172.24.0.2:6379 fd=15 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20847 addr=172.24.0.1:47862 laddr=172.24.0.2:6379 fd=17 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20848 addr=172.24.0.1:47864 laddr=172.24.0.2:6379 fd=18 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20849 addr=172.24.0.1:47866 laddr=172.24.0.2:6379 fd=19 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20867 addr=172.24.0.1:47902 laddr=172.24.0.2:6379 fd=37 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20868 addr=172.24.0.1:47904 laddr=172.24.0.2:6379 fd=38 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20869 addr=172.24.0.1:47906 laddr=172.24.0.2:6379 fd=39 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20870 addr=172.24.0.1:47908 laddr=172.24.0.2:6379 fd=40 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20871 addr=172.24.0.1:47910 laddr=172.24.0.2:6379 fd=41 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20872 addr=172.24.0.1:47912 laddr=172.24.0.2:6379 fd=42 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20873 addr=172.24.0.1:47914 laddr=172.24.0.2:6379 fd=43 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20874 addr=172.24.0.1:47916 laddr=172.24.0.2:6379 fd=44 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20875 addr=172.24.0.1:47918 laddr=172.24.0.2:6379 fd=45 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20876 addr=172.24.0.1:47920 laddr=172.24.0.2:6379 fd=46 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20877 addr=172.24.0.1:47922 laddr=172.24.0.2:6379 fd=47 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20878 addr=172.24.0.1:47924 laddr=172.24.0.2:6379 fd=48 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20879 addr=172.24.0.1:47926 laddr=172.24.0.2:6379 fd=49 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20880 addr=172.24.0.1:47928 laddr=172.24.0.2:6379 fd=50 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20881 addr=172.24.0.1:47930 laddr=172.24.0.2:6379 fd=51 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20882 addr=172.24.0.1:47932 laddr=172.24.0.2:6379 fd=52 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20883 addr=172.24.0.1:47934 laddr=172.24.0.2:6379 fd=53 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20884 addr=172.24.0.1:47936 laddr=172.24.0.2:6379 fd=54 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=
id=20885 addr=172.24.0.1:47938 laddr=172.24.0.2:6379 fd=55 name= age=4 idle=4 flags=N db=0 sub=0 psub=0 ssub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=0 omem=0 tot-mem=1928 events=r cmd=hgetall user=default redir=-1 resp=2 lib-name= lib-ver=

Code of Conduct

venticello commented 10 months ago

I tried to get the same values from the same database through NodeJS version and no additional connections were created there for such selections.

divmgl commented 10 months ago

You can pass in an ioredis instance, so I'm not sure this is true.

venticello commented 10 months ago

You can pass in an ioredis instance, so I'm not sure this is true.

Everything is fine in the nodejs (ioredis), but not with Python. I noticed this problem only occurs if the Redis library was installed, and the bullmq was placed on top. If you remove Redis and reinstall it, there is no such problem.

nodegin commented 6 months ago

Getting the same issue, passing ioredis instance seem fixing the issue