search

cenkalti / kuyruk

⚙️ Simple task queue for Python
https://kuyruk.readthedocs.org/
MIT License
231 stars 17 forks source link

Tune performance #53

Closed frol closed 7 years ago

frol commented 7 years ago

I am very interested in getting maximum available performance out of RabbitMQ as I have a need to schedule a heap of tasks (I want to handle 100k tasks in a reasonable time, a few minutes). Given my old experiments reported in #51 (238 tasks/second), it will take ~400 seconds (~7 minutes) to schedule 100k tasks.

I think the current bottleneck is in amqp library, and it seems that rabbitmq will work better as it implemented CFFI bindings to librabbitmq native library.

This issue is just a placeholder for the related experiments, yet if it is doable, it is better to optimize it sooner rather later.

frol commented 7 years ago

@cenkalti I would love to hear any comment from you on the matter. I might dig into the performance tuning myself next month, so I would love to have any valuable information on hands like whether you see any complexity of switching amqp to rabbitmq module right away given that the interfaces should be quite similar since they are just drivers to the same RabbitMQ server.

cenkalti commented 7 years ago

Hi @frol. Sorry about the delayed answer. We didn't design Kuyruk for high performance scenarios and I don't want to give up the beauty of the pure-Python library by replacing it with rabbitmq package that uses CFFI bindings. I would like to close this issue if it is okay for you.

frol commented 7 years ago

@cenkalti I see. Well, I like Kuyruk API, so I will tune its performance rather than switch to anything else. If I manage to get a better performance without sacrificing the beauty, I will make a PR.