search

celery / ceps

Celery Enhancement Proposals
27 stars 15 forks source link

AMQP, disable connection pooling #6

Open carlhoerberg opened 6 years ago

carlhoerberg commented 6 years ago

Default to broker_pool_limit = 1. There is no point in having many connections to a RabbitMQ server from a single process. Instead each worker/thread should have their own channel.

jheld commented 5 years ago

Can you elaborate further about this? I for one am not as knowledgeable about the problem area.

Why would each sub-process want their own channel? If I'm reading right. Not sure about the worker/thread definition here, too (sub-process/coroutine?).

thedrow commented 5 years ago

Unfortunately for us, our AMQP client is not threadsafe. We will have to prototype and figure out how this is going to work. In addition, there's a maximum amount of channels on each connection. It's rather large so it's not usually a problem but users do unexpected things.

The real advantage of using multiple connection is avoiding locks and paralleling the output of messages. Channels only multiplex messages over a single connection. Sometimes that's too slow for the throughput you are after.