Multiple bot instances in multithreaded/multiprocess environment

[mike0sv]

Hi! I use telepot with celery to process updates asynchronously, and noticed some strange behavior/errors. For example, sometimes .editMessageReplyMarkup fails with ReadTimeoutError, which is 30 seconds. And sometimes .sendPhoto takes >10 seconds to execute. My question is, is it because celery runs multiple processes, so multiple instances of same bot runs on one machine or something like this?

minimal code example is like this

import telepot
from celery import shared_task

bot = None

@shared_task
def send(token, chat_id):
    if bot is None:
        bot = telepot.Bot(token)
    bot.sendMessage(chat_id, 'kek')

def handle(msg):
    chat_id = msg['from']['id']
    send.delay(token, chat_id)

token = '...'
if __name__ == '__main__':
    bot = telepot.Bot(token)
    bot.message_loop(handle, run_forever=True)

[nickoala]

I don't know how celery works, but the following few points are true for sure:

1. You cannot have more than one program calling message_loop() on the same token, because Telegram does not allow multiple processes to poll on the same token.

2. You can have many programs sending things using the same token.

3. ReadTimeoutError and how long it takes to send something, these are problems that are hard for me to say exactly why. It could be the network or Telegram servers. Without investigating further, it's hard to say where the real problem lies.

[mike0sv]

Basically, when you use celery tasks, instead of direct call, it serializes them and puts into queue (external broker like rabbitmq or redis). Then you start your celery worker, which is a python process that imports the same code, takes tasks from queue and runs them. It can utilize pool of threads or processes (processes in my case). So,

1. message_loop is called only in my main process
2. In every process of workers' pool bot is initialized, so I thought that might be a problem, but you say it's ok actually.
3. How do I investigate this? I cannot reproduce it manually, it happens when the load is high. Or is it typical for telegram servers?