sqlite3.OperationalError: database is locked

[andrei-zamfirescu]

Hello,

Running telethon client with multi workers under pool executor causes constant database locks:

ERROR:telethon.telegram_bare_client:Connection was reset while receiving items. Reconnecting ERROR:telethon.telegram_bare_client:Connection was reset while receiving items. Reconnecting ERROR:telethon.telegram_bare_client:Unknown exception in the read thread! Disconnecting and leaving it to main t Traceback (most recent call last): File "/usr/local/lib/python3.4/dist-packages/telethon/telegram_bare_client.py", line 690, in _recv_thread_impl self.idle(stop_signals=tuple()) File "/usr/local/lib/python3.4/dist-packages/telethon/telegram_bare_client.py", line 659, in idle self._sender.receive(update_state=self.updates) File "/usr/local/lib/python3.4/dist-packages/telethon/network/mtproto_sender.py", line 147, in receive self._process_msg(remote_msg_id, remote_seq, reader, update_state) File "/usr/local/lib/python3.4/dist-packages/telethon/network/mtproto_sender.py", line 197, in _process_msg return self._handle_rpc_result(msg_id, sequence, reader) File "/usr/local/lib/python3.4/dist-packages/telethon/network/mtproto_sender.py", line 496, in _handle_rpc_res self.session.process_entities(request.result) File "/usr/local/lib/python3.4/dist-packages/telethon/session.py", line 405, in process_entities 'insert or replace into entities values (?,?,?,?,?)', rows sqlite3.OperationalError: database is locked

Is there a way we can fix this?

Thank you, Andrei.

[Lonami]

Is there a way we can fix this?

Is there any way I can reproduce this?

[andrei-zamfirescu]

I use this script for multi worker:

from concurrent.futures import ThreadPoolExecutor
from inc.db.getGroups import *
from inc.db.getLatestMessageID import *
from syncMessages import *
from syncExtendedData import *

executors_list = []
with ThreadPoolExecutor(max_workers=10) as executor:
    for group in GetGroups():
        print("Processing group %s" % group.name)
        executors_list.append(executor.submit(SyncMessages, group.group_id))
for x in executors_list:
    print(x.result())

And this script is calling telethon client:

import telethon
import json
from telethon.tl.types import InputPeerChat
from inc.db.insertMessage import *
import time

def TelethonSync(group_id,last_message_id):
    try:

        api_id = 
        api_hash = ''
        phone = ''
        #Construct client
        client = telethon.TelegramClient('session', api_id, api_hash)
        client.connect()
        client.start()
        messages = client.get_message_history(group_id, limit=None, min_id = last_message_id)
        print("Received messages for group %s" % group_id)
        for message in messages:
            print("Insert message id %s" % message.id)
            InsertMessage(message_id=message.id,
                        chat_id=group_id,
                        from_id=message.from_id,
                        message_datetime=message.date,
                        raw_messages=message.stringify())

    except Error as error:
        logging.error(error)
    finally:
        return "Sync successfull"

What I basically do is adding group messages to mysql db based on last inserted message id. Please let me know if this is somehow helpful or if I can provide you more info.

[Lonami]

Please format your comment better (use ```).

[Lonami]

Okay well the problem is that you're using the same 'session' file from many TelegramClient's at once. There should only be one TelegramClient running on a given 'session' file. So either you reuse the same client or you use multiple session filenames.

[andrei-zamfirescu]

I see, is there a way i can check if a session is already used?

[Lonami]

Well you should know if you have a connected client on a certain file.

[andrei-zamfirescu]

Yes, but in my multi thread scenario i'm not sure how i can do it rather than send this as a parm when running executors. Let's suppose i have generated 10 session files, how can i check from TelethonSync if session1 is taken and just increment the suffix until I reach an available session file. I really appreciate your help! Thanks

[Lonami]

That's not a problem related to the library, and you should ask somewhere else.

[andrei-zamfirescu]

Okay, I thought that there might be some available methods in the library. Thanks a lot! Wish you all the best.

[masoudcom]

Hi, When use same functionality such as GetMessageHistory and ... Telegram API store hash_access and channel_id into entities. You must disable save entities and try to perform multi thread and every thing. Thanks @Lonami for write this perfect library.

[Lonami]

Disabling saving entities is generally not a good idea, since they are required for a lot of things. The library is not thread-safe. It's designed to work well in asyncio, not with threads, and that probably wouldn't solve the locked issue. I'm glad you like the library, though. But avoiding the issue is as simple as either using different session names or not running two scripts at once with the same file…

[Abolfazl-Dalily]

hello i get this errors

ERROR:telethon.telegram_bare_client:Unknown exception in the read thread! Disconnecting and leaving it to main thread
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/telethon/telegram_bare_client.py", line 735, in _recv_thread_impl
    self.idle(stop_signals=tuple())
  File "/usr/local/lib/python3.5/dist-packages/telethon/telegram_bare_client.py", line 696, in idle
    self._sender.receive(update_state=self.updates)
  File "/usr/local/lib/python3.5/dist-packages/telethon/network/mtproto_sender.py", line 180, in receive
    self._process_msg(remote_msg_id, remote_seq, reader, update_state)
  File "/usr/local/lib/python3.5/dist-packages/telethon/network/mtproto_sender.py", line 247, in _process_msg
    obj = reader.tgread_object()
  File "/usr/local/lib/python3.5/dist-packages/telethon/extensions/binary_reader.py", line 143, in tgread_object
    return clazz.from_reader(self)
  File "/usr/local/lib/python3.5/dist-packages/telethon/tl/types/__init__.py", line 20530, in from_reader
    _x = reader.tgread_object()
  File "/usr/local/lib/python3.5/dist-packages/telethon/extensions/binary_reader.py", line 143, in tgread_object
    return clazz.from_reader(self)
  File "/usr/local/lib/python3.5/dist-packages/telethon/tl/types/__init__.py", line 19149, in from_reader
    _message = reader.tgread_object()
  File "/usr/local/lib/python3.5/dist-packages/telethon/extensions/binary_reader.py", line 143, in tgread_object
    return clazz.from_reader(self)
  File "/usr/local/lib/python3.5/dist-packages/telethon/tl/types/__init__.py", line 10676, in from_reader
    _media = reader.tgread_object()
  File "/usr/local/lib/python3.5/dist-packages/telethon/extensions/binary_reader.py", line 141, in tgread_object
    raise TypeNotFoundError(constructor_id)
telethon.errors.common.TypeNotFoundError: Could not find a matching Constructor ID for the TLObject that was supposed to be read with ID 0xb5223b0f. Most likely, a TLObject was trying to be read when it should not be read.

version of telethon is 0.19.1.6

my code :

from telethon import TelegramClient
import logging
from logHandler import *

logging.basicConfig(level=logging.ERROR)

class InterActiveTelegramClient(TelegramClient):

    def __init__(self, session, api_id, api_hash, default_channel):

        self.__default_channel = default_channel

        print('Initializing InteractiveTelegramClient ...')

        super().__init__(
            session=session, api_id=api_id, api_hash=api_hash,update_workers=3)

        print('Connecting to Telegram servers...')
        self.start()

    def profile(self):
        me = self.get_me()
        print("#########################################")
        print(" \t\tYour Profile:\t\t")
        str = " \tfirstName : {} \n" \
              " \tlastName : {} \n" \
              " \tuserName : {} \n" \
              " \tPhone : {} \n" \
              " \tId : {} \n"
        print(str.format(me.first_name, me.last_name, me.username, me.phone,me.id))
        print("#########################################")

    def load_all_dialogs(self):
        try:

            dialogs = self.get_dialogs(limit=None)

            result = "{0:20s}{1:20s}{2:20s}{3:40s}{4:20s}".format("Number", "Type", "ID", "Username", "Title") + "\n"
            for i, dialog in enumerate(dialogs, start=1):

                entity = dialog.entity
                if type(entity).__name__ == 'Channel':
                    result += "{0:20s}{1:20s}{2:20s}{3:40s}{4:20s}".format(str(i),"Channel/MetaGroup", str(entity.id), str(entity.username),
                                                                str(entity.title)) + "\n"

                elif type(entity).__name__ == "Chat":
                    result += "{0:20s}{1:20s}{2:20s}{3:40s}{4:20s}".format(str(i),"Group", str(entity.id), "-----------",
                                                                    str(entity.title)) + "\n"

                elif type(entity).__name__ == "User":
                    result += "{0:20s}{1:20s}{2:20s}{3:40s}{4:20s}".format(str(i),"User", str(entity.id), str(entity.username),
                                                                    str(entity.first_name)) + "\n"
                else:
                    result += "{0:20s}{1:20s}{2:20s}{3:40s}{4:20s}".format(str(i), "None", str(entity.id),
                                                                           "-----------",
                                                                          str(dialog.name)) + "\n"

            fHandler = open('contacts.txt', 'w', encoding='utf-8')

            fHandler.write(result)
            fHandler.close()

        except Exception as e :
            logger.elog("Telegram Client Can not Fetch All Dialogs Because : " + str(e))
            return None

api_id = int(config.get('telegram', 'api_id'))
api_hash = config.get('telegram', 'api_hash')
session = config.get('telegram', 'session')
default_channel = config.get('telegram','default_channel')
try:
    print("in name of GOD")
    client = InterActiveTelegramClient(session, api_id, api_hash,default_channel)
    client.profile()
    print("loadings all dialogs ....")
    client.load_all_dialogs()
    client.disconnect()
    print("loading all dialogs successfully !")
except Exception as e :
    print("Can not Create Client Because : "+str(e))
    print("Please relaunch client !")

[snowman]

In my case, I have ipdb debug session with Ctrl-Z, use fg to return back

[simbadmorehod]

@andrei-zamfirescu Have you resolved the blocking issue? I'm also interested in this, since I have to stop the asynchronous call session and start it again, which comes with additional errors besides blocking, I would like to be able to reuse the session, would it be enough for me to create a copy of the session file with a different name?

@Lonami Wouldn't it be easier to make it possible to use another database for the session store so that no locking occurs?

[Lonami]

Locking is a feature, not a bug. It prevents user errors. Under normal operation, this error does not occur.

If you believe you're doing it correctly, and SQLite is giving you trouble, you're welcome to use a different storage. This is a supported and documented use-case.

[simbadmorehod]

Locking is a feature, not a bug. It prevents user errors. Under normal operation, this error does not occur.

If you believe you're doing it correctly, and SQLite is giving you trouble, you're welcome to use a different storage. This is a supported and documented use-case.

Yes, I understand what you mean, I just don't seem to understand possible mistakes, thank you for the quick response, I will try to describe what I'm doing:

I have created a solution that starts several sessions at once in the same queue for executing asynchronous functions, they seem to work simultaneously and listen for events.

Sometimes I need to change the settings, for this to get, for example, user chats, this is a separate request that requires connection through a session that has already taken sqlite, for this I have to send a shutdown of an asynchronous task that works in the background, it does not always work correctly, since knowledge in programming is weak, but I realized that I can for individual requests, for example, to get a list of chats, create a separate session with a different name, this will allow me to execute requests without blocking and stopping the main session.

Perhaps there is a better way, I would be very grateful, thank you again.

[gumumans]

If you have encountered this problem, it is obvious that you have several Telegram clients accessing the session. Due to the peculiarities of Sqlite, it is not possible to access the connection simultaneously. As an option, change the session and analyze the code. Or use a singleton client