tdlib / td

Cross-platform library for building Telegram clients
https://core.telegram.org/tdlib
Boost Software License 1.0
7.11k stars 1.44k forks source link

Downloading files crashes TDLib #2441

Closed AlexXanderGrib closed 1 year ago

AlexXanderGrib commented 1 year ago

Hello, i am making a bot, that automatically detects and NSFW content in chats.

I am using following code:

async function checkFile(api: TelegramAPI, file: file) {
  setTimeout(() => {
    api.cancelDownloadFile({ file_id: file.id }).catch(noop);
  }, 5000);

  return await fromTryAsync(async () => {
    const downloaded = await api.downloadFile({
      file_id: file.id,
      limit: 0,
      offset: 0,
      priority: 30,
      synchronous: true,
    });

    if (!downloaded.local.path) {
      throw new Error("Unable to download file");
    }

    const isNsfw = await nsfwDetector.detect(downloaded);
    await api.deleteFile({ file_id: file.id });

    return isNsfw;
  });
}

And sometimes, TDLib produces following logs. (new_verbosity_level=1). After them, TDLib exits, and my bot crashes.

[ 1][t 1][1684572435.828005552][SessionConnection.cpp:1051][#1][!Session:2:main]        RawConnection::flush took 0.167136 seconds, written 88 bytes, read 0 bytes and returned OK
[ 1][t 2][1684572435.827948808][SessionConnection.cpp:1051][#1][!Session:2:download#0]  RawConnection::flush took 0.157084 seconds, written 88 bytes, read 0 bytes and returned OK
[ 1][t 2][1684572436.308269023][SessionConnection.cpp:1051][#1][!Session:2:download#0]  RawConnection::flush took 0.268666 seconds, written 0 bytes, read 92 bytes and returned OK
[ 1][t 1][1684572436.324269771][SessionConnection.cpp:1051][#1][!Session:2:main]        RawConnection::flush took 0.284437 seconds, written 0 bytes, read 92 bytes and returned OK

UPD: OS: Ubuntu 22.04.2 LTS RAM: 1GB CPU: 1 x 3GHz Network: 200Mb/s

levlam commented 1 year ago

The logs are unrelated. It is highly unlikely that TDLib has crashed during file downloading. Check that the process wasn't killed by OOM killer.

AlexXanderGrib commented 1 year ago

Yes, i was actually an OOM kill. Is it possible to fix this memory leak?

levlam commented 1 year ago

If your bot has a lot users and was added to many chats, then you need more RAM for the cache.

AlexXanderGrib commented 1 year ago

It have not. Cache is filling up over span of a 20 hours.

(UPD: I am explicitly deleting files after scanning, and deleteFile's description state that "Deletes a file from the TDLib file cache")

levlam commented 1 year ago

Files are stored only on disk, they occupy no RAM.

AlexXanderGrib commented 1 year ago

Memory leak was a mistake on my part, thank you for trying to help me