junedkh / jmdkh-mltb

Unofficial Repository: Telegram bot which can download torrents, mega, google drive, telegram file and direct links + all yt-dlp supported sites, then upload them to google drive or telegram cloud. This bot supports more features like zip or extract before upload, seed after upload using qBittorrent or aria2c and many more, read features below.
https://t.me/+5d2_gaR0QwwyNGE1
GNU General Public License v3.0
13 stars 12 forks source link

Segmentation Fault Error when Bot gets a more than 3 forwarded leeach #55

Closed m-hoseyny closed 1 year ago

m-hoseyny commented 1 year ago

Hi,

I wanna use your project; however, when the bot gets more than 3 forwarded messages to start the leeching, the container stops, and I get these errors in my docker logs.

2023-04-28 07:15:33,920 - root - WARNING - MEGA Credentials not provided!
2023-04-28 07:15:33,921 - root - WARNING - BASE_URL not provided!
2023-04-28 07:15:35,063 - root - INFO - Initializing Aria2c
2023-04-28 07:15:36,608 - root - INFO - Creating client from BOT_TOKEN
2023-04-28 07:15:37,711 - pyrogram.session.auth - INFO - Start creating a new auth key on DC2
2023-04-28 07:15:37,711 - pyrogram.connection.connection - INFO - Connecting...
2023-04-28 07:15:37,776 - pyrogram.connection.connection - INFO - Connected! Production DC2 - IPv4
2023-04-28 07:15:38,765 - pyrogram.session.auth - INFO - Done auth key exchange: DhGenOk
2023-04-28 07:15:38,766 - pyrogram.connection.connection - INFO - Disconnected
2023-04-28 07:15:39,009 - pyrogram.connection.connection - INFO - Connecting...
2023-04-28 07:15:39,073 - pyrogram.connection.connection - INFO - Connected! Production DC2 - IPv4
2023-04-28 07:15:39,074 - pyrogram.session.session - INFO - NetworkTask started
2023-04-28 07:15:39,403 - pyrogram.session.session - INFO - Session initialized: Layer 158
2023-04-28 07:15:39,403 - pyrogram.session.session - INFO - Device: CPython 3.10.6 - Pyrogram 2.0.104
2023-04-28 07:15:39,404 - pyrogram.session.session - INFO - System: Linux 5.15.0-69-generic (en)
2023-04-28 07:15:39,404 - pyrogram.session.session - INFO - Session started
2023-04-28 07:15:39,504 - pyrogram.session.session - INFO - PingTask started
2023-04-28 07:15:39,618 - pyrogram.session.session - INFO - PingTask stopped
2023-04-28 07:15:39,618 - pyrogram.connection.connection - INFO - Disconnected
2023-04-28 07:15:39,619 - pyrogram.session.session - INFO - NetworkTask stopped
2023-04-28 07:15:39,619 - pyrogram.session.session - INFO - Session stopped
2023-04-28 07:15:39,872 - pyrogram.session.auth - INFO - Start creating a new auth key on DC4
2023-04-28 07:15:39,872 - pyrogram.connection.connection - INFO - Connecting...
2023-04-28 07:15:39,934 - pyrogram.connection.connection - INFO - Connected! Production DC4 - IPv4
2023-04-28 07:15:41,048 - pyrogram.session.auth - INFO - Done auth key exchange: DhGenOk
2023-04-28 07:15:41,049 - pyrogram.connection.connection - INFO - Disconnected
2023-04-28 07:15:41,359 - pyrogram.connection.connection - INFO - Connecting...
2023-04-28 07:15:41,417 - pyrogram.connection.connection - INFO - Connected! Production DC4 - IPv4
2023-04-28 07:15:41,419 - pyrogram.session.session - INFO - NetworkTask started
2023-04-28 07:15:41,744 - pyrogram.session.session - INFO - Session initialized: Layer 158
2023-04-28 07:15:41,744 - pyrogram.session.session - INFO - Device: CPython 3.10.6 - Pyrogram 2.0.104
2023-04-28 07:15:41,744 - pyrogram.session.session - INFO - System: Linux 5.15.0-69-generic (en)
2023-04-28 07:15:41,745 - pyrogram.session.session - INFO - Session started
2023-04-28 07:15:41,745 - pyrogram.session.session - INFO - PingTask started
2023-04-28 07:15:42,365 - pyrogram.dispatcher - INFO - Started 6 HandlerTasks
2023-04-28 07:15:42,910 - bot - INFO - Creating Telegraph Account
2023-04-28 07:15:42,959 - bot - INFO - Users data has been imported from Database
2023-04-28 07:15:44,752 - apscheduler.scheduler - INFO - Adding job tentatively -- it will be properly scheduled when the scheduler starts
2023-04-28 07:15:44,753 - apscheduler.scheduler - INFO - Added job "RSS" to job store "default"
2023-04-28 07:15:44,754 - apscheduler.scheduler - INFO - Scheduler started
2023-04-28 07:15:44,897 - bot - INFO - Bot Started!
2023-04-28 07:16:04,755 - apscheduler.executors.default - INFO - Running job "RSS (trigger: interval[0:15:00], next run at: 2023-04-28 07:31:04 UTC)" (scheduled at 2023-04-28 07:16:04.751997+00:00)
2023-04-28 07:16:04,755 - bot - WARNING - RSS_CHAT_ID not added! Shutting down rss scheduler...
2023-04-28 07:16:04,756 - apscheduler.executors.default - INFO - Job "RSS (trigger: interval[0:15:00], next run at: 2023-04-28 07:31:04 UTC)" executed successfully
2023-04-28 07:16:04,756 - apscheduler.scheduler - INFO - Scheduler has been shut down
2023-04-28 07:16:07,816 - bot - INFO - http://dummy.com/Series/Love.and.Death/Love.and.Death.S01E03.720p.10bit.WEBRip.2CH.x265.HEVC.PSA.mkv
2023-04-28 07:16:07,828 - bot - INFO - http://dummy.com/Series/Citadel/S01/SoftSub/1080p.x265/Citadel.S01E01.1080p.10bit.WEBRip.6CH.x265.HEVC.PSA.SoftSub.mkv
2023-04-28 07:16:07,855 - bot - INFO - http://dummy.com/Series/Citadel/480p/Citadel.S01E01.The.Human.Enigma.480p.WEB-HD.x264.Pahe.mkv
2023-04-28 07:16:07,899 - bot - INFO - http://dummy.com/Series/Citadel/720p/Citadel.S01E01.The.Human.Enigma.720p.WEB-HD.x264.Pahe.mkv
2023-04-28 07:16:07,944 - bot - INFO - http://dummy.com/Series/Citadel/720p.x265/Citadel.S01E01.720p.10bit.WEBRip.2CH.x265.HEVC.PSA.mkv
Fatal Python error: Segmentation fault

Thread 0x00007fb0caffd640 (most recent call first):
  File "/usr/local/lib/python3.10/dist-packages/requests_cache/backends/sqlite.py", line 228 in connection
  File "/usr/lib/python3.10/contextlib.py", line 135 in __enter__
  File "/usr/local/lib/python3.10/dist-packages/requests_cache/backends/sqlite.py", line 282 in __setitem__
  File "/usr/local/lib/python3.10/dist-packages/requests_cache/backends/base.py", line 91 in save_response
  File "/usr/local/lib/python3.10/dist-packages/requests_cache/session.py", line 234 in _send_and_cache
  File "/usr/local/lib/python3.10/dist-packages/requests_cache/session.py", line 206 in send
  File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 587 in request
  File "/usr/local/lib/python3.10/dist-packages/requests_cache/session.py", line 159 in request
  File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 59 in request
  File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 100 in head
  File "/usr/src/app/bot/helper/ext_utils/bot_utils.py", line 242 in get_content_type
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58 in run
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 83 in _worker
  File "/usr/lib/python3.10/threading.py", line 953 in run
  File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap

Current thread 0x00007fb0cb7fe640 (most recent call first):
  File "/usr/local/lib/python3.10/dist-packages/requests_cache/backends/sqlite.py", line 237 in close
  File "/usr/local/lib/python3.10/dist-packages/requests_cache/backends/base.py", line 105 in close
  File "/usr/local/lib/python3.10/dist-packages/requests_cache/session.py", line 311 in close
  File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 455 in __exit__
  File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 58 in request
  File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 100 in head
  File "/usr/src/app/bot/helper/ext_utils/bot_utils.py", line 242 in get_content_type
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58 in run
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 83 in _worker
  File "/usr/lib/python3.10/threading.py", line 953 in run
  File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007fb0cbfff640 (most recent call first):
  File "/usr/lib/python3.10/socket.py", line 955 in getaddrinfo
  File "/usr/lib/python3.10/socket.py", line 824 in create_connection
  File "/usr/lib/python3.10/http/client.py", line 941 in connect
  File "/usr/lib/python3.10/http/client.py", line 975 in send
  File "/usr/lib/python3.10/http/client.py", line 1037 in _send_output
  File "/usr/lib/python3.10/http/client.py", line 1277 in endheaders
  File "/usr/lib/python3.10/http/client.py", line 1328 in _send_request
  File "/usr/lib/python3.10/http/client.py", line 1282 in request
  File "/usr/lib/python3.10/urllib/request.py", line 1348 in do_open
  File "/usr/lib/python3.10/urllib/request.py", line 1377 in http_open
  File "/usr/lib/python3.10/urllib/request.py", line 496 in _call_chain
  File "/usr/lib/python3.10/urllib/request.py", line 536 in _open
  File "/usr/lib/python3.10/urllib/request.py", line 519 in open
  File "/usr/lib/python3.10/urllib/request.py", line 216 in urlopen
  File "/usr/src/app/bot/helper/ext_utils/bot_utils.py", line 247 in get_content_type
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58 in run
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 83 in _worker
  File "/usr/lib/python3.10/threading.py", line 953 in run
  File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007fb104816640 (most recent call first):
  File "/usr/local/lib/python3.10/dist-packages/requests_cache/backends/sqlite.py", line 228 in connection
  File "/usr/lib/python3.10/contextlib.py", line 135 in __enter__
  File "/usr/local/lib/python3.10/dist-packages/requests_cache/backends/sqlite.py", line 282 in __setitem__
  File "/usr/local/lib/python3.10/dist-packages/requests_cache/backends/base.py", line 91 in save_response
  File "/usr/local/lib/python3.10/dist-packages/requests_cache/session.py", line 234 in _send_and_cache
  File "/usr/local/lib/python3.10/dist-packages/requests_cache/session.py", line 206 in send
  File "/usr/local/lib/python3.10/dist-packages/requests/sessions.py", line 587 in request
  File "/usr/local/lib/python3.10/dist-packages/requests_cache/session.py", line 159 in request
  File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 59 in request
  File "/usr/local/lib/python3.10/dist-packages/requests/api.py", line 100 in head
  File "/usr/src/app/bot/helper/ext_utils/bot_utils.py", line 242 in get_content_type
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58 in run
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 83 in _worker
  File "/usr/lib/python3.10/threading.py", line 953 in run
  File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007fb0d4ed2640 (most recent call first):
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 81 in _worker
  File "/usr/lib/python3.10/threading.py", line 953 in run
  File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007fb0dffff640 (most recent call first):
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 81 in _worker
  File "/usr/lib/python3.10/threading.py", line 953 in run
  File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007fb0f4ddb640 (most recent call first):
  File "/usr/local/lib/python3.10/dist-packages/websocket/_socket.py", line 87 in _recv
  File "/usr/local/lib/python3.10/dist-packages/websocket/_socket.py", line 108 in recv
  File "/usr/local/lib/python3.10/dist-packages/websocket/_core.py", line 529 in _recv
  File "/usr/local/lib/python3.10/dist-packages/websocket/_abnf.py", line 373 in recv_strict
  File "/usr/local/lib/python3.10/dist-packages/websocket/_abnf.py", line 294 in recv_header
  File "/usr/local/lib/python3.10/dist-packages/websocket/_abnf.py", line 338 in recv_frame
  File "/usr/local/lib/python3.10/dist-packages/websocket/_core.py", line 445 in recv_frame
  File "/usr/local/lib/python3.10/dist-packages/websocket/_core.py", line 406 in recv_data_frame
  File "/usr/local/lib/python3.10/dist-packages/websocket/_core.py", line 385 in recv_data
  File "/usr/local/lib/python3.10/dist-packages/websocket/_core.py", line 362 in recv
  File "/usr/local/lib/python3.10/dist-packages/aria2p/client.py", line 1745 in listen_to_notifications
  File "/usr/lib/python3.10/threading.py", line 953 in run
  File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007fb0f55dc640 (most recent call first):
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 81 in _worker
  File "/usr/lib/python3.10/threading.py", line 953 in run
  File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007fb0f6efd640 (most recent call first):
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 81 in _worker
  File "/usr/lib/python3.10/threading.py", line 953 in run
  File "/usr/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/usr/lib/python3.10/threading.py", line 973 in _bootstrap

Thread 0x00007fb10907d000 (most recent call first):
  File "/usr/lib/python3.10/sre_compile.py", line 110 in _compile
  File "/usr/lib/python3.10/sre_compile.py", line 225 in _compile
  File "/usr/lib/python3.10/sre_compile.py", line 631 in _code
  File "/usr/lib/python3.10/sre_compile.py", line 792 in compile
  File "/usr/lib/python3.10/re.py", line 303 in _compile
  File "/usr/lib/python3.10/re.py", line 190 in match
  File "/usr/src/app/bot/modules/mirror_leech.py", line 156 in _mirror_leech
  File "/usr/src/app/bot/__main__.py", line 206 in <module>
  File "/usr/lib/python3.10/runpy.py", line 86 in _run_code
  File "/usr/lib/python3.10/runpy.py", line 196 in _run_module_as_main

Extension modules: tgcrypto, bson._cbson, pymongo._cmessage, _brotli, charset_normalizer.md, uvloop.loop, psutil._psutil_linux, psutil._psutil_posix, lxml._elementpath, lxml.etree, PIL._imaging, mega._mega, multidict._multidict, yarl._quoting_c, aiohttp._helpers, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket, frozenlist._frozenlist, _cffi_backend (total: 20)
start.sh: line 1:    31 Segmentation fault      (core dumped) python3 -m bot

Could you please help me to solve this problem? This problem exists in the parent repo (anasty17) too.

I searched and increased ulimit on the hosted machine, but the problem still exists.

System Info: Intel(R) Xeon(R) CPU E5-2699C v4 @ 2.20GHz CPU family: 6 Model: 79 Thread(s) per core: 1 Core(s) per socket: 2 Socket(s): 1 Stepping: 1 BogoMIPS: 4394.86

Ram 2 GB Storage 30 Gb (SSD) OS: Ubuntu 22.04 LTS (Jammy Jellyfish)

hitesh920 commented 1 year ago

your system is crashing due to low/poor specs the issue is with your server, not with the repo

m-hoseyny commented 1 year ago

Thanks for your reply. I believe it is not. I added a little sleep before download = (await sync_to_async(aria2.add, link, a2c_opt))[0] in aria2.downloader.py and also disabled sendStatusMessage when it comes updating the status message a lot in a little time interval and the problem has been solved.

FYI. I have been using the sync old parent repo for more than 1.5 years. So, I am aware of the structure of the code.

anasty17 commented 1 year ago

This from creating new thread... i will do it from one thread after repo install. But also ur server has low specifications

m-hoseyny commented 1 year ago

I think it comes from Aria2. Sometimes my uploads stuck with Aria2 too. Yes, my server config is low. But the container must not get an error with 4 forwarded leech commands.

Namlee123 commented 1 year ago

Hi @m-hoseyny maybe as you said, the problem is not related to low profile. I also had the same problem a while back, when using aria2 running multiple tasks (10-15 tasks) I even created a discussion on Anasty's original repo about "start.sh killer python -m bot". I handled it by restricting "sendStatusMessage". and it was error free.

anasty17 commented 1 year ago

I think it comes from Aria2. Sometimes my uploads stuck with Aria2 too. Yes, my server config is low. But the container must not get an error with 4 forwarded leech commands.

Not related to aria2c

anasty17 commented 1 year ago

Hi @m-hoseyny maybe as you said, the problem is not related to low profile. I also had the same problem a while back, when using aria2 running multiple tasks (10-15 tasks) I even created a discussion on Anasty's original repo about "start.sh killer python -m bot". I handled it by restricting "sendStatusMessage". and it was error free.

It's related to profile! And i should take this in consideration why both of u assuming! I have added 100 torrent together with multi task and no issues because my vps is from oracle

m-hoseyny commented 1 year ago

Thanks for your reply.

After my edit on code, I did not get the error anymore. I got what I wanted in my small server. Please check it with the leech command and aria2. I didn't get any error with qbittorrent either.

anasty17 commented 1 year ago

Thanks for your reply.

After my edit on code, I did not get the error anymore. I got what I wanted in my small server. Please check it with the leech command and aria2. I didn't get any error with qbittorrent either.

What u edit in repo ? SendStatusMessage on each task or the interval status update?

m-hoseyny commented 1 year ago

Actually, I changed two parts of the code:

  1. Restrict SendStatusMessage to get many updates in small intervals (delete the old status message and send it again when leech or qleech sent.)
  2. Add a little sleep before download = (await sync_to_async(aria2.add, link, a2c_opt))[0] in aria2.downloader.py (I mentioned them before.)

I didn't put further effort into finding out which one was effective. Telegram is very sensitive in updating and deleting messages and is very very slow in doing these actions (of course you know it better than me)

anasty17 commented 1 year ago

Actually, I changed two parts of the code:

  1. Restrict SendStatusMessage to get many updates in small intervals (delete the old status message and send it again when leech or qleech sent.)
  2. Add a little sleep before download = (await sync_to_async(aria2.add, link, a2c_opt))[0] in aria2.downloader.py (I mentioned them before.)

I didn't put further effort into finding out which one was effective. Telegram is very sensitive in updating and deleting messages and is very very slow in doing these actions (of course you know it better than me)

Well it's very fast in my tests even when i abuse all related to thread create

m-hoseyny commented 1 year ago

Humm.. well I think you can close the issue. I solved my problem on small server. Because it is on my business costs to get better server. This conversation will remain for other people with the same problem.

PS I checked the project with Hetzner Arm64 4 core cpu and 8 gb ram and still got the error.

anasty17 commented 1 year ago

Humm.. well I think you can close the issue. I solved my problem on small server. Because it is on my business costs to get better server. This conversation will remain for other people with the same problem.

PS I checked the project with Hetzner Arm64 4 core cpu and 8 gb ram and still got the error.

No I'm waiting repo to be reinstalled. Then will push fix for it and want u to test it because i can't produce this error

anasty17 commented 1 year ago

I have pushed commit mostly will fix the issue. @junedkh i have created only 1 extra thread for all blocking functions with 1000 worker it's not good idea to add this number but we need unlimited tasks 😂

Namlee123 commented 1 year ago

I think: the problem is in compatibility with vps, because of some restrictions of the provider, they are always limited to users in some regions. Or some restriction that they don't publish according to the laws of their country. Like some software or games, it runs high on Nvidia discrete cards but not necessarily works well on Radeon cards.

The repo is deployed and tested on oracle vps, that's an advantage for oracle users, of course there will be a better vps provider, but not sure to run this repo well. This is just my personal opinion.

hitesh920 commented 1 year ago

The repo is deployed and tested on oracle vps, that's an advantage for oracle users

"The repo is deployed and tested on oracle vps, that's an advantage for oracle users" bro you are stupid, thats not how this thing works

"This is just my personal opinion" no one asked you for a openion ok?

anasty17 commented 1 year ago

I think: the problem is in compatibility with vps, because of some restrictions of the provider, they are always limited to users in some regions. Or some restriction that they don't publish according to the laws of their country. Like some software or games, it runs high on Nvidia discrete cards but not necessarily works well on Radeon cards.

The repo is deployed and tested on oracle vps, that's an advantage for oracle users, of course there will be a better vps provider, but not sure to run this repo well. This is just my personal opinion.

Bruh i asked u to test after the commit! Did u tested it ? And if yes, what is the result ? What vps compatability? Explain to me this new thing!!

hitesh920 commented 1 year ago

I think: the problem is in compatibility with vps, because of some restrictions of the provider, they are always limited to users in some regions. Or some restriction that they don't publish according to the laws of their country. Like some software or games, it runs high on Nvidia discrete cards but not necessarily works well on Radeon cards. The repo is deployed and tested on oracle vps, that's an advantage for oracle users, of course there will be a better vps provider, but not sure to run this repo well. This is just my personal opinion.

Bruh i asked u to test after the commit! Did u tested it ? And if yes, what is the result ? What vps compatability? Explain to me this new thing!!

thats a diffrent guy xD

anasty17 commented 1 year ago

I think: the problem is in compatibility with vps, because of some restrictions of the provider, they are always limited to users in some regions. Or some restriction that they don't publish according to the laws of their country. Like some software or games, it runs high on Nvidia discrete cards but not necessarily works well on Radeon cards. The repo is deployed and tested on oracle vps, that's an advantage for oracle users, of course there will be a better vps provider, but not sure to run this repo well. This is just my personal opinion.

Bruh i asked u to test after the commit! Did u tested it ? And if yes, what is the result ? What vps compatability? Explain to me this new thing!!

thats a diffrent guy xD

Ohh, @m-hoseyny