azamaulanaaa / botkaca

Bot that leech (re-upload) internet contents including torrent to telegram
GNU General Public License v3.0
133 stars 610 forks source link

BOT CRASHED ON DOWNLOAD DURING LARGE DATA REQUEST #3

Open talkaal opened 4 years ago

talkaal commented 4 years ago

A 38 GB (single files each) file torrent was put but it happened to crash after a while.

As well as two feature request also if possible add!

A minor 🐞Bug

Log of the activity ( HEROKU)

2020-09-29T10:26:57.191869+00:00 app[worker.1]: During handling of the above exception, another exception occurred: 2020-09-29T10:26:57.191869+00:00 app[worker.1]: 2020-09-29T10:26:57.191869+00:00 app[worker.1]: Traceback (most recent call last): 2020-09-29T10:26:57.191870+00:00 app[worker.1]: File "/app/bot/handlers/leech_handler.py", line 110, in progress_dl 2020-09-29T10:26:57.191870+00:00 app[worker.1]: await message.edit( 2020-09-29T10:26:57.191871+00:00 app[worker.1]: File "/usr/local/lib/python3.8/site-packages/pyrogram/client/types/messages_and_media/message.py", line 2507, in edit_text 2020-09-29T10:26:57.191872+00:00 app[worker.1]: return await self._client.edit_message_text( 2020-09-29T10:26:57.191873+00:00 app[worker.1]: File "/usr/local/lib/python3.8/site-packages/pyrogram/client/methods/messages/edit_message_text.py", line 78, in edit_message_text 2020-09-29T10:26:57.191873+00:00 app[worker.1]: r = await self.send( 2020-09-29T10:26:57.191874+00:00 app[worker.1]: File "/usr/local/lib/python3.8/site-packages/pyrogram/client/client.py", line 1424, in send 2020-09-29T10:26:57.191874+00:00 app[worker.1]: r = await self.session.send(data, retries, timeout, self.sleep_threshold) 2020-09-29T10:26:57.191874+00:00 app[worker.1]: File "/usr/local/lib/python3.8/site-packages/pyrogram/session/session.py", line 451, in send 2020-09-29T10:26:57.191875+00:00 app[worker.1]: return await self.send(data, retries - 1, timeout) 2020-09-29T10:26:57.191875+00:00 app[worker.1]: File "/usr/local/lib/python3.8/site-packages/pyrogram/session/session.py", line 429, in send 2020-09-29T10:26:57.191876+00:00 app[worker.1]: return await self._send(data, timeout=timeout) 2020-09-29T10:26:57.191876+00:00 app[worker.1]: File "/usr/local/lib/python3.8/site-packages/pyrogram/session/session.py", line 399, in _send 2020-09-29T10:26:57.191877+00:00 app[worker.1]: RPCError.raise_it(result, type(data)) 2020-09-29T10:26:57.191877+00:00 app[worker.1]: File "/usr/local/lib/python3.8/site-packages/pyrogram/errors/rpc_error.py", line 79, in raise_it 2020-09-29T10:26:57.191878+00:00 app[worker.1]: raise getattr( 2020-09-29T10:26:57.191878+00:00 app[worker.1]: pyrogram.errors.exceptions.bad_request_400.MessageNotModified: [400 MESSAGE_NOT_MODIFIED]: The message was not modified (caused by "messages.EditMessage")

azamaulanaaa commented 4 years ago

everything you ask for are possible, it just me don't add it. I will add it soon.

A minor 🐞Bug

  • Thumbnail generation in video before upload is much slower.

for thumbnail is so slow due to ffmpeg needs to seek to the middle of video to generates thumbnail. from many post I read, it is possible to just jump to the middle to snapshot instead of seek but the drawback is sometimes it does not work, in my case mostly does not work. the solution I knew is snapshot the first frame, the drawback it does not represent the video. I still working with it along side optimization for video split.

A 38 GB (single files each) file torrent was put but it happened to crash after a while.

for large file, as far as I know is due to limitation from the heroku itself. I did 5gb+ and most the time the file gone from the service. the app already optimize for binary split ( like hjsplit does) but it just help upto 4-5gb files. I am still working on this issue on my other project torrent-mirror with nodejs since python does not have torrent module natively

talkaal commented 4 years ago

CRASHED AGAIN THIS TIME AS PER YOU HAVE TOLD 3GB TORRENT I'VE ADDED.

LOGS

2020-09-29T18:48:17.398735+00:00 app[worker.1]: File "/usr/local/lib/python3.8/email/message.py", line 578, in get_content_type 2020-09-29T18:48:17.398737+00:00 app[worker.1]: value = self.get('content-type', missing) 2020-09-29T18:48:17.398737+00:00 app[worker.1]: File "/usr/local/lib/python3.8/email/message.py", line 471, in get 2020-09-29T18:48:17.398738+00:00 app[worker.1]: return self.policy.header_fetch_parse(k, v) 2020-09-29T18:48:17.398738+00:00 app[worker.1]: File "/usr/local/lib/python3.8/email/_policybase.py", line 316, in header_fetch_parse 2020-09-29T18:48:17.398738+00:00 app[worker.1]: return self._sanitize_header(name, value) 2020-09-29T18:48:17.398739+00:00 app[worker.1]: File "/usr/local/lib/python3.8/email/_policybase.py", line 287, in _sanitize_header 2020-09-29T18:48:17.398739+00:00 app[worker.1]: if _has_surrogates(value): 2020-09-29T18:48:17.398740+00:00 app[worker.1]: File "/usr/local/lib/python3.8/email/utils.py", line 57, in _has_surrogates 2020-09-29T18:48:17.398740+00:00 app[worker.1]: s.encode() 2020-09-29T18:48:17.398741+00:00 app[worker.1]: RecursionError: maximum recursion depth exceeded while calling a Python object 2020-09-29T18:48:17.398741+00:00 app[worker.1]: 2020-09-29T18:48:17.398742+00:00 app[worker.1]: During handling of the above exception, another exception occurred: 2020-09-29T18:48:17.398742+00:00 app[worker.1]: 2020-09-29T18:48:17.398743+00:00 app[worker.1]: Traceback (most recent call last): 2020-09-29T18:48:17.398743+00:00 app[worker.1]: File "/app/bot/handlers/leech_handler.py", line 124, in progress_dl 2020-09-29T18:48:17.398744+00:00 app[worker.1]: return await progress_dl(message, aria2_api, gid, text) 2020-09-29T18:48:17.398745+00:00 app[worker.1]: File "/app/bot/handlers/leech_handler.py", line 139, in progress_dl 2020-09-29T18:48:17.398745+00:00 app[worker.1]: download.remove(force=True) 2020-09-29T18:48:17.398754+00:00 app[worker.1]: UnboundLocalError: local variable 'download' referenced before assignment

Minor UI Alteration if possible

Future feature addition Request (Just a thought)

Thank you for your Interaction based quick response and necessary updates. Hope that I'm not pressurizing you😅

azamaulanaaa commented 4 years ago

hi @talkaal, mind if you share the torrent? so I can replicate your bug

azamaulanaaa commented 4 years ago
  • The download progress blocks be extended to 20 rather than currently 10.

let me give you option to customize it from local/default.py, please wait.

  • Add to Specific telegram group for usage rather than individual bot interaction. Multiple users if using the same bot run on the script, the connection to bot will be more and may result in Speed reduction on individual bot interaction.
  • In group this problem might be rectified as group will be a single interactor for request providing files. you can actually add the bot to specific group.
  1. set the password
  2. add the bot to the group
  3. verify the bot at that group using command /pass [yourpass]
  • A command to know available server space

I can actually do it but there is a catch. Since most service using shared storage, it might not represent actual storage space for you.

another command to clear residual downloads in the server still remaining this can be used to add torrent after previous file request are uploaded

Pardon me, I do not understand.

Note : please create new issue with label enhancement for requested feature. thank you.

talkaal commented 4 years ago

hi @talkaal, mind if you share the torrent? so I can replicate your bug

Sure

magnet:?xt=urn:btih:15610ff6ee0d0fe11d491c805cf141235f184e32&tr=udp%3A%2F%2Ftracker.leechers-paradise.org%3A6969%2Fannounce&tr=udp%3A%2F%2Fp4p.arenabg.com%3A1337%2Fannounce&tr=udp%3A%2F%2F9.rarbg.me%3A2710%2Fannounce&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80%2Fannounce&tr=udp%3A%2F%2Ftracker.moeking.me%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.cyberia.is%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker4.itzmx.com%3A2710%2Fannounce&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.internetwarriors.net%3A1337%2Fannounce&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce&tr=udp%3A%2F%2F9.rarbg.to%3A2710%2Fannounce&tr=udp%3A%2F%2Fexodus.desync.com%3A6969%2Fannounce&tr=udp%3A%2F%2Fexplodie.org%3A6969%2Fannounce&tr=udp%3A%2F%2Fpublic.popcorn-tracker.org%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.vanitycore.co%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker1.itzmx.com%3A8080%2Fannounce&tr=udp%3A%2F%2Fipv4.tracker.harry.lu%3A80%2Fannounce&tr=udp%3A%2F%2Ftracker.tiny-vps.com%3A6969%2Fannounce&tr=udp%3A%2F%2Fdenis.stalker.upeer.me%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.torrent.eu.org%3A451%2Fannounce&tr=udp%3A%2F%2Ftracker.port443.xyz%3A6969%2Fannounce&tr=udp%3A%2F%2Fopen.stealth.si%3A80%2Fannounce&tr=udp%3A%2F%2Fopen.demonii.si%3A1337%2Fannounce&tr=udp%3A%2F%2Fbt.xxx-tracker.com%3A2710%2Fannounce&tr=udp%3A%2F%2Fthetracker.org%3A80%2Fannounce&tr=udp%3A%2F%2Fretracker.lanta-net.ru%3A2710%2Fannounce&tr=udp%3A%2F%2Ftracker.uw0.xyz%3A6969%2Fannounce&dn=%5BSolidTorrents.net%5D+Sense8+Season+2+S02+720p+WEBRip+x265+HEVC+Crazy4ad

talkaal commented 4 years ago

Pardon me, I do not understand.

Since the files are transfered first onto running server and reuploaded to Telegram. There's a chance that the files on server might not delete automatically, so in that case clearing out some space available on server might help.

That's what I meant hope you understood?

talkaal commented 4 years ago

let me give you option to customize it from local/default.py, please wait.

Did you change to customizable variable?

azamaulanaaa commented 4 years ago

Since the files are transfered first onto running server and reuploaded to Telegram. There's a chance that the files on server might not delete automatically, so in that case clearing out some space available on server might help.

oh I see, it already been done. the app already optimize for that condition. every files that uploaded will deleted automatically. So if you face some problem after file downloaded. the files might stays there. if you face while downloading, you can cancel them and the files automatically deleted

azamaulanaaa commented 4 years ago

@talkaal now, you can customize bar size on ./__init__.py. sorry not on local/default.py

azamaulanaaa commented 4 years ago

hi @talkaal, mind if you share the torrent? so I can replicate your bug

Sure

magnet:?xt=urn:btih:15610ff6ee0d0fe11d491c805cf141235f184e32&tr=udp%3A%2F%2Ftracker.leechers-paradise.org%3A6969%2Fannounce&tr=udp%3A%2F%2Fp4p.arenabg.com%3A1337%2Fannounce&tr=udp%3A%2F%2F9.rarbg.me%3A2710%2Fannounce&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A80%2Fannounce&tr=udp%3A%2F%2Ftracker.moeking.me%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.cyberia.is%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker4.itzmx.com%3A2710%2Fannounce&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.internetwarriors.net%3A1337%2Fannounce&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce&tr=udp%3A%2F%2F9.rarbg.to%3A2710%2Fannounce&tr=udp%3A%2F%2Fexodus.desync.com%3A6969%2Fannounce&tr=udp%3A%2F%2Fexplodie.org%3A6969%2Fannounce&tr=udp%3A%2F%2Fpublic.popcorn-tracker.org%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.vanitycore.co%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker1.itzmx.com%3A8080%2Fannounce&tr=udp%3A%2F%2Fipv4.tracker.harry.lu%3A80%2Fannounce&tr=udp%3A%2F%2Ftracker.tiny-vps.com%3A6969%2Fannounce&tr=udp%3A%2F%2Fdenis.stalker.upeer.me%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.torrent.eu.org%3A451%2Fannounce&tr=udp%3A%2F%2Ftracker.port443.xyz%3A6969%2Fannounce&tr=udp%3A%2F%2Fopen.stealth.si%3A80%2Fannounce&tr=udp%3A%2F%2Fopen.demonii.si%3A1337%2Fannounce&tr=udp%3A%2F%2Fbt.xxx-tracker.com%3A2710%2Fannounce&tr=udp%3A%2F%2Fthetracker.org%3A80%2Fannounce&tr=udp%3A%2F%2Fretracker.lanta-net.ru%3A2710%2Fannounce&tr=udp%3A%2F%2Ftracker.uw0.xyz%3A6969%2Fannounce&dn=%5BSolidTorrents.net%5D+Sense8+Season+2+S02+720p+WEBRip+x265+HEVC+Crazy4ad

looks like the torrent has no seeder

azamaulanaaa commented 4 years ago

sorry wrong commit reference

talkaal commented 3 years ago

The crash of large torrents after a while might be due to two factors of time I think. Don't know whether it's helpful.

Maximum process timeout Maximum time to wait for torrent to start

I hope it'll be a valuable thought. Thank you for your effort on the code

PLEASE DO CLOSE IN CASE THE THOUGHT IS AN INVALID ONE.

fernandooliveira2 commented 3 years ago

Since the files are transfered first onto running server and reuploaded to Telegram. There's a chance that the files on server might not delete automatically, so in that case clearing out some space available on server might help.

oh I see, it already been done. the app already optimize for that condition. every files that uploaded will deleted automatically. So if you face some problem after file downloaded. the files might stays there. if you face while downloading, you can cancel them and the files automatically deleted

Any chance to youtube download too? 👍

azamaulanaaa commented 3 years ago

Since the files are transfered first onto running server and reuploaded to Telegram. There's a chance that the files on server might not delete automatically, so in that case clearing out some space available on server might help.

oh I see, it already been done. the app already optimize for that condition. every files that uploaded will deleted automatically. So if you face some problem after file downloaded. the files might stays there. if you face while downloading, you can cancel them and the files automatically deleted

Any chance to youtube download too? 👍

sorry out of topic