search

EDM115 / unzip-bot

Telegram bot deployable to heroku that can extract every archive !
https://t.me/unzip_edm115bot
MIT License
103 stars 131 forks source link

Split files above 2 Gb & merge splitted files #26

Closed EDM115 closed 1 year ago

EDM115 commented 2 years ago
pr13260 commented 2 years ago

How about uploading files as zip or unziped to public clouds like anonfiles or user provided You can use APIs or Rclone it

EDM115 commented 2 years ago

@pr13260 something like an unzip to cloud (all unzipped files are separately uploaded to anonfiles or so) ?
Or something like merge the files then upload merged one to cloud ?

pr13260 commented 2 years ago

@pr13260 something like an unzip to cloud (all unzipped files are separately uploaded to anonfiles or so) ?
Or something like merge the files then upload merged one to cloud ?

any would work not all files with size more than 2gb should not be splitted and be uploaded to any cloud services this will make bot 🔥

EDM115 commented 2 years ago

Alright, I can work on this now

EDM115 commented 2 years ago

References :

pr13260 commented 2 years ago

Alright, I can work on this now

Awesome 👍👍

EDM115 commented 1 year ago 
2023-07-01T18:02:33.207948+00:00 app[worker.1]: 2023-07-01 18:02:33,169 - INFO - unzipper - asyncio_1 - 
2023-07-01T18:02:33.207964+00:00 app[worker.1]: 7-Zip [64] 17.04 : Copyright (c) 1999-2021 Igor Pavlov : 2017-08-28
2023-07-01T18:02:33.207965+00:00 app[worker.1]: p7zip Version 17.04 (locale=C.UTF-8,Utf16=on,HugeFiles=on,64 bits,8 CPUs x64)
2023-07-01T18:02:33.207966+00:00 app[worker.1]: 
2023-07-01T18:02:33.207966+00:00 app[worker.1]: Scanning the drive for archives:
2023-07-01T18:02:33.207966+00:00 app[worker.1]: 1 file, 2431906561 bytes (2320 MiB)
2023-07-01T18:02:33.207967+00:00 app[worker.1]: 
2023-07-01T18:02:33.207967+00:00 app[worker.1]: Extracting archive: /app/Downloaded/854158484/archive_from_854158484.zip
2023-07-01T18:02:33.207967+00:00 app[worker.1]: --
2023-07-01T18:02:33.207967+00:00 app[worker.1]: Path = /app/Downloaded/854158484/archive_from_854158484.zip
2023-07-01T18:02:33.207968+00:00 app[worker.1]: Type = zip
2023-07-01T18:02:33.207968+00:00 app[worker.1]: Physical Size = 2431906561
2023-07-01T18:02:33.207968+00:00 app[worker.1]: 
2023-07-01T18:02:33.207969+00:00 app[worker.1]: Everything is Ok
2023-07-01T18:02:33.207969+00:00 app[worker.1]: 
2023-07-01T18:02:33.207969+00:00 app[worker.1]: Folders: 1
2023-07-01T18:02:33.207969+00:00 app[worker.1]: Files: 2
2023-07-01T18:02:33.207969+00:00 app[worker.1]: Size:       2431905867
2023-07-01T18:02:33.207970+00:00 app[worker.1]: Compressed: 2431906561
2023-07-01T18:02:44.082203+00:00 app[worker.1]: 2023-07-01 18:02:44,082 - INFO - unzipper - MainThread - ext_a paths : ['/app/Downloaded/854158484/extracted/Interstellar (2014) (2014) [1080p]/Interstellar.2014.2014.1080p.BluRay.x264.YIFY.mp4', '/app/Downloaded/854158484/extracted/Interstellar (2014) (2014) [1080p]/WWW.YTS.RE.jpg']
2023-07-01T18:02:45.141825+00:00 app[worker.1]: 2023-07-01 18:02:45,141 - WARNING - pyrogram.session.session - MainThread - [UnzipperBot] Waiting for 3 seconds before continuing (required by "messages.SendMedia")
2023-07-01T18:03:16.618827+00:00 app[worker.1]: 2023-07-01 18:03:16,594 - WARNING - pyrogram.session.session - MainThread - [UnzipperBot] Waiting for 3 seconds before continuing (required by "messages.SendMessage")
2023-07-01T18:03:40.174314+00:00 app[worker.1]: 2023-07-01 18:03:40,173 - INFO - unzipper - asyncio_0 - 
2023-07-01T18:03:40.174336+00:00 app[worker.1]: 7-Zip [64] 17.04 : Copyright (c) 1999-2021 Igor Pavlov : 2017-08-28
2023-07-01T18:03:40.174337+00:00 app[worker.1]: p7zip Version 17.04 (locale=C.UTF-8,Utf16=on,HugeFiles=on,64 bits,8 CPUs x64)
2023-07-01T18:03:40.174338+00:00 app[worker.1]: 
2023-07-01T18:03:40.174339+00:00 app[worker.1]: Scanning the drive:
2023-07-01T18:03:40.174340+00:00 app[worker.1]: 1 file, 2431697820 bytes (2320 MiB)
2023-07-01T18:03:40.174341+00:00 app[worker.1]: 
2023-07-01T18:03:40.174350+00:00 app[worker.1]: Creating archive: /app/Downloaded/splitted/854158484/Interstellar.2014.2014.1080p.BluRay.x264.YIFY.mp4
2023-07-01T18:03:40.174350+00:00 app[worker.1]: 
2023-07-01T18:03:40.174350+00:00 app[worker.1]: Items to compress: 1
2023-07-01T18:03:40.174351+00:00 app[worker.1]: 
2023-07-01T18:03:40.174351+00:00 app[worker.1]: 
2023-07-01T18:03:40.174351+00:00 app[worker.1]: Files read from disk: 1
2023-07-01T18:03:40.174352+00:00 app[worker.1]: Archive size: 2431698052 bytes (2320 MiB)
2023-07-01T18:03:40.174352+00:00 app[worker.1]: Everything is Ok
2023-07-01T18:03:40.767309+00:00 app[worker.1]: 2023-07-01 18:03:40,767 - WARNING - unzipper - MainThread - Can't upload files bigger than 2000 MiB
2023-07-01T18:03:40.890045+00:00 app[worker.1]: 2023-07-01 18:03:40,889 - WARNING - unzipper - MainThread - Failed to decode "/app/Downloaded/splitted/854158484/Interstellar.2014.2014.1080p.BluRay.x264.YIFY.mp4.002". The value does not represent an existing local file, HTTP URL, or valid file id.
2023-07-01T18:03:40.894168+00:00 app[worker.1]: 2023-07-01 18:03:40,890 - ERROR - pyrogram.dispatcher - MainThread - [Errno 2] No such file or directory: '/app/Downloaded/splitted/854158484'
2023-07-01T18:03:40.894169+00:00 app[worker.1]: Traceback (most recent call last):
2023-07-01T18:03:40.894169+00:00 app[worker.1]:   File "/venv/lib/python3.11/site-packages/pyrogram/file_id.py", line 228, in decode
2023-07-01T18:03:40.894169+00:00 app[worker.1]:     file_type = FileType(file_type)
2023-07-01T18:03:40.894170+00:00 app[worker.1]:                 ^^^^^^^^^^^^^^^^^^^
2023-07-01T18:03:40.894170+00:00 app[worker.1]:   File "/usr/lib/python3.11/enum.py", line 714, in __call__
2023-07-01T18:03:40.894170+00:00 app[worker.1]:     return cls.__new__(cls, value)
2023-07-01T18:03:40.894170+00:00 app[worker.1]:            ^^^^^^^^^^^^^^^^^^^^^^^
2023-07-01T18:03:40.894171+00:00 app[worker.1]:   File "/usr/lib/python3.11/enum.py", line 1130, in __new__
2023-07-01T18:03:40.894171+00:00 app[worker.1]:     raise ve_exc
2023-07-01T18:03:40.894171+00:00 app[worker.1]: ValueError: -60183811 is not a valid FileType
2023-07-01T18:03:40.894171+00:00 app[worker.1]: 
2023-07-01T18:03:40.894172+00:00 app[worker.1]: During handling of the above exception, another exception occurred:
2023-07-01T18:03:40.894172+00:00 app[worker.1]: 
2023-07-01T18:03:40.894172+00:00 app[worker.1]: Traceback (most recent call last):
2023-07-01T18:03:40.894173+00:00 app[worker.1]:   File "/venv/lib/python3.11/site-packages/pyrogram/utils.py", line 49, in get_input_media_from_file_id
2023-07-01T18:03:40.894173+00:00 app[worker.1]:     decoded = FileId.decode(file_id)
2023-07-01T18:03:40.894173+00:00 app[worker.1]:               ^^^^^^^^^^^^^^^^^^^^^^
2023-07-01T18:03:40.894173+00:00 app[worker.1]:   File "/venv/lib/python3.11/site-packages/pyrogram/file_id.py", line 230, in decode
2023-07-01T18:03:40.894174+00:00 app[worker.1]:     raise ValueError(f"Unknown file_type {file_type} of file_id {file_id}")
2023-07-01T18:03:40.894174+00:00 app[worker.1]: ValueError: Unknown file_type -60183811 of file_id /app/Downloaded/splitted/854158484/Interstellar.2014.2014.1080p.BluRay.x264.YIFY.mp4.002
2023-07-01T18:03:40.894174+00:00 app[worker.1]: 
2023-07-01T18:03:40.894174+00:00 app[worker.1]: During handling of the above exception, another exception occurred:
2023-07-01T18:03:40.894174+00:00 app[worker.1]: 
2023-07-01T18:03:40.894174+00:00 app[worker.1]: Traceback (most recent call last):
2023-07-01T18:03:40.894175+00:00 app[worker.1]:   File "/app/unzipper/modules/ext_script/up_helper.py", line 144, in send_file
2023-07-01T18:03:40.894175+00:00 app[worker.1]:     await unzip_bot.send_document(
2023-07-01T18:03:40.894175+00:00 app[worker.1]:   File "/venv/lib/python3.11/site-packages/pyrogram/methods/messages/send_document.py", line 178, in send_document
2023-07-01T18:03:40.894175+00:00 app[worker.1]:     media = utils.get_input_media_from_file_id(document, FileType.DOCUMENT)
2023-07-01T18:03:40.894175+00:00 app[worker.1]:             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-07-01T18:03:40.894176+00:00 app[worker.1]:   File "/venv/lib/python3.11/site-packages/pyrogram/utils.py", line 51, in get_input_media_from_file_id
2023-07-01T18:03:40.894176+00:00 app[worker.1]:     raise ValueError(
2023-07-01T18:03:40.894177+00:00 app[worker.1]: ValueError: Failed to decode "/app/Downloaded/splitted/854158484/Interstellar.2014.2014.1080p.BluRay.x264.YIFY.mp4.002". The value does not represent an existing local file, HTTP URL, or valid file id.
2023-07-01T18:03:40.894177+00:00 app[worker.1]: 
2023-07-01T18:03:40.894178+00:00 app[worker.1]: During handling of the above exception, another exception occurred:
2023-07-01T18:03:40.894178+00:00 app[worker.1]: 
2023-07-01T18:03:40.894178+00:00 app[worker.1]: Traceback (most recent call last):
2023-07-01T18:03:40.894178+00:00 app[worker.1]:   File "/venv/lib/python3.11/site-packages/pyrogram/dispatcher.py", line 240, in handler_worker
2023-07-01T18:03:40.894178+00:00 app[worker.1]:     await handler.callback(self.client, *args)
2023-07-01T18:03:40.894178+00:00 app[worker.1]:   File "/app/unzipper/modules/callbacks.py", line 667, in unzipper_cb
2023-07-01T18:03:40.894179+00:00 app[worker.1]:     await send_file(
2023-07-01T18:03:40.894179+00:00 app[worker.1]:   File "/app/unzipper/modules/ext_script/up_helper.py", line 181, in send_file
2023-07-01T18:03:40.894179+00:00 app[worker.1]:     shutil.rmtree(full_path)
2023-07-01T18:03:40.894179+00:00 app[worker.1]:   File "/usr/lib/python3.11/shutil.py", line 722, in rmtree
2023-07-01T18:03:40.894179+00:00 app[worker.1]:     onerror(os.lstat, path, sys.exc_info())
2023-07-01T18:03:40.894179+00:00 app[worker.1]:   File "/usr/lib/python3.11/shutil.py", line 720, in rmtree
2023-07-01T18:03:40.894180+00:00 app[worker.1]:     orig_st = os.lstat(path, dir_fd=dir_fd)
2023-07-01T18:03:40.894180+00:00 app[worker.1]:               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2023-07-01T18:03:40.894180+00:00 app[worker.1]: FileNotFoundError: [Errno 2] No such file or directory: '/app/Downloaded/splitted/854158484'

I assume that the 2g file size is slightly too big

EDM115 commented 1 year ago 
2023-07-01T18:49:52.288423+00:00 app[worker.1]: 2023-07-01 18:49:52,288 - INFO - unzipper - MainThread - ext_a paths : ['/app/Downloaded/854158484/extracted/Interstellar (2014) (2014) [1080p]/Interstellar.2014.2014.1080p.BluRay.x264.YIFY.mp4', '/app/Downloaded/854158484/extracted/Interstellar (2014) (2014) [1080p]/WWW.YTS.RE.jpg']
2023-07-01T18:50:38.187771+00:00 app[worker.1]: 2023-07-01 18:50:38,186 - INFO - unzipper - asyncio_1 - 
2023-07-01T18:50:38.187783+00:00 app[worker.1]: 7-Zip [64] 17.04 : Copyright (c) 1999-2021 Igor Pavlov : 2017-08-28
2023-07-01T18:50:38.187784+00:00 app[worker.1]: p7zip Version 17.04 (locale=C.UTF-8,Utf16=on,HugeFiles=on,64 bits,8 CPUs x64)
2023-07-01T18:50:38.187784+00:00 app[worker.1]: 
2023-07-01T18:50:38.187785+00:00 app[worker.1]: Scanning the drive:
2023-07-01T18:50:38.187785+00:00 app[worker.1]: 1 file, 2431697820 bytes (2320 MiB)
2023-07-01T18:50:38.187785+00:00 app[worker.1]: 
2023-07-01T18:50:38.187786+00:00 app[worker.1]: Creating archive: /app/Downloaded/splitted/854158484/Interstellar.2014.2014.1080p.BluRay.x264.YIFY.mp4
2023-07-01T18:50:38.187786+00:00 app[worker.1]: 
2023-07-01T18:50:38.187786+00:00 app[worker.1]: Items to compress: 1
2023-07-01T18:50:38.187787+00:00 app[worker.1]: 
2023-07-01T18:50:38.187787+00:00 app[worker.1]: 
2023-07-01T18:50:38.187787+00:00 app[worker.1]: Files read from disk: 1
2023-07-01T18:50:38.187787+00:00 app[worker.1]: Archive size: 2431698052 bytes (2320 MiB)
2023-07-01T18:50:38.187787+00:00 app[worker.1]: Everything is Ok
2023-07-01T18:50:38.188058+00:00 app[worker.1]: 2023-07-01 18:50:38,187 - INFO - unzipper - MainThread - ['/app/Downloaded/splitted/854158484/Interstellar.2014.2014.1080p.BluRay.x264.YIFY.mp4.001', '/app/Downloaded/splitted/854158484/Interstellar.2014.2014.1080p.BluRay.x264.YIFY.mp4.002']

Solved with c1528bc5b72f21f15f0d6051f0e5ce202d3541f2 and 4afea91dd8828748a5791d7b444824c0c2e5c616

Now it's time to fine tune this, as uploaded files are 1907,3 Mb