search

pyload / pyload

The free and open-source Download Manager written in pure Python
https://pyload.net
Other
3.28k stars 709 forks source link

Very slow to combine chunks #4421

Open DrekoDev opened 8 months ago

DrekoDev commented 8 months ago

Description

Hi

I use PyLoad via a seedbox service (seedbox.fr).

It works very well for downloading videos, but it takes a lot of time to combine different chunks. I recently downloaded a movie and it took more than 20 minutes to combine to finalize the download.

I had to set the "Maximum connections for one download" setting to 1 to decrease that length of time, because before that, it took half an hour.

Debug log

06.01.2024 16:12:34 | INFO | ADDON UpdateManager: All plugins are up to date! 06.01.2024 16:12:40 | INFO | ADDON ClickNLoad: Proxy listening on 0.0.0.0:9666 06.01.2024 19:45:33 | INFO | Added package Equalizer containing 1 links 06.01.2024 19:45:33 | DEBUG | Redirected import module.plugins.internal.SimpleHoster -> userplugins.internal.SimpleHoster 06.01.2024 19:45:34 | INFO | Download starts: https://1fichier.com/?4en40rf9uxeoc9ik02sf&af=4814702 06.01.2024 19:45:34 | DEBUG | ADDON ExternalScripts: No script found under folder download_preparing 06.01.2024 19:45:34 | DEBUG | ADDON UserAgentSwitcher: Setting connection timeout to 60 seconds 06.01.2024 19:45:34 | DEBUG | ADDON UserAgentSwitcher: Setting maximum redirections to 10 06.01.2024 19:45:34 | DEBUG | ADDON UserAgentSwitcher: Use custom user-agent string Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:71.0) Gecko/20100101 Firefox/71.0 06.01.2024 19:45:34 | DEBUG | HOSTER OneFichierCom[58]: Plugin version: 1.19 06.01.2024 19:45:34 | DEBUG | HOSTER OneFichierCom[58]: Plugin status: testing 06.01.2024 19:45:34 | WARNING | HOSTER OneFichierCom[58]: Plugin may be unstable 06.01.2024 19:45:34 | DEBUG | ACCOUNT OneFichierCom: Reached login timeout for user dre******* 06.01.2024 19:45:34 | INFO | ACCOUNT OneFichierCom: Relogin user dre*******... 06.01.2024 19:45:34 | DEBUG | ACCOUNT OneFichierCom: LOAD URL https://1fichier.com/login.pl?lg=en | redirect=True | cookies=True | get={} | req=None | decode=True | multipart=False | post={} | ref=True | just_header=False 06.01.2024 19:45:34 | WARNING | ACCOUNT OneFichierCom: Skipped login user dre******* | Already signed in 06.01.2024 19:45:34 | INFO | ACCOUNT OneFichierCom: Grabbing account info for user dre*******... 06.01.2024 19:45:34 | DEBUG | ACCOUNT OneFichierCom: LOAD URL https://1fichier.com/console/abo.pl | redirect=True | cookies=True | get={} | req=None | decode=True | multipart=False | post={} | ref=True | just_header=False 06.01.2024 19:45:34 | DEBUG | ACCOUNT OneFichierCom: Expire date: 2024-07-23 06.01.2024 19:45:34 | DEBUG | ACCOUNT OneFichierCom: Account info for user dre*******: {'login': {'timestamp': 1704570334.050134, 'password': '**', 'stats': [2, 0], 'valid': True}, 'data': {'validuntil': 1721689200.0, 'plugin': , 'trafficleft': -1, 'maxtraffic': None, 'premium': True, 'login': 'dre', 'type': 'OneFichierCom', 'options': {'limitDL': ['0']}}} 06.01.2024 19:45:34 | DEBUG | ACCOUNT OneFichierCom: Using account dre 06.01.2024 19:45:34 | INFO | HOSTER OneFichierCom[58]: Processing url: https://1fichier.com/?4en40rf9uxeoc9ik02sf&af=4814702 06.01.2024 19:45:34 | INFO | HOSTER OneFichierCom[58]: Looking for direct download link... 06.01.2024 19:45:34 | DEBUG | HOSTER OneFichierCom[58]: LOAD URL https://1fichier.com/?4en40rf9uxeoc9ik02sf | redirect=True | cookies=True | get={} | req=None | decode=True | multipart=False | post={} | ref=True | just_header=True 06.01.2024 19:45:34 | DEBUG | HOSTER OneFichierCom[58]: Redirect #1 to: https://c-1.1fichier.com/p916877324 06.01.2024 19:45:34 | DEBUG | HOSTER OneFichierCom[58]: LOAD URL https://c-1.1fichier.com/p916877324 | redirect=True | cookies=True | get={} | req=None | decode=True | multipart=False | post={} | ref=True | justheader=True 06.01.2024 19:45:35 | INFO | HOSTER OneFichierCom[58]: Direct download link detected 06.01.2024 19:45:35 | INFO | HOSTER OneFichierCom[58]: Downloading file... 06.01.2024 19:45:35 | DEBUG | HOSTER OneFichierCom[58]: DOWNLOAD URL https://1fichier.com/?4en40rf9uxeoc9ik02sf | cookies=True | resume=None | get={} | disposition=True | chunks=None | post={} | ref=True 06.01.2024 19:45:36 | DEBUG | Content-Disposition: **(Name of the movie)_** 06.01.2024 19:45:36 | DEBUG | Chunked with range 0-

DrekoDev commented 7 months ago

Hi @GammaC0de , do you have any clue ? If you need more logs here they are

` 7 23.01.2024 18:05:25 DEBUG ADDON UserAgentSwitcher: Use custom user-agent string Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:71.0) Gecko/20100101 Firefox/71.0
38 23.01.2024 18:05:25 DEBUG HOSTER OneFichierCom[240]: Plugin version: 1.19
39 23.01.2024 18:05:25 DEBUG HOSTER OneFichierCom[240]: Plugin status: testing
40 23.01.2024 18:05:25 WARNING HOSTER OneFichierCom[240]: Plugin may be unstable
41 23.01.2024 18:05:25 INFO ACCOUNT OneFichierCom: Grabbing account info for user dre*******...
42 23.01.2024 18:05:25 DEBUG ACCOUNT OneFichierCom: LOAD URL https://1fichier.com/console/abo.pl | redirect=True | cookies=True | get={} | req=None | decode=True | multipart=False | post={} | ref=True | just_header=False
43 23.01.2024 18:05:25 DEBUG ACCOUNT OneFichierCom: Expire date: 2024-07-23
44 23.01.2024 18:05:25 DEBUG ACCOUNT OneFichierCom: Account info for user dre*******: {'login': {'timestamp': 1706032894.382562, 'password': '**', 'stats': [2, 1706033009.192815], 'valid': True}, 'data': {'validuntil': 1721689200.0, 'plugin': , 'trafficleft': -1, 'maxtraffic': None, 'premium': True, 'login': 'dre***', 'type': 'OneFichierCom', 'options': {'limitDL': ['0']}}}
45 23.01.2024 18:05:25 DEBUG ACCOUNT OneFichierCom: Using account dre***
46 23.01.2024 18:05:25 INFO HOSTER OneFichierCom[240]: Processing url: https://1fichier.com/?x1pkrjifk6w5gezkcop0&af=4814702
47 23.01.2024 18:05:25 INFO HOSTER OneFichierCom[240]: Looking for direct download link...
48 23.01.2024 18:05:25 DEBUG HOSTER OneFichierCom[240]: LOAD URL https://1fichier.com/?x1pkrjifk6w5gezkcop0 | redirect=False | cookies=True | get={} | req=None | decode=True | multipart=False | post={} | ref=True | just_header=True
49 23.01.2024 18:05:27 DEBUG HOSTER OneFichierCom[240]: Redirect #1 to: https://c-4.1fichier.com/p947107115
50 23.01.2024 18:05:27 DEBUG HOSTER OneFichierCom[240]: LOAD URL https://c-4.1fichier.com/p947107115 | redirect=False | cookies=True | get={} | req=None | decode=True | multipart=False | post={} | ref=True | just_header=True
51 23.01.2024 18:05:28 INFO HOSTER OneFichierCom[240]: Direct download link detected
52 23.01.2024 18:05:28 INFO HOSTER OneFichierCom[240]: Downloading file...
53 23.01.2024 18:05:28 DEBUG HOSTER OneFichierCom[240]: DOWNLOAD URL https://1fichier.com/?x1pkrjifk6w5gezkcop0 | cookies=True | resume=None | get={} | disposition=True | chunks=None | post={} | ref=True
54 23.01.2024 18:05:29 DEBUG Content-Disposition: Percy.Jackson.and.the.Olympians.S01E06.FRENCH.1080p.WEB.H264-FW-Wawacity.boats.mkv
55 23.01.2024 18:05:29 DEBUG Chunked with range 0-
56 23.01.2024 18:12:05 INFO HOSTER OneFichierCom[240]: File saved
57 23.01.2024 18:12:05 INFO HOSTER OneFichierCom[240]: Checking download...
58 23.01.2024 18:12:05 INFO HOSTER OneFichierCom[240]: File is OK
59 23.01.2024 18:12:05 INFO HOSTER OneFichierCom[240]: Checking file (with built-in rules)...
60 23.01.2024 18:12:05 INFO HOSTER OneFichierCom[240]: Checking file (with custom rules)...
61 23.01.2024 18:12:05 INFO HOSTER OneFichierCom[240]: Checking for link errors...
62 23.01.2024 18:12:05 INFO HOSTER OneFichierCom[240]: No errors found
63 23.01.2024 18:12:05 INFO HOSTER OneFichierCom[240]: No errors found
64 23.01.2024 18:12:05 DEBUG ADDON ExternalScripts: No script found under folder download_processed
65 23.01.2024 18:12:05 DEBUG ADDON ExternalScripts: No script found under folder package_processed
66 23.01.2024 18:12:05 INFO Download finished: Percy.Jackson.and.the.Olympians.S01E06.FRENCH.1080p.WEB.H264-FW-Wawacity.boats.mkv
67 23.01.2024 18:12:05 DEBUG ADDON ExternalScripts: No script found under folder download_finished
68 23.01.2024 18:12:05 INFO Package finished: percy
69 23.01.2024 18:12:05 DEBUG ADDON ExternalScripts: No script found under folder package_finished

`

GammaC0de commented 7 months ago

but it takes a lot of time to combine different chunks

The process of chunks combination is just a matter of simple file copy: https://github.com/pyload/pyload/blob/ed850e886d2930e27fec44eab45da8c23775aaf3/src/pyload/core/network/http/http_download.py#L88-L104

What is the output of http://<pyload_ip>/info?

Without having access to the seedbox I cannot do anything..

DrekoDev commented 7 months ago

but it takes a lot of time to combine different chunks

The process of chunks combination is just a matter of simple file copy:

https://github.com/pyload/pyload/blob/ed850e886d2930e27fec44eab45da8c23775aaf3/src/pyload/core/network/http/http_download.py#L88-L104

What is the output of http://<pyload_ip>/info?

Without having access to the seedbox I cannot do anything..

Here is the information tab:

Python:2.7.18 (default, May 3 2020, 20:31:45) [GCC 9.2.0]OS:posix linux2 Linux 7facc51a3a80 3.10.0-1160.102.1.el7.x86_64 #1 SMP Tue Oct 17 15:42:21 UTC 2023 x86_64Version de pyLoad:0.4.20Dossier d'Installation:/app/pyloadDossier de Configuration:/configDossier de Téléchargement:/downloadsEspace libre :152.20 GiBLangue:enWebinterface Theme:modern by [GammaC0de](https://github.com/GammaC0de) based on work of [Marco Fernandes](https://github.com/xunil75) and Dogan BagciPort de l'Interface Web8000Port de l'Interface à Distance7227

Would you need an access to the pyload on my seedbox ?

GammaC0de commented 7 months ago

Would you need an access to the pyload on my seedbox

To get access just to the pyload on the seedbox would not help because this is a system and OS issue.

You can read here https://unix.stackexchange.com/questions/79751/is-there-a-faster-alternative-to-cp-for-copying-large-files-20-gb for possible reasons why this is happening.