search

pyload / pyload

The free and open-source Download Manager written in pure Python
https://pyload.net
Other
3.32k stars 713 forks source link

Downloads either "fake" completes or fails w/ both messages HTTPRequest instance has no attribute 'cj' and 'NoneType' object has no attribute 'package' #2265

Closed darkpella closed 8 years ago

darkpella commented 8 years ago

Hi, I'm getting weird failures with all easybytez premium downloads.

They either

  1. "fake" complete: The download seems to complete in a second, but the resulting file is very small and, once changed extension to .html and opened in a web browser it shows to be a http/404 error page
  2. fail altogether with 2 different error messages each: "HTTPRequest instance has no attribute 'cj' ", "'NoneType' object has no attribute 'package' ",

In either case, grabbing the original download URL form pyload log and entering it into a web browser leads to a successful download

I already had all files and folders in my userplugins folders replaced with those found under module/plugins from latest commit available 5888dd692eb91c299101864c4963685a1bf089e8.

Below you may find:

  1. pyLoad log for either case pyLoad log.txt
  2. HTTP headers produced by a web browser (Firefox) when entering URL from pyLoad log of "fake" completed download HTTP headers fake completed downoad.txt
  3. HTTP headers produced by a web browser (Firefox) when entering URL from pyLoad log of failed download HTTP headers failed download.txt
  4. "Fake" downloaded file (was originally .rar which is not supported by Github, rename to .html for viewing) Sopravvissuto.The Martian.2015.iTA.ENG.iTUNES.WEBDL.1080p.AVC-21K.part01.zip
  5. pyLoad Debug log debug_EasybytezCom_05-01-2016_23-10-07.zip

Thanks for assistance, guys.

darkpella

GammaC0de commented 8 years ago

@vuolter as for issue 2, I think I know what causing this: I noticed weird stuff that going on lately: There a 2+ processing threads (not download threads) for every download: if you put self.log_debug("Processing IDs: %s" % self.pyload.threadManager.processingIds()) in Base.py's preprocessing function you can see that also.

Watch the Processing IDs lines in the log:

02.01.2016 08:42:35 INFO      pyLoad is up and running
02.01.2016 08:42:35 DEBUG     ADDON UpdateManager: LOAD URL http://updatemanager.pyload.org | redirect=True | cookies=True | get={'v': '0.4.9'} | req=None | decode=True | multipart=False | post={} | ref=True | just_header=False
02.01.2016 08:42:37 INFO      ADDON ExternalScripts: Activated scripts in folder `download_finished`: s.sh
02.01.2016 08:42:37 INFO      ADDON ExternalScripts: Activated scripts in folder `package_deleted`: s.sh
02.01.2016 08:42:37 INFO      ADDON ExternalScripts: Activated scripts in folder `package_extracted`: s.sh
02.01.2016 08:42:41 INFO      ADDON ClickNLoad: Proxy listening on 0.0.0.0:9666
02.01.2016 08:42:49 INFO      Download starts: X-Men First Class 2011 MULTi VFF 1080p HDLight x265 HEVC.GHT-www.zone-telechargement.com.part3.rar (2)
02.01.2016 08:42:49 DEBUG     ADDON ExternalScripts: No script found under folder `download_preparing`
02.01.2016 08:42:49 DEBUG     ADDON UserAgentSwitcher: Use custom user-agent string `Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Firefox/41.0`
02.01.2016 08:42:50 DEBUG     HOSTER OneFichierCom[187]: Plugin version: 0.98
02.01.2016 08:42:50 DEBUG     HOSTER OneFichierCom[187]: Plugin status: testing
02.01.2016 08:42:50 WARNING   HOSTER OneFichierCom[187]: Plugin may be unstable
02.01.2016 08:42:50 DEBUG     Redirected import module.plugins.accounts.OneFichierCom -> userplugins.accounts.OneFichierCom
02.01.2016 08:42:50 DEBUG     Processing IDs: [187]
02.01.2016 08:42:50 INFO      HOSTER OneFichierCom[187]: Grabbing link info...
02.01.2016 08:42:51 INFO      Download starts: X-Men First Class 2011 MULTi VFF 1080p HDLight x265 HEVC.GHT-www.zone-telechargement.com.part3.rar (2)
02.01.2016 08:42:51 DEBUG     ADDON ExternalScripts: No script found under folder `download_preparing`
02.01.2016 08:42:51 DEBUG     ADDON UserAgentSwitcher: Use custom user-agent string `Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Firefox/41.0`
02.01.2016 08:42:51 DEBUG     HOSTER OneFichierCom[187]: Link info: {'status': 2, 'hash': {}, 'name': 'X-Men First Class 2011 MULTi VFF 1080p HDLight x265 HEVC.GHT-www.zone-telechargement.com.part3.rar', 'url': u'http://1fichier.com/?ar1rcablei', 'pattern': {'HOST': u'1fichier.com', 'U': u'MB', 'S': u'1000', 'N': u'X-Men First Class 2011 MULTi VFF 1080p HDLight x265 HEVC.GHT-www.zone-telechargement.com.part3.rar'}, 'size': 1048576000}
02.01.2016 08:42:51 DEBUG     HOSTER OneFichierCom[187]: Previous link info: {}
02.01.2016 08:42:51 INFO      HOSTER OneFichierCom[187]: Link name: X-Men First Class 2011 MULTi VFF 1080p HDLight x265 HEVC.GHT-www.zone-telechargement.com.part3.rar
02.01.2016 08:42:51 INFO      HOSTER OneFichierCom[187]: Link size: 1000.00 MiB (1048576000 bytes)
02.01.2016 08:42:51 INFO      HOSTER OneFichierCom[187]: Link status: online
02.01.2016 08:42:51 INFO      HOSTER OneFichierCom[187]: Processing url: http://1fichier.com/?ar1rcablei
02.01.2016 08:42:51 INFO      HOSTER OneFichierCom[187]: Looking for direct download link...
02.01.2016 08:42:51 DEBUG     HOSTER OneFichierCom[187]: Redirect #0 to: http://1fichier.com/?ar1rcablei
02.01.2016 08:42:51 DEBUG     HOSTER OneFichierCom[187]: LOAD URL http://1fichier.com/?ar1rcablei | redirect=True | cookies=True | get={} | req=None | decode=True | multipart=False | post={} | ref=True | just_header=True
02.01.2016 08:42:51 DEBUG     HOSTER OneFichierCom[187]: Redirect #1 to: https://1fichier.com/?ar1rcablei
02.01.2016 08:42:51 DEBUG     HOSTER OneFichierCom[187]: LOAD URL https://1fichier.com/?ar1rcablei | redirect=True | cookies=True | get={} | req=None | decode=True | multipart=False | post={} | ref=True | just_header=True
02.01.2016 08:42:52 DEBUG     HOSTER OneFichierCom[187]: Plugin version: 0.98
02.01.2016 08:42:52 DEBUG     HOSTER OneFichierCom[187]: Plugin status: testing
02.01.2016 08:42:52 WARNING   HOSTER OneFichierCom[187]: Plugin may be unstable
02.01.2016 08:42:50 DEBUG     Processing IDs: [187, 187]
02.01.2016 08:42:52 INFO      HOSTER OneFichierCom[187]: Grabbing link info...
02.01.2016 08:42:52 INFO      HOSTER OneFichierCom[187]: Direct download link not found
02.01.2016 08:42:52 DEBUG     HOSTER OneFichierCom[187]: LOAD URL http://1fichier.com/?ar1rcablei | redirect=True | cookies=[('1fichier.com', 'LG', 'en')] | get={} | req=None | decode=True | multipart=False | post={} | ref=False | just_header=False
02.01.2016 08:42:53 INFO      HOSTER OneFichierCom[187]: Checking for link errors...
02.01.2016 08:42:53 INFO      HOSTER OneFichierCom[187]: No errors found
02.01.2016 08:42:53 INFO      HOSTER OneFichierCom[187]: Processing as free download...
02.01.2016 08:42:53 DEBUG     HOSTER OneFichierCom[187]: handle_free
02.01.2016 08:42:53 DEBUG     HOSTER OneFichierCom[187]: LOAD URL https://1fichier.com/?ar1rcablei | redirect=True | cookies=True | get={} | req=None | decode=True | multipart=False | post={u'dl_no_ssl': 'on', u'dlinline': ''} | ref=True | just_header=False
02.01.2016 08:42:53 DEBUG     HOSTER OneFichierCom[187]: Link info: {'status': 2, 'hash': {}, 'name': 'X-Men First Class 2011 MULTi VFF 1080p HDLight x265 HEVC.GHT-www.zone-telechargement.com.part3.rar', 'url': u'http://1fichier.com/?ar1rcablei', 'pattern': {'HOST': u'1fichier.com', 'U': u'MB', 'S': u'1000', 'N': u'X-Men First Class 2011 MULTi VFF 1080p HDLight x265 HEVC.GHT-www.zone-telechargement.com.part3.rar'}, 'size': 1048576000}
02.01.2016 08:42:53 DEBUG     HOSTER OneFichierCom[187]: Previous link info: {'status': 2, 'hash': {}, 'name': 'X-Men First Class 2011 MULTi VFF 1080p HDLight x265 HEVC.GHT-www.zone-telechargement.com.part3.rar', 'url': u'http://1fichier.com/?ar1rcablei', 'pattern': {'HOST': u'1fichier.com', 'U': u'MB', 'S': u'1000', 'N': u'X-Men First Class 2011 MULTi VFF 1080p HDLight x265 HEVC.GHT-www.zone-telechargement.com.part3.rar'}, 'size': 1048576000}
02.01.2016 08:42:53 INFO      HOSTER OneFichierCom[187]: Link name: X-Men First Class 2011 MULTi VFF 1080p HDLight x265 HEVC.GHT-www.zone-telechargement.com.part3.rar
02.01.2016 08:42:53 INFO      HOSTER OneFichierCom[187]: Link size: 1000.00 MiB (1048576000 bytes)
02.01.2016 08:42:53 INFO      HOSTER OneFichierCom[187]: Link status: online
02.01.2016 08:42:53 INFO      HOSTER OneFichierCom[187]: Processing url: http://1fichier.com/?ar1rcablei
02.01.2016 08:42:53 INFO      HOSTER OneFichierCom[187]: Looking for direct download link...
02.01.2016 08:42:53 DEBUG     HOSTER OneFichierCom[187]: Redirect #0 to: http://1fichier.com/?ar1rcablei
02.01.2016 08:42:53 DEBUG     HOSTER OneFichierCom[187]: LOAD URL http://1fichier.com/?ar1rcablei | redirect=True | cookies=True | get={} | req=None | decode=True | multipart=False | post={} | ref=True | just_header=True
02.01.2016 08:42:53 DEBUG     pycurl exception 0: ('cannot invoke setopt() - perform() is currently running',)
02.01.2016 08:42:53 ERROR     pycurl error 0: ('cannot invoke setopt() - perform() is currently running',)
Traceback (most recent call last):
  File "/usr/share/pyload/module/PluginThread.py", line 191, in run
    pyfile.plugin.preprocessing(self)
  File "/home/user/.pyload-0.4.9/userplugins/internal/Base.py", line 297, in preprocessing
    return self._process(*args, **kwargs)
  File "/home/user/.pyload-0.4.9/userplugins/internal/Hoster.py", line 118, in _process
    self.process(self.pyfile)
  File "/home/user/.pyload-0.4.9/userplugins/internal/SimpleHoster.py", line 263, in process
    self.handle_direct(pyfile)
  File "/home/user/.pyload-0.4.9/userplugins/internal/SimpleHoster.py", line 442, in handle_direct
    self.link = pyfile.url if self.isresource(pyfile.url) else None
  File "/home/user/.pyload-0.4.9/userplugins/internal/Hoster.py", line 177, in isresource
    header = self.load(url, just_header=True)
  File "/home/user/.pyload-0.4.9/userplugins/internal/Base.py", line 504, in load
    return super(Base, self).load(*args, **kwargs)
  File "/home/user/.pyload-0.4.9/userplugins/internal/Plugin.py", line 204, in load
    html = req.load(url, get, post, ref, bool(cookies), just_header, multipart, decode is True)  #@TODO: Fix network multipart in 0.4.10
  File "/usr/share/pyload/module/network/Browser.py", line 97, in load
    return self.http.load(*args, **kwargs)
  File "/usr/share/pyload/module/network/HTTPRequest.py", line 187, in load
    self.setRequestContext(url, get, post, referer, cookies, multipart)
  File "/usr/share/pyload/module/network/HTTPRequest.py", line 156, in setRequestContext
    self.c.setopt(pycurl.URL, url)
error: cannot invoke setopt() - perform() is currently running
02.01.2016 08:42:53 INFO      Debug Report written to debug_OneFichierCom_02-01-2016_08-42-53.zip
adrienyhuel commented 8 years ago

:+1: I have that for all hosters I use (OneFichier, BasePlugin, UpToBoxCom), and I did not understood what was going on...

aronze06 commented 8 years ago

Hi all ! Did someone find a solution ? Thanks Alexis

darkpella commented 8 years ago

In @vuolter & @GammaC0de we trust ;) !

vuolter commented 8 years ago

I'll have a look at this tomorrow. ;)

vuolter commented 8 years ago

@GammaC0de Try disabling the XFileSharing addon.

GammaC0de commented 8 years ago

Unfortunately, did not helped..

adrienyhuel commented 8 years ago

@GammaC0de I tried too, and same results...

darkpella commented 8 years ago

@GammaC0de I tried it last night and i woke up this morning with lots of (real) completed downloads... downloads are running correctly right now as well. The only weird thing is that, after de-activating XFileSharing plugin and adding a new list of downloads, pyLoad would kick me to the login page and not let me past it until I rebooted the system. I'm pretty sure I entered the right password, since I also tried entering the wrong one and pyLoad would tell me, which did not happen when I entered the right one. Nonetheless, in the background it kept downloading.

darkpella commented 8 years ago

@GammaC0de for sake of completeness, after I de-activated XFileShring plugin, I tried to restart downloading those files I had problems downloading with and those would not work. I then removed those files form the download list and added a new list and, as I found out later, those would work.

GammaC0de commented 8 years ago

@darkpella ok, will try that

adrienyhuel commented 8 years ago

@darkpella Maybe you had luck.

My pyload is up to date with last stable.zip, XFileSharing Plugin disabled, and I restarted Pyload.

I just tried to add a package with two UpToBox links.

The two links download, but I see in the web interface, in the count of active downloads : "3/2/2".

For me, it means that two threads are launched for one of the downloads...

The first file finished OK. The second file is downloaded, the file is good on filesystem, but the status is KO with error "[Errno 2] No such file or directory: u'/HDD4/Pyload/RES106.rar.chunk0' 350.36 MiBUptoboxCom"

GammaC0de commented 8 years ago

@darkpella Tried it, did not helped either.

GammaC0de commented 8 years ago

Quick and dirty fix would be to add:

        if self.pyload.threadManager.processingIds().count(self.pyfile.id) > 1:
            return

to Base.py#L289

@vuolter should we add this lines or do you have a cleaner solution?

adrienyhuel commented 8 years ago

@GammaC0de Your fix doesn't fully work.

I modified Base.py and added a package with 7 links inside.

All the links are downloading good, I see only 7 processes (7/7/7 in web gui). BUT, the download finished script was executed for one of the links : 

07.01.2016 18:54:46 WARNING   HOSTER UptoboxCom[22]: Plugin may be unstable
07.01.2016 18:54:46 INFO      ACCOUNT UptoboxCom: Grabbing account info for user `youl_junior`...
07.01.2016 18:54:47 INFO      HOSTER UptoboxCom[22]: Grabbing link info...
07.01.2016 18:54:47 INFO      Download starts: http://uptobox.com/7oivxgufzdjb
07.01.2016 18:54:47 INFO      HOSTER UptoboxCom[22]: Link name: INVB 1080.part3.rar
07.01.2016 18:54:47 INFO      HOSTER UptoboxCom[22]: Link size: 950.00 MiB (996147200 bytes)
07.01.2016 18:54:47 INFO      HOSTER UptoboxCom[22]: Link status: en ligne
07.01.2016 18:54:47 INFO      HOSTER UptoboxCom[22]: Processing url: http://uptobox.com/7oivxgufzdjb
07.01.2016 18:54:47 INFO      HOSTER UptoboxCom[22]: Looking for direct download link...
07.01.2016 18:54:48 INFO      HOSTER UptoboxCom[22]: Direct download link not found
07.01.2016 18:54:48 INFO      HOSTER UptoboxCom[22]: Checking for link errors...
07.01.2016 18:54:48 INFO      HOSTER UptoboxCom[22]: No errors found
07.01.2016 18:54:48 INFO      HOSTER UptoboxCom[22]: Processing as premium download...
07.01.2016 18:54:48 INFO      HOSTER UptoboxCom[22]: Checking for link errors...
07.01.2016 18:54:48 INFO      HOSTER UptoboxCom[22]: No errors found
07.01.2016 18:54:48 INFO      Download finished: INVB 1080.part3.rar
07.01.2016 18:54:48 INFO      ADDON ExternalScripts: Executing scripts in folder `download_finished`...
07.01.2016 18:54:48 INFO      HOSTER UptoboxCom[22]: Downloading file...

It says "download finished" then "downloading file"...

GammaC0de commented 8 years ago

@youlJunior Ok, so it a partial fix but at least it works: All of this weird exceptions about "cj" or "perform() alread running" are gone Side effect is that ExeternalScripts makes false calls to download_finished scripts

Until we find a better solution

adrienyhuel commented 8 years ago

Yeah thanks for that small fix.

Maybe our Lord @vuolter will find a nice solution to protect us from the great Evil Python ;-)

valentino-90 commented 8 years ago

I have the same problem with Easybytez Premium Account. After i have disabled XFileSharing ADDON, my premium data are visible, but my downloaded files have size of 388bytes "or HTTPRequest instance has no attribute 'cj' " error. I don't understand the solution. Please help me.. thanks

GammaC0de commented 8 years ago

@valentix See my partial fix above

valentino-90 commented 8 years ago

I add the code to line 289 of Base.py and restart and re-add links to download (i have already disabled Checksum plugin), but now it download always fake file of 388bytes..

GammaC0de commented 8 years ago

But no HTTPRequest instance has no attribute 'cj' " error no more, correct?

adrienyhuel commented 8 years ago

@valentix : Are you sure you updated the good Base.py file (in the case you have one in pyload install folder, and one in userplugins folder) ? After adding code in Base.py and restarted, did you clean your download folder (fake_file, fake_file.chunks, fake_file.chunk0, ...) What is the data in the fake file ?

WARNING : The Base.py has been upgraded 1 hour ago, so if you have automatic update, you have to redo the manual modification of that file.

valentino-90 commented 8 years ago

@youlJunior Today clone repository with git, empty the user folder (i leave only old pyload.conf user), re add the user for web interface, remove all downloads in folder and re add premium account easybytez. The error is "Error calling event handler package_failed: >, (,), PyPackage instance has no attribute 'trunc'"

ascim commented 8 years ago

Dear all, I probably have the same issue with Share-Online hoster. Could you please advise on how to add your fix in the Base.py file? Below are the lines around line #289 of the actual Base.py file. Thank you!

287 #: Deprecated method, use _process instead (Remove in 0.4.10) 288 def preprocessing(self, _args, _kwargs): 289 time.sleep(1) #@NOTE: Recheck info thread synchronization in 0.4.10 290 return self._process(_args, _kwargs)

Should it be inserted before or after the the actual line (289 time.sleep(1) #@NOTE: Recheck info thread synchronization in 0.4.10)?

Because if I test both possibilities the download is not even transferred from "queue" to "active downloads".

GammaC0de commented 8 years ago

@valentix remember to re-add the lines mentioned above to `Base.py'

valentino-90 commented 8 years ago

@GammaC0de @ascim @youlJunior Mmmmm i try... but Should it be inserted before or after the the actual line (289 time.sleep(1) #@NOTE: Recheck info thread synchronization in 0.4.10)? The XFileSharing and XFileSharingFolder plugins must be disabled?

adrienyhuel commented 8 years ago

@valentix I tried before and after the line "time.sleep(1)" and both work. I disabled XFileSharing plugin only.

valentino-90 commented 8 years ago

@GammaC0de @youlJunior Now I have to wait like I'm not premium , but in the configuration it detects my premium data. LOG: "You have reached the download-limit: 512 Mb for last 1 days"

valentino-90 commented 8 years ago

@vuolter @GammaC0de I used all the fixes, now recognizes the size but puts me on hold...

LOG: HOSTER EasybytezCom[3]: Premium download failed INFO
Download restarted: DNIM.GFME.SPE6.FNW.CWK.part04.rar | Fallback to free processing

valentino-90 commented 8 years ago

News?

GammaC0de commented 8 years ago

@valentix Please post in #2283 as the currnet issue related to another problem, thanks.

thegounet commented 8 years ago

fix doesn't work for me... (#2288)

thegounet commented 8 years ago 
13.01.2016 21:44:40 INFO      Added package film containing 1 links
13.01.2016 21:44:40 INFO      Download starts: https://1fichier.com/?56du7k6jgy
13.01.2016 21:44:41 WARNING   HOSTER OneFichierCom[187]: Plugin may be unstable
13.01.2016 21:44:41 INFO      HOSTER OneFichierCom[187]: Grabbing link info...
13.01.2016 21:44:42 INFO      Download starts: https://1fichier.com/?56du7k6jgy
13.01.2016 21:44:42 INFO      Download finished: https://1fichier.com/?56du7k6jgy
13.01.2016 21:44:42 WARNING   Download failed: https://1fichier.com/?56du7k6jgy | No file downloaded
13.01.2016 21:44:42 INFO      ADDON UnSkipOnFail: Looking for skipped duplicates of: https://1fichier.com/?56du7k6jgy (pid:103)
13.01.2016 21:44:42 INFO      ADDON UnSkipOnFail: No duplicates found
13.01.2016 21:44:43 WARNING   Download failed: https://1fichier.com/?56du7k6jgy | 'NoneType' object has no attribute 'id'
13.01.2016 21:44:43 INFO      ADDON UnSkipOnFail: Looking for skipped duplicates of: https://1fichier.com/?56du7k6jgy (pid:103)
13.01.2016 21:44:43 INFO      ADDON UnSkipOnFail: No duplicates found
13.01.2016 21:44:43 ERROR     Error executing hooks: plugin
GammaC0de commented 8 years ago

Fake completes should be gone with https://github.com/pyload/pyload/commit/45dbb10747f606286934741d8ea4ab6e02618632

GammaC0de commented 8 years ago

https://github.com/pyload/pyload/commit/cad33cec0ca6d19c66beaef2746f30630ff98102