search

mikf / gallery-dl

Command-line program to download image galleries and collections from several image hosting sites
GNU General Public License v2.0
11.7k stars 953 forks source link

reactor.cc error #148

Closed wankio closed 5 years ago

wankio commented 5 years ago

sorry for late reply, last topic closed so i just create new topic

1- i think only nsfw pornreactor supported multiple tags search eg : http://pornreactor.cc/search/muscle+aka6 2- error when downloading

rllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/6 HTTP/1.1" 302 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/221 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/220 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/219 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/218 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/217 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/216 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/215 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/214 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/213 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/212 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/211 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/210 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/209 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/208 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/207 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/206 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/205 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/204 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/203 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/202 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/201 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/200 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/199 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/198 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/197 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/196 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/195 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/194 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/193 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/192 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/191 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/190 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/189 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/188 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/187 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/186 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/185 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/184 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/183 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/182 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/181 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/180 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/179 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/178 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/177 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/176 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/175 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/174 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/173 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/172 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/171 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/170 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/169 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/168 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/167 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/166 HTTP/1.1" 200 None urllib3.connectionpool: http://pornreactor.cc:80 "GET /tag/Dota/165 HTTP/1.1" 200 None pornreactor: An unexpected error occurred: JSONDecodeError - Invalid control character at: line 1 column 1234 (char 1233). Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues . pornreactor: Traceback (most recent call last): File "c:\users\gen32uc\appdata\local\programs\python\python37\lib\site-packages\gallery_dl\extractor\joyreactor.py", line 72, in _parse_post data = json.loads(script) File "c:\users\gen32uc\appdata\local\programs\python\python37\lib\json__init__.py", line 348, in loads return _default_decoder.decode(s) File "c:\users\gen32uc\appdata\local\programs\python\python37\lib\json\decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "c:\users\gen32uc\appdata\local\programs\python\python37\lib\json\decoder.py", line 353, in raw_decode obj, end = self.scan_once(s, idx) json.decoder.JSONDecodeError: Invalid control character at: line 30 column 22 (char 1287)

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "c:\users\gen32uc\appdata\local\programs\python\python37\lib\site-packages\gallery_dl\job.py", line 51, in run for msg in self.extractor: File "c:\users\gen32uc\appdata\local\programs\python\python37\lib\site-packages\gallery_dl\extractor\joyreactor.py", line 37, in items for image in self._parse_post(post): File "c:\users\gen32uc\appdata\local\programs\python\python37\lib\site-packages\gallery_dl\extractor\joyreactor.py", line 77, in _parse_post .replace("\r", "")) File "c:\users\gen32uc\appdata\local\programs\python\python37\lib\json__init__.py", line 348, in loads return _default_decoder.decode(s) File "c:\users\gen32uc\appdata\local\programs\python\python37\lib\json\decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "c:\users\gen32uc\appdata\local\programs\python\python37\lib\json\decoder.py", line 353, in raw_decode obj, end = self.scan_once(s, idx) json.decoder.JSONDecodeError: Invalid control character at: line 1 column 1234 (char 1233)

mikf commented 5 years ago

  1. multi tag searches are now supported (https://github.com/mikf/gallery-dl/commit/a36f52a7307bc635427f3e6d3587e42f21160fb9). I only looked at the results of http://joyreactor.com/search/... when writing the first extractors and, as noted in the other issue, it only returns tag suggestions as results - no posts. I therefore never really looked at the search results on all the other domains until today ...

  2. The JSON data of a post sometimes contains invalid characters (*) and I already had some code in place that was supposed to fix that, but apparently that wasn't enough. With https://github.com/mikf/gallery-dl/commit/8753627ef4cd7d910b2198a9791b4a4fe218d7d8 I improved upon that and posts that are still unable to be parsed are now just ignored instead of "crashing" the whole program.

There seams to be another issue with paginating over the search results of http://joyreactor.cc/search/..., but my computer appears to be currently blocked from accessing any of the ...reactor sites, so that will have to wait.

(*) when visiting the same post several times, the issue usually "fixes" itself, which makes testing not so easy

wankio commented 5 years ago

reactor domain seem like have ddos protection like sankakucomplex, when u run it too fast it will prevent you from access it in xx minutes. maybe set delay between each request can fix this problem

wankio commented 5 years ago

and it have subdomain eg : http://dota.reactor.cc/tag/Dota+Art it would be nice if it support *.reactor.cc too because i think it have many subdomain

mikf commented 5 years ago

Subdomains are now supported, there are wait-min and wait-max options to wait between HTTP requests, and another minor issue has been fixed (https://github.com/mikf/gallery-dl/commit/1734a6c87966dba54a7b13ff069fe8cb85a28a62). I hope that covers everything.