search

Xonshiz / comic-dl

Comic-dl is a command line tool to download manga and comics from various comic and manga sites. Supported sites : readcomiconline.to, mangafox.me, comic naver and many more.
https://github.com/Xonshiz/comic-dl
MIT License
538 stars 68 forks source link

Can't dowload from readcomiconline.to #252

Closed mateus2k2 closed 4 years ago

mateus2k2 commented 4 years ago

I used both the windows binary and the Python comand

comic_dl.exe -i main.py -i

And I get stuck on the "Fooling CloudFlare...Please Wait..." Other sites still work like comicextra.com

Saw other theds talking about that but didin't uderestand much.

Gist Error Log = https://gist.github.com/mateus2k2/725f0a042df6eaf425336e42027aba61

Erro Comic DL

dsanchezseco commented 4 years ago

It is a problem with cfscrape not being able to resolve the challenges of cloudfare. We have to wait

thejoefletch commented 4 years ago

Has the issue with cfscrape been resolved?

StrangeDOS commented 4 years ago

Hi dsanchezseco,

Been gone for a while. But it looks like this is still a problem. Can you verify?

dsanchezseco commented 4 years ago

@StrangeDOS @thejoefletch @mateus2k2 Working with the latest, the cloudfare scrapper was changed and updated. Maybe the win binary must be recreated

TL;DR be sure to update the pips with git pull && pip remove -r requirements-txt && pip install -r requirements.txt --user

ghost commented 4 years ago

Will the win binary on this site be updated for download? The current one is dated May 26, 2019. Thanks.

StrangeDOS commented 4 years ago

Same question. I'm using the Windows binary and that's where I'm running into the problem. I can still use it on other sites.

Dissentrix commented 4 years ago

Still fooling cloudflare right now.

knexfour commented 4 years ago

Also have been patiently waiting for an updated Windows binary :) Would be greatly appreciated!

StrangeDOS commented 4 years ago

Any word on an updated Windows binary? :)

BobaFett218 commented 4 years ago

His older program for this site still works: https://github.com/Xonshiz/ReadComicOnline-Downloader

Aemilian666 commented 4 years ago

His older program for this site still works: https://github.com/Xonshiz/ReadComicOnline-Downloader

I can confirm this. And honestly? It's better than this one.

knexfour commented 4 years ago

The fact that it works currently makes it way better, but I love this version's auto download feature that keeps up with latest issues released of multiple comics all from one config file. Don't think the older version has something that robust (but please let me know if it does!)

Aemilian666 commented 4 years ago

The fact that it works currently makes it way better, but I love this version's auto download feature that keeps up with latest issues released of multiple comics all from one config file. Don't think the older version has something that robust (but please let me know if it does!)

It doesn't, but it is good for downloading 'Completed' issues. :)

Aemilian666 commented 4 years ago

Spoke too soon. Enter The URL of Series Issue : https://readcomiconline.to/Comic/Injustice-Gods-Among-Us-I Enter The Range : none Bypassing the check. Wait for a few seconds please. Message: Screenshot: available via screen

Quality To Download : HQ Order To Download : OLD Message: Screenshot: available via screen

Series Name : Injustice Gods Among Us I - 1

#####################################

#####################################

Quality To Download : HQ Order To Download : OLD Message: Screenshot: available via screen

Series Name : Injustice Gods Among Us I - 1

#####################################

#####################################

Are you fucking serious? It was working just fine before, but then it started to fail. First it wouldn't even download from a single link, now it doesn't download shit.

Damn. How horrible. Comic-DL doesn't download from ReadComicOnline, because whomever wrote the Scraper script refuses to give a damn decent tutorial on how to actually get that thing to run.

FML.

Aemilian666 commented 4 years ago

Quality To Download : HQ Order To Download : OLD Traceback (most recent call last): File "readcomic.py", line 447, in Url_Fetching() File "readcomic.py", line 126, in Url_Fetching Whole_Series(driver,url,Quality) File "readcomic.py", line 400, in Whole_Series Single_Issue(url,Quality) File "readcomic.py", line 157, in Single_Issue browser = webdriver.PhantomJS(service_args=['--load-images=no']) File "site-packages\selenium\webdriver\phantomjs\webdriver.py", line 52, in init File "site-packages\selenium\webdriver\common\service.py", line 87, in start File "site-packages\selenium\webdriver\common\service.py", line 103, in is_connectable File "site-packages\selenium\webdriver\common\utils.py", line 106, in is_connectable File "socket.py", line 566, in create_connection KeyboardInterrupt Failed to execute script readcomic

dsanchezseco commented 4 years ago

Try to use it with the python scripts instead of with the windows exe. With the python files you can run pip install --upgrade --force-reinstall -r requirements.txt and get the latest scrapper(which is keep up to date with cloudfare regularly, actually the scrapper was changed to a new one as the previous was lagging behind with the changes).

The exe is built with the scrapper package inside, so when a new scrapper version appears the exe needs to be updated(no idea how, i don't speak windows except for gaming).

BTW, just tested and it works

Aemilian666 commented 4 years ago

Sure, sounds good, once I figure out how to actually USE THE SCRAPER AS I SAID ABOVE. lol.

Aemilian666 commented 4 years ago

Also, I am talking about ReadComicOnline Downloader, no Comic-DL which I've given up on for the moment. CFScraper does no have a proper readme so I have no idea how o use it.

dsanchezseco commented 4 years ago

I'm talking about this one(comic-dl). Why do you need to know CFScrapper if it is already managed?

Aemilian666 commented 4 years ago

What are you talking about 'If i's already managed'? I just tried using Comic-DL an hour ago and it didn't even let me. It said to that there was a bug in the Cloudflare.

I'll keep trying ReadCOmicOnline. Thanks for the help.

dsanchezseco commented 4 years ago

Did you tried with the python files or the exe? Have you updated the python requirements? Sometimes when downloading a lot of issues you need to manually access one issue to refresh whatever CF does in the background to prevent bots.

I tried earlier this morning doing a clean pull from master and pip install -r requirements.txt and it worked without problem with python comic_dl/__main__.py -i https://readcomiconline.to/Comic/Injustice-Gods-Among-Us-I.

Aemilian666 commented 4 years ago

I do appreciate your advice and help, but I think it's safe to say that I don't know anything about how to use Python, it took me all day yesterday to figure out how to install using pip.

That's what's so frustrating. I did figure out how to install CFScraper via Pip, but I have zero idea on how o use it in collaboration with Comic-DL. Simply install/updating CFscraper did not seem to solve the problem, despite receiving an 'updated' confirmation.

I really enjoy this program, I do, but I feel that perhaps I began to use it at the wrong time, when CloudFlare was rampant, as it wasn't always.

I launched COmic-DL with Node.js and it worked fine, though there were some 404 errors. However, ReadComicONline has better offerings, so I tried to go that route.

Also, opening 'Readcomic.py' via Command Prompt only opens it as a notepad document.

Aemilian666 commented 4 years ago

C:\Windows\comic-dl-master>python comic_dl/main.py -i https://readcomiconline.to/Comic/Injustice-Gods-Among-Us-I Traceback (most recent call last): File "comic_dl/main.py", line 4, in from comic_dl import ComicDL File "C:\Windows\comic-dl-master\comic_dl\comic_dl.py", line 4, in from tqdm import tqdm ModuleNotFoundError: No module named 'tqdm'

Aemilian666 commented 4 years ago

Also, according to the original readme, it says that Python isn't required, did that change? Lol.

dsanchezseco commented 4 years ago

yeah, you almost got it, you are missing the rest of the requirements. Run pip install --upgrade --force-reinstall -r requirements.txt.

Python is not required if using the exe.

Aemilian666 commented 4 years ago

Wait, I think we're close! I didn't have the binary or Phantom in the folder. I ran a terminal in the repository folder, ran that bit of code you generously provided, and have a long string including:

"Collecting beautifulsoup4 Downloading beautifulsoup4-4.9.1-py3-none-any.whl (115 kB) |████████████████████████████████| 115 kB ... Requirement already satisfied: chardet<4,>=3.0.2 in c:\python\lib\site-packages (from requests->-r requirements.txt (line 3)) (3.0.4) Requirement already satisfied: certifi>=2017.4.17 in c:\python\lib\site-packages (from requests->-r requirements.txt (line 3)) (2020.4.5.2) Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in c:\python\lib\site-packages (from requests->-r requirements.txt (line 3)) (1.25.9) Requirement already satisfied: idna<3,>=2.5 in c:\python\lib\site-packages (from requests->-r requirements.txt (line 3)) (2.9) Collecting args Downloading args-0.1.0.tar.gz (3.0 kB) Collecting Pillow Downloading Pillow-7.1.2-cp38-cp38-win_amd64.whl (2.0 MB)"

Is this what I am supposed to see?

Aemilian666 commented 4 years ago

C:\Windows\comic-dl-master>python comic_dl/main.py -i https://readcomiconline.to/Comic/Injustice-Gods-Among-Us-I C:\Windows\comic-dl-master\comic_dl\sites\mangaHere.py:23: SyntaxWarning: "is" with a literal. Did you mean "=="? if len(url_split) is 5: C:\Windows\comic-dl-master\comic_dl\sites\rawSenManga.py:58: SyntaxWarning: "is" with a literal. Did you mean "=="? if x is 0: C:\Windows\comic-dl-master\comic_dl\sites\mangaFox.py:23: SyntaxWarning: "is" with a literal. Did you mean "=="? if len(url_split) is 5: Fooling CloudFlare...Please Wait... Fooling CloudFlare...Please Wait...

YAY!!! You're the best, man! Thank you so, so, so, so, so, so, so much!!!! Unfortunately I have to get to work, so I cannot enjoy it, but thank you so much! You're the best!

dsanchezseco commented 4 years ago

yup, now to get comics

dsanchezseco commented 4 years ago

you're are welcome!

Aemilian666 commented 4 years ago

Sorry to re-open this, but with this scraper, do we still need to change IPs to download from ReadComicOnline? After about 70 or so, it stops downloading, yet it doesn't actually stop the program, just downloads and converts 0kb pdfs.

dsanchezseco commented 4 years ago

"Sometimes when downloading a lot of issues you need to manually access one issue to refresh whatever CF does in the background to prevent bots." From above. Changing IPs should work to I guess, but you need to restart the program

Aemilian666 commented 4 years ago

So when downloading via the program and it starts getting 0kb files, I have to go to the website, "https://readcomiconline.to/Comic/Nation-X-X-Factor", for example? Then it will let me redownload the files if I run python comic_dl/main.py -a?

If that were the case, wouldn't it be working by now off of a VPN? I must have done a lot of things manually (to retrieve new links, you understand) then try it, yet it still downloads those small files, until I hop VPN's. Or does it have to be the SPECIFIC issue I'm trying to download that is giving 0kb files?

Aemilian666 commented 4 years ago

Because off of the VPN, I manually accessed "https://readcomiconline.to/Comic/Nation-X-X-Factor/Full?id=69160"

and this is what I got:

D:\Comics\comic-dl-master>python comic_dl/main.py -i https://readcomiconline.to/Comic/Nation-X-X-Factor Fooling CloudFlare...Please Wait... Fooling CloudFlare...Please Wait... [Comic-dl] Done : Nation X X Factor [Full] : : 0image(s) [00:00, ?image(s)/s] Total Time Taken To Complete : 24.277180671691895

D:\Comics\comic-dl-master>python comic_dl/main.py -i https://readcomiconline.to/Comic/Nation-X-X-Factor/Full?id=69160 Fooling CloudFlare...Please Wait... [Comic-dl] Done : Nation X X Factor [Full] : : 0image(s) [00:00, ?image(s)/s] Total Time Taken To Complete : 11.999600648880005

I do not like ComicExtra, because sometimes they upload titles of chapters that aren't even uploaded.

dsanchezseco commented 4 years ago

Yes, you have to enter an issue in the browser like if you were going to read it online. I hadn't tried what you suggest, so I do not know if it could work.

Aemilian666 commented 4 years ago

Then maybe I should have CFScraper. I think I might have removed it for some reason, lol. Could that be a factor? Since the readme was a little confusing for that.

Since I'm not sure how something as simple as that would be a fix. :P

Aemilian666 commented 4 years ago

Okay, so background information:

Do not seem to have CFScraper in the Windows directory, do I have to re-download that, despite not knowing how to use it?

Secondly, using my 'blacklisted' IP address, I tried to download Injustice, after manually accessing Issue 1 from the web browser. This was the result.

"D:\Comics\comic-dl-master>python comic_dl/main.py -i https://readcomiconline.to/Comic/Injustice-Gods-Among-Us-I Fooling CloudFlare...Please Wait... Fooling CloudFlare...Please Wait... [Comic-dl] Done : Injustice Gods Among Us I [Issue - 36] : : 0image(s) [00:00, ?image(s)/s] Fooling CloudFlare...Please Wait... [Comic-dl] Done : Injustice Gods Among Us I [Issue - 35] : : 0image(s) [00:00, ?image(s)/s] Fooling CloudFlare...Please Wait..."

As you can see, despite having accessed the issue via web browser, then attempting to redownload this link, the same thing happens.

Is this because the IP was already banned? (Not blacklisted. Banned. This seems permanent)

Aemilian666 commented 4 years ago

"C:\Users>pip install -U cfscrape Requirement already up-to-date: cfscrape in c:\python\lib\site-packages (2.1.1) Requirement already satisfied, skipping upgrade: requests>=2.6.1 in c:\python\lib\site-packages (from cfscrape) (2.23.0) Requirement already satisfied, skipping upgrade: chardet<4,>=3.0.2 in c:\python\lib\site-packages (from requests>=2.6.1->cfscrape) (3.0.4) Requirement already satisfied, skipping upgrade: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in c:\python\lib\site-packages (from requests>=2.6.1->cfscrape) (1.25.9) Requirement already satisfied, skipping upgrade: idna<3,>=2.5 in c:\python\lib\site-packages (from requests>=2.6.1->cfscrape) (2.9) Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in c:\python\lib\site-packages (from requests>=2.6.1->cfscrape) (2020.4.5.2)"

From the directions on Anorov's repository.

Aemilian666 commented 4 years ago

This is without using a VPN.

I accessed Harrow County Issue 25 via the browser.

Then I attempted to download it via the code.

As you can see, nothing. @dsanchezseco notice

dsanchezseco commented 4 years ago

Then I do not know, it's been a while since the last time I used it or looked at the code. Anyway that's a issue of the scrapper, so try looking for help in its repo

Aemilian666 commented 4 years ago

So then I was doing it correctly. Damn.

Yes, I just submitted a 'bug' report. Thanks for trying!

Aemilian666 commented 4 years ago

@dsanchezseco Is Anorov's the only CFScraper? He doesn't respond to issues.