dipu-bd / lightnovel-crawler

Generate and download e-books from online sources.
https://pypi.org/project/lightnovel-crawler/
GNU General Public License v3.0
1.46k stars 288 forks source link

Please add this source: https://goblinslate.com/ #2076

Open Nariod14 opened 1 year ago

Nariod14 commented 1 year ago
zGadli commented 1 year ago

This source doesn't seem to have a lot of novels. It seems to be a website created by and for the author.

zGadli commented 1 year ago

Also, I don't think this source has cloudfare protection.

Nariod14 commented 1 year ago

You're right this is a translator site for a few novels that I'd really like to read offline. The way I checked for cloudflare is through this site: https://checkforcloudflare.selesti.com It said it had it, so I said yes. I've thus far managed to get the novel to epub using Web2Epub but that was a very time consuming process, so I suggested it here hoping you could use whatever web scraping magic you use to add this one too.

The consideration is very much appreciated, this tool is awesome and your work on it is very valued.

kuwoyuki commented 1 year ago

Also, I don't think this source has cloudfare protection.

$ curl -v goblinslate.com
*   Trying 104.21.92.246:80...
* Connected to goblinslate.com (104.21.92.246) port 80 (#0)
...
< Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=DtRSCf1fvTUN0cuuiBv%2BYoConyo%2BFuc2BDICv7JknGLZqGtTLFrThc5qipp9o4uD%2FCfVhrahtWx3oYvGmIrKfTBPjOV%2Fl9SkNVNF0ppFMgnWzDCqGtTZKx8LmeI7eDJP7cs%3D"}],"group":"cf-nel","max_age":604800}
< NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< Server: cloudflare
< CF-RAY: 802d804a9e94382b-FRA

and

NetRange:       104.16.0.0 - 104.31.255.255
CIDR:           104.16.0.0/12
NetName:        CLOUDFLARENET
NetHandle:      NET-104-16-0-0-1
Parent:         NET104 (NET-104-0-0-0-0)
NetType:        Direct Allocation
OriginAS:       AS13335
Organization:   Cloudflare, Inc. (CLOUD14)
zGadli commented 1 year ago

@kuwoyuki I see. Thanks for the info.