dipu-bd / lightnovel-crawler

Generate and download e-books from online sources.
https://pypi.org/project/lightnovel-crawler/
GNU General Public License v3.0
1.5k stars 293 forks source link

Support on lnmtl and mtlnovel raw text #671

Closed ghost closed 3 years ago

ghost commented 3 years ago

Both lnmtl and mtlnovel have toggle-able access to the raw, with lnmtl giving the option to also disable the English translation. mtlnovel requires log-in to view the raw, and since lncrawler doesn't currently have log-in support for mtlnovel, lnmtl should probably be prioritized.

Essentially requesting the ability to toggle the raw and english on or off for when you parse lnmtl novels, which would give 3 options

  1. Crawling with only the translation
  2. Crawling with translation + raw
  3. Crawling with only raw
dipu-bd commented 3 years ago

Note: For this, we can add additional crawler to crawl raw only. Currently, only one to one relation is possible between a source link and crawler file. We can enable one to many relation by taking user input on which crawler to use for scraping.

dipu-bd commented 3 years ago

it seems this will take too much work. all the bots needs to be updated for this change. won't work on this for now

Galunid commented 3 years ago

Wouldn't it be possible to use ruby tags for raw? Toggling can be done by adding ruby{ display: none;} to stylesheets. This would basically solve points 1 and 2

dipu-bd commented 3 years ago

Currently the app only has one to one map from an URL to a crawler. For implementing support for both translation + raw, we have to use the same URL for two different crawlers. This is going to be an issue, since the entire architecture needs to be changed for console app, discord and telegram bots. In terminal, we would have to show a choice to select which crawler to use, on discord/telegram bots a question needs to be added to ask whether to download translation/raw/both.

Since this requires changing current architecture in many different ways, I have put this away for now.