dipu-bd / lightnovel-crawler

Generate and download e-books from online sources.
https://pypi.org/project/lightnovel-crawler/
GNU General Public License v3.0
1.36k stars 267 forks source link

[template] WordPress MangaStream (WP MangaStream) #1644

Open idMysteries opened 1 year ago

idMysteries commented 1 year ago

https://github.com/dipu-bd/lightnovel-crawler/issues/1643

eplister (eplisterfull) image https://themesia.com/mangastream-wordpress-theme/

Our crawlers: https://github.com/dipu-bd/lightnovel-crawler/blob/2d631a85cd214775b629f8864e3469996afb8efa/sources/en/n/novelsemperor.py https://github.com/dipu-bd/lightnovel-crawler/blob/f772831b61192828184ec692d50d45a774e57d08/sources/en/t/tamagotl.py https://github.com/dipu-bd/lightnovel-crawler/blob/b40a99846a6af0666cb81d8aa1af1567de2a7425/sources/en/m/mltnovels.py https://github.com/dipu-bd/lightnovel-crawler/blob/b30a47fd2832f618c462274e302663124c4c23cc/sources/pt/centralnovel.py

idMysteries commented 1 year ago

https://github.com/manga-download/hakuneko/blob/master/src/web/mjs/connectors/templates/WordPressMangastream.mjs

this.queryChapters = 'div#chapterlist ul li div.eph-num a';

https://github.com/manga-download/haruneko/blob/322a1b4c71925649b4aa8bce6656e66ea96be031/src/engine/websites/KolNovel.ts

@MangaStream.MangasSinglePageCSS('div#content div.soralist ul li a.series', '/series/list-mode/') @MangaStream.ChaptersSinglePageCSS('div.eplister ul li a', 'div:not(.epl-date)')

some sites use eplister (and eplisterfull), some #chapterlist

idMysteries commented 1 year ago

https://imperfectcomic.org/manga/it-starts-with-a-kingpin-account/ div#chapterlist ul li div.eph-num a

idMysteries commented 1 year ago

There is no WP MangaStream code on github. Maybe I'll find it on the Internet to find out what principle it works on. Why do some use eplister and others use chapterlist

idMysteries commented 1 year ago

Hmmm...

<domain>/series/<slug> -> eplister
<domain>/manga/<slup> -> #chapterlist ul li .eph-num a

UPD: not for all sources.

idMysteries commented 1 year ago

https://github.com/dipu-bd/lightnovel-crawler/issues/659 https://github.com/dipu-bd/lightnovel-crawler/issues/1539 https://github.com/dipu-bd/lightnovel-crawler/issues/1676 => #chapterlist ul li .eph-num a

dipu-bd commented 1 year ago

After changes in #501, now we can test a novel url against templates. Having more templates will be very useful.

idMysteries commented 1 year ago

Can this test be run in github actions?

dipu-bd commented 1 year ago

No, it is not actually a test. It is for matching the templates against the novel url and auto-generating a source file. for it

dipu-bd commented 1 year ago

Since you have already done some research on this template, can you make a PR creating one?

idMysteries commented 1 year ago

OK... I'll try

idMysteries commented 1 year ago

@dipu-bd I have a question about the Chapter id. Should it be an order from 1 to N or can it be a random order, for example from 342 to 101. Will the result then be sorted by id? image image image

I noticed that some chapters are sorted in descending order and some in ascending order. And they have an id that seems to be correct.

idMysteries commented 1 year ago

Or should I just write a small order detector? Like... "if next_id < prev_id then reverse else no"

dipu-bd commented 1 year ago

~Chapters will not be sorted. Please assign 1 to N for now. I will add some logic for checking it later~

Edit: I added the logic to sort volumes and chapters.

idMysteries commented 1 year ago

https://github.com/dipu-bd/lightnovel-crawler/issues/1687

dipu-bd commented 1 year ago

@idMysteries Are you working on this eplister template?

idMysteries commented 1 year ago

@idMysteries Are you working on this eplister template?

Yes, I haven't had time in the last couple of days. But now I will do it according to the new templates.