search

Difegue / LANraragi

Web application for archival and reading of manga/doujinshi. Lightweight and Docker-ready for NAS/servers.
https://lrr.tvc-16.science
MIT License
2.2k stars 154 forks source link

Chaika plugin returning incorrect results during metadata scrape #523

Closed pandafeliz closed 3 years ago

pandafeliz commented 3 years ago

LRR Version and OS Version 0.8.0, "Black Tie White Noise" OS: Windows 10

Bug Details After the last update, the Chaika plugin will sometimes return an incorrect entry while doing a manual metadata scrap

Matching Logs None.

Screenshots

chrome_wW3O7zhTox

will return this

chrome_UA2GvKeQoZ

instead of this which is the correct archive but using the url field

chrome_B2cjviyK4U

Difegue commented 3 years ago

Could you add some logs in debug mode to this issue?
I'm suspecting the search is less precise due to the recent changes but it'd be convenient to have repro data.

pandafeliz commented 3 years ago

Here is the log in an installation with only that one file since doing it on my main folder fills the logs with

[2021-10-13 11:06:34] [Tag Stats] [debug] Trimmed:


C o n v e r s i o n   i n   p r o g r e s s ,   t h i s   m a y   t a k e   a   f e w   m i n u t e s . . .

 T h e   d i s t r i b u t i o n   i s   a l r e a d y   t h e   r e q u e s t e d   v e r s i o n .

Executing the following command on WSL: export LRR_FORCE_DEBUG=1 && export LRR_NETWORK=http://*:3000 && export LRR_DATA_DIRECTORY='/mnt/c/test' && export LRR_THUMB_DIRECTORY='/mnt/c/test/cache' && cd /home/koyomi/lanraragi && rm -f public/temp/server.pid && mkdir -p log && mkdir -p content && mkdir -p database && sysctl vm.overcommit_memory=1 && redis-server /home/koyomi/lanraragi/tools/build/docker/redis.conf --dir '/mnt/c/test/' --daemonize yes && perl ./script/launcher.pl -f ./script/lanraragi
vm.overcommit_memory = 1
Server PID will be at /home/koyomi/lanraragi/public/temp/server.pid
Scalar value @ids[...] better written as $ids[...] at /home/koyomi/lanraragi/script/../lib/LANraragi/Model/Search.pm line 118.

ｷﾀ━━━━━━(ﾟ∀ﾟ)━━━━━━!!!!!
Enabling AOF on Redis... This might take a while.
[LANraragi] [info] LANraragi 0.8.0 (re-)started. (Debug Mode)
[LANraragi] [info] Plugin Detected: Chaika.moe
[LANraragi] [info] Plugin Detected: Tag Copier
[LANraragi] [info] Plugin Detected: Date Added
[LANraragi] [info] Plugin Detected: E-Hentai
[LANraragi] [info] Plugin Detected: eze
[LANraragi] [info] Plugin Detected: FAKKU
[LANraragi] [info] Plugin Detected: Hdoujin
[LANraragi] [info] Plugin Detected: koromo
[LANraragi] [info] Plugin Detected: Mayriad's EH Master Script
[LANraragi] [info] Plugin Detected: Filename Parsing
[LANraragi] [info] Plugin Detected: nHentai
[LANraragi] [info] Script Detected: Blacklist Migration
[LANraragi] [info] Script Detected: Subfolders to Categories
[LANraragi] [info] Script Detected: Source Finder
[LANraragi] [info] Script Detected: nHentai Source Converter
[LANraragi] [info] Downloader Detected: Chaika.moe Downloader
[LANraragi] [info] Downloader Detected: E*Hentai Downloader
Killing process 26 from /home/koyomi/lanraragi/public/temp/minion.pid
Minion will use the Redis database at 127.0.0.1:6379/1
[LANraragi] [info] Successfully connected to Minion database.
[LANraragi] [debug] Registered tasks with Minion.
[Minion] [info] Starting new Minion worker in subprocess with 4 parallel jobs.
[Minion] [info] Minion worker 26 started
Killing process 27 from /home/koyomi/lanraragi/public/temp/shinobu.pid
[LANraragi] [debug] Shinobu Worker new PID is 27
[LANraragi] [info] Routing done! Ready to receive requests.
[2021-10-13 11:33:54.38070] [26] [debug] Process 28 is performing job "1" with task "warm_cache"
[Mojolicious] Process 28 is performing job "1" with task "warm_cache"
[Minion] [info] Warming up search cache...
[Search Engine] [debug] Search request: --title-asc-0-0
[Search Engine] [debug] No cache available, doing a full DB parse.
[2021-10-13 11:33:54.40942] [14] [info] Listening at "http://*:3000"
[Minion] [info] Done!
[Mojolicious] Listening at "http://*:3000"
Web application available at http://127.0.0.1:3000
[2021-10-13 11:33:54.41376] [14] [info] Manager 14 started
[Mojolicious] Manager 14 started
[2021-10-13 11:33:54.43706] [26] [debug] Process 29 is performing job "2" with task "build_stat_hashes"
[Mojolicious] Process 29 is performing job "2" with task "build_stat_hashes"
[2021-10-13 11:33:54.44814] [30] [info] Worker 30 started
[Mojolicious] Worker 30 started
[Tag Stats] [debug] Building stat indexes...
[Tag Stats] [debug] Done!
[2021-10-13 11:33:54.48282] [31] [info] Worker 31 started
[Mojolicious] Worker 31 started
[2021-10-13 11:33:54.51602] [32] [info] Worker 32 started
[Mojolicious] Worker 32 started
[2021-10-13 11:33:54.53898] [14] [info] Creating process id file "/home/koyomi/lanraragi/public/temp/server.pid"
[2021-10-13 11:33:54.54312] [33] [info] Worker 33 started
[Mojolicious] Creating process id file "/home/koyomi/lanraragi/public/temp/server.pid"
[Mojolicious] Worker 33 started
Scalar value @ids[...] better written as $ids[...] at /home/koyomi/lanraragi/lib/../lib/LANraragi/Model/Search.pm line 118.
[Shinobu] [info] Shinobu File Watcher started.
[Shinobu] [info] Content folder is /mnt/c/test.
[Shinobu] [info] Scanning content folder for changes...
[Shinobu] [info] Found 1 new files.
[Shinobu] [info] 0 files were found on the filemap but not on the filesystem.
[Shinobu] [debug] Number of available cores for processing: 4
[Shinobu] [debug] Adding /mnt/c/test/[Mozu] Office Baby (COMIC Kairakuten 2014-07) [English].zip to Shinobu filemap.
[Shinobu] [debug] Computed ID is 9e2b1155849dc9a9f7080035caad167ae63bca82.
[Shinobu] [info] Adding new file /mnt/c/test/[Mozu] Office Baby (COMIC Kairakuten 2014-07) [English].zip with ID 9e2b1155849dc9a9f7080035caad167ae63bca82
[Archive] [debug] Pushing to redis on ID 9e2b1155849dc9a9f7080035caad167ae63bca82:
[Archive] [debug] File Name: [Mozu] Office Baby (COMIC Kairakuten 2014-07) [English]
[Archive] [debug] Filesystem Path: /mnt/c/test/[Mozu] Office Baby (COMIC Kairakuten 2014-07) [English].zip
[Auto-Plugin] [info] Executing enabled metadata plugins on archive with id 9e2b1155849dc9a9f7080035caad167ae63bca82.
[Shinobu] [info] Initial scan complete! Adding watcher to content folder to monitor for further file edits.
[Shinobu] [debug] Watcher class is File::ChangeNotify::Watcher::Inotify
[Shinobu] [info] All done! Now dutifully watching your files.
[2021-10-13 11:34:00.46156] [30] [trace] [p6vVNGrX8s-W] GET "/"
[Mojolicious] [p6vVNGrX8s-W]
[2021-10-13 11:34:00.46622] [30] [trace] [p6vVNGrX8s-W] Routing to controller "LANraragi::Controller::Index" and action "index"
[Mojolicious] [p6vVNGrX8s-W]
[2021-10-13 11:34:00.50700] [30] [debug] Rendering template "index.html.tt2"
[Mojolicious] Rendering template "index.html.tt2"
[2021-10-13 11:34:00.53618] [30] [debug] [p6vVNGrX8s-W] 200 OK (0.074652s, 13.396/s)
[Mojolicious] [p6vVNGrX8s-W]
[2021-10-13 11:34:00.72348] [30] [trace] [8rp1uSjdY25g] GET "/search"
[Mojolicious] [8rp1uSjdY25g]
[2021-10-13 11:34:00.72622] [30] [trace] [8rp1uSjdY25g] Routing to controller "LANraragi::Controller::Api::Search" and action "handle_datatables"
[Mojolicious] [8rp1uSjdY25g]
[Search Engine] [debug] Search request: --title-0--
[2021-10-13 11:34:00.75002] [33] [trace] [GtjtvT6WDI9-] GET "/search"
[Mojolicious] [GtjtvT6WDI9-]
[Search Engine] [debug] No cache available, doing a full DB parse.
[2021-10-13 11:34:00.78028] [32] [trace] [fG1HzNxKVEuF] GET "/api/categories"
[Mojolicious] [fG1HzNxKVEuF]
[2021-10-13 11:34:00.79066] [33] [trace] [GtjtvT6WDI9-] Routing to controller "LANraragi::Controller::Api::Search" and action "handle_datatables"
[2021-10-13 11:34:00.79076] [32] [trace] [fG1HzNxKVEuF] Routing to controller "LANraragi::Controller::Api::Category" and action "get_category_list"
[Mojolicious] [GtjtvT6WDI9-]
[Mojolicious] [fG1HzNxKVEuF]
[2021-10-13 11:34:00.79754] [32] [debug] [fG1HzNxKVEuF] 200 OK (0.017198s, 58.146/s)
[Mojolicious] [fG1HzNxKVEuF]
[2021-10-13 11:34:00.80856] [32] [trace] [i-4nk4RSRS8c] GET "/api/database/stats"
[Mojolicious] [i-4nk4RSRS8c]
[2021-10-13 11:34:00.81096] [32] [trace] [i-4nk4RSRS8c] Routing to controller "LANraragi::Controller::Api::Database" and action "serve_tag_stats"
[Mojolicious] [i-4nk4RSRS8c]
[Search Engine] [debug] Search request: --title-0--
[Tag Stats] [debug] Serving tag statistics with a minimum weight of 2
[Search Engine] [debug] No cache available, doing a full DB parse.
[2021-10-13 11:34:00.82068] [32] [debug] [i-4nk4RSRS8c] 200 OK (0.012318s, 81.182/s)
[Mojolicious] [i-4nk4RSRS8c]
[2021-10-13 11:34:00.83402] [32] [trace] [PBqFOwSL8A_T] GET "/api/categories"
[Mojolicious] [PBqFOwSL8A_T]
[2021-10-13 11:34:00.84258] [32] [trace] [PBqFOwSL8A_T] Routing to controller "LANraragi::Controller::Api::Category" and action "get_category_list"
[Mojolicious] [PBqFOwSL8A_T]
[2021-10-13 11:34:00.85674] [32] [debug] [PBqFOwSL8A_T] 200 OK (0.02224s, 44.964/s)
[Mojolicious] [PBqFOwSL8A_T]
[2021-10-13 11:34:01.86838] [30] [debug] [8rp1uSjdY25g] 200 OK (1.144849s, 0.874/s)
[Mojolicious] [8rp1uSjdY25g]
[2021-10-13 11:34:01.90912] [33] [debug] [GtjtvT6WDI9-] 200 OK (1.159162s, 0.862/s)
[Mojolicious] [GtjtvT6WDI9-]
[2021-10-13 11:34:01.94706] [33] [trace] [jJ0Jnci-q8JN] GET "/api/archives/9e2b1155849dc9a9f7080035caad167ae63bca82/thumbnail"
[Mojolicious] [jJ0Jnci-q8JN]
[2021-10-13 11:34:01.94990] [33] [trace] [jJ0Jnci-q8JN] Routing to controller "LANraragi::Controller::Api::Archive" and action "serve_thumbnail"
[Mojolicious] [jJ0Jnci-q8JN]
[2021-10-13 11:34:01.95352] [33] [debug] [jJ0Jnci-q8JN] 200 OK (0.006448s, 155.086/s)
[Mojolicious] [jJ0Jnci-q8JN]
[2021-10-13 11:34:03.24702] [33] [trace] [J2fMjJXFBGAD] GET "/api/archives/9e2b1155849dc9a9f7080035caad167ae63bca82/categories"
[Mojolicious] [J2fMjJXFBGAD]
[2021-10-13 11:34:03.24932] [33] [trace] [J2fMjJXFBGAD] Routing to controller "LANraragi::Controller::Api::Archive" and action "get_categories"
[Mojolicious] [J2fMjJXFBGAD]
[2021-10-13 11:34:03.25260] [33] [debug] [J2fMjJXFBGAD] 200 OK (0.005518s, 181.226/s)
[Mojolicious] [J2fMjJXFBGAD]
[2021-10-13 11:34:03.26372] [33] [trace] [Rjs12DNGC33r] GET "/css/vendor/font/context-menu-icons.woff2"
[Mojolicious] [Rjs12DNGC33r]
[2021-10-13 11:34:03.27096] [33] [debug] Template "not_found.development.html.tt2" not found
[Mojolicious] Template "not_found.development.html.tt2" not found
[2021-10-13 11:34:03.27214] [33] [debug] Template "not_found.html.tt2" not found
[Mojolicious] Template "not_found.html.tt2" not found
[2021-10-13 11:34:03.27476] [33] [trace] [Rjs12DNGC33r] Rendering template "mojo/debug.html.ep"
[Mojolicious] [Rjs12DNGC33r]
[2021-10-13 11:34:03.30080] [33] [debug] [Rjs12DNGC33r] 404 Not Found (0.037082s, 26.968/s)
[Mojolicious] [Rjs12DNGC33r]
[2021-10-13 11:34:03.30952] [33] [trace] [8KKHC8SVKSEU] GET "/css/vendor/font/context-menu-icons.woff"
[Mojolicious] [8KKHC8SVKSEU]
[2021-10-13 11:34:03.31134] [33] [debug] Template "not_found.development.html.tt2" not found
[Mojolicious] Template "not_found.development.html.tt2" not found
[2021-10-13 11:34:03.31210] [33] [debug] Template "not_found.html.tt2" not found
[Mojolicious] Template "not_found.html.tt2" not found
[2021-10-13 11:34:03.31264] [33] [trace] [8KKHC8SVKSEU] Rendering cached template "mojo/debug.html.ep"
[Mojolicious] [8KKHC8SVKSEU]
[2021-10-13 11:34:03.31628] [33] [debug] [8KKHC8SVKSEU] 404 Not Found (0.00675s, 148.148/s)
[Mojolicious] [8KKHC8SVKSEU]
[2021-10-13 11:34:03.32698] [33] [trace] [QVLmhWQ8nZwz] GET "/css/vendor/font/context-menu-icons.ttf"
[Mojolicious] [QVLmhWQ8nZwz]
[2021-10-13 11:34:03.32862] [33] [debug] Template "not_found.development.html.tt2" not found
[Mojolicious] Template "not_found.development.html.tt2" not found
[2021-10-13 11:34:03.32942] [33] [debug] Template "not_found.html.tt2" not found
[Mojolicious] Template "not_found.html.tt2" not found
[2021-10-13 11:34:03.32996] [33] [trace] [QVLmhWQ8nZwz] Rendering cached template "mojo/debug.html.ep"
[Mojolicious] [QVLmhWQ8nZwz]
[2021-10-13 11:34:03.33480] [33] [debug] [QVLmhWQ8nZwz] 404 Not Found (0.007804s, 128.140/s)
[Mojolicious] [QVLmhWQ8nZwz]
[2021-10-13 11:34:03.94516] [33] [trace] [zODsD2eq4jX2] GET "/edit"
[Mojolicious] [zODsD2eq4jX2]
[2021-10-13 11:34:03.94746] [33] [trace] [zODsD2eq4jX2] Routing to controller "LANraragi::Controller::Login" and action "logged_in"
[Mojolicious] [zODsD2eq4jX2]
[2021-10-13 11:34:03.94864] [33] [trace] [zODsD2eq4jX2] Routing to controller "LANraragi::Controller::Edit" and action "index"
[Mojolicious] [zODsD2eq4jX2]
[2021-10-13 11:34:03.95728] [33] [debug] Rendering template "edit.html.tt2"
[Mojolicious] Rendering template "edit.html.tt2"
[2021-10-13 11:34:03.98582] [33] [debug] [zODsD2eq4jX2] 200 OK (0.039295s, 25.448/s)
[Mojolicious] [zODsD2eq4jX2]
[2021-10-13 11:34:04.03880] [33] [trace] [WVjyoxGExVcs] GET "/api/database/stats"
[Mojolicious] [WVjyoxGExVcs]
[2021-10-13 11:34:04.04348] [33] [trace] [WVjyoxGExVcs] Routing to controller "LANraragi::Controller::Api::Database" and action "serve_tag_stats"
[Mojolicious] [WVjyoxGExVcs]
[Tag Stats] [debug] Serving tag statistics with a minimum weight of 2
[2021-10-13 11:34:04.04696] [33] [debug] [WVjyoxGExVcs] 200 OK (0.008148s, 122.730/s)
[Mojolicious] [WVjyoxGExVcs]
[2021-10-13 11:34:05.60168] [33] [trace] [ACKKVjtrTSBm] PUT "/api/archives/9e2b1155849dc9a9f7080035caad167ae63bca82/metadata"
[Mojolicious] [ACKKVjtrTSBm]
[2021-10-13 11:34:05.60392] [33] [trace] [ACKKVjtrTSBm] Routing to controller "LANraragi::Controller::Login" and action "logged_in_api"
[Mojolicious] [ACKKVjtrTSBm]
[2021-10-13 11:34:05.60648] [33] [trace] [ACKKVjtrTSBm] Routing to controller "LANraragi::Controller::Api::Archive" and action "update_metadata"
[Mojolicious] [ACKKVjtrTSBm]
[2021-10-13 11:34:05.63506] [33] [debug] [ACKKVjtrTSBm] 200 OK (0.033365s, 29.972/s)
[Mojolicious] [ACKKVjtrTSBm]
Mojo::Reactor::Poll: I/O watcher failed: Can't call method "emit" on an undefined value at /usr/local/share/perl5/site_perl/Mojo/Redis.pm line 60.
Mojo::Reactor::Poll: I/O watcher failed: Can't call method "emit" on an undefined value at /usr/local/share/perl5/site_perl/Mojo/Redis.pm line 60.
[2021-10-13 11:34:05.64946] [33] [trace] [AJscOr2AkTr9] POST "/api/plugins/use"
[Mojolicious] [AJscOr2AkTr9]
[2021-10-13 11:34:05.65074] [33] [trace] [AJscOr2AkTr9] Routing to controller "LANraragi::Controller::Login" and action "logged_in_api"
[Mojolicious] [AJscOr2AkTr9]
[2021-10-13 11:34:05.65408] [33] [trace] [AJscOr2AkTr9] Routing to controller "LANraragi::Controller::Api::Other" and action "use_plugin_sync"
[Mojolicious] [AJscOr2AkTr9]
[Plugin System] [info] Thumbnail hash invalid, regenerating.
[Plugin System] [info] No login plugin specified, returning empty UserAgent.
[Chaika.moe] [debug] Chaika API returned this JSON: [{"category": "Manga", "download": "/archive/26552/download/", "expunged": false, "filecount": 16, "filesize": 31800398, "fjord": false, "gallery": 22497, "id": 26552, "posted": 1510992507, "rating": 4.4, "tags": ["female:infantilism", "female:diaper", "artist:mozu", "female:garter_belt", "male:glasses", "male:dilf", "full_censorship", "female:urination", "female:exhibitionism", "female:stockings", "female:sex_toys", "language:translated", "language:english"], "title": "[Mozu] Office Baby (COMIC Kairakuten 2014-07) [English]", "title_jpn": "[もず] Office Baby (COMIC 快楽天 2014年7月号) [英訳]", "uploader": "drakenxx"}]
[Chaika.moe] [debug] Gallery ID detected(26552), trying to switch to it.
[Chaika.moe] [debug] Chaika API returned this JSON: {"archives": [{"download": "/archive/29032/download/", "id": 29032}], "category": "Manga", "expunged": false, "filecount": 20, "filesize": 0, "fjord": false, "posted": 0, "rating": 0.0, "tags": ["magazine:comic_x-eros_#70", "big_dick", "kimono", "publisher:fakku", "blowjob", "creampie", "eyebrows", "subscription", "monster_girl", "booty", "fangs", "bakunyuu", "chubby", "shota", "x-ray", "hentai", "artist:nanao_yukiji", "uncensored", "language:english"], "title": "A Godly Summer Vacation", "title_jpn": "", "uploader": ""}
[Chaika.moe] [info] Sending the following tags to LRR: magazine:comic x-eros #70, big dick, kimono, publisher:fakku, blowjob, creampie, eyebrows, subscription, monster girl, booty, fangs, bakunyuu, chubby, shota, x-ray, hentai, artist:nanao yukiji, uncensored, language:english
[Plugin System] [info] Applying tag rules...
[2021-10-13 11:34:08.18082] [33] [debug] [AJscOr2AkTr9] 200 OK (2.531343s, 0.396/s)
[Mojolicious] [AJscOr2AkTr9]
[2021-10-13 11:34:09.53732] [26] [debug] Process 37 is performing job "3" with task "warm_cache"
[Mojolicious] Process 37 is performing job "3" with task "warm_cache"
[Minion] [info] Warming up search cache...
[Search Engine] [debug] Search request: --title-asc-0-0
[Search Engine] [debug] No cache available, doing a full DB parse.
[2021-10-13 11:34:09.57396] [26] [debug] Process 39 is performing job "4" with task "build_stat_hashes"
[Mojolicious] Process 39 is performing job "4" with task "build_stat_hashes"
[Tag Stats] [debug] Building stat indexes...
[Tag Stats] [debug] Done!
[2021-10-13 11:34:10.55492] [14] [info] Creating process id file "/home/koyomi/lanraragi/public/temp/server.pid"
[Mojolicious] Creating process id file "/home/koyomi/lanraragi/public/temp/server.pid"
[Minion] [info] Done!
Difegue commented 3 years ago

Turns out the API changed a bit more than I thought and there's no need for the weird go-fetch-the-gallery-after-search workflow anymore, so I've simply removed it.