Plugin System Improvements + ExHentai.org support

[Difegue]

Currently, Plugins can only accept two arguments, and one of them is reserved for file-specific operations (when you specify an EH URL for example).

Being able to give more arguments to a Plugin would make them more configurable, by allowing more than one option field in Plugin configuration.

[AnyByte]

It will be nice to have exhentai.org support in addition to e-hentai.org, because e-hentai.org is blocked in some countries (Russia). It should be easy because API should be nearly identical. I tried to shitcode exhentai plugin myself just by changing every e-hentai to exhentai and it still didnt work.

Also logs were useless cause I should have seen output from $logger->debug("E-H API returned this JSON: $textrep"); on line 207 somewhere after every request but I didn't.

[Difegue]

exhentai requires to code in the entire sadpanda cookie bullshit thing, which there's (understandably) little documentation for. (disclaimer I do have exhentai access, just don't know how to do it programmatically)

I tried to do it back when making the e-hentai plugin, basing myself on the happypanda code, but couldn't get past the panda and gave up.

There are scraps of my attempt as comments in this version of the plugin, but I doubt it'll be of much help. Proves I tried tho 👀

As a workaround, you can use the e-hentai plugin manually with exhentai URLs when editing metadata on a per-archive basis, it just works™

(Also - debug logs only print when LRR is in debug mode!)

[Difegue]

This is going on the burner for 0.5.5 - Current plan is to switch the global plugin arguments to accept an array instead of a single string, and work from there.

ExH support will be done through mocking a login to the E-H forums instead of using the ipb_* pair.

[Difegue]

Developer documentation will need to be edited to reflect the upgrade to the plugin system.

[Difegue]

All good in the hood as of the latest commit! Documentation will be updated later.

[AnyByte]

Im sorry I might be annoying but Im getting this error while trying to get metadata from e-hentai with or without gallery url:

[Sun Oct 21 12:32:31 2018] [debug] POST "/api/use_plugin" (c391a5b8)
[Sun Oct 21 12:32:31 2018] [debug] Routing to controller "LANraragi::Controller::Login" and action "logged_in"
[Sun Oct 21 12:32:31 2018] [debug] Routing to controller "LANraragi::Controller::Api" and action "use_plugin"
[Sun Oct 21 12:32:31 2018] [error] Can't use an undefined value as a HASH reference at /home/koyomi/lanraragi/script/../lib/LANraragi/Plugin/EHentai.pm line 311.

[Sun Oct 21 12:32:31 2018] [debug] Template "exception.development.html.tt2" not found
[Sun Oct 21 12:32:31 2018] [debug] Template "exception.html.tt2" not found
[Sun Oct 21 12:32:31 2018] [debug] Rendering cached template "mojo/debug.html.ep"
[Sun Oct 21 12:32:31 2018] [debug] Rendering cached template "mojo/menubar.html.ep"
[Sun Oct 21 12:32:31 2018] [debug] 500 Internal Server Error (0.067946s, 14.718/s)

In plugin logs it says that Im successfully logged in:

[2018-10-21 12:29:54] [E-Hentai] [info] E-Hentai credentials present, trying to login as user MaxAvatar
[2018-10-21 12:29:54] [ExHentai] [info] Login successful! Trying to access ExHentai...
[2018-10-21 12:29:55] [ExHentai] [info] ExHentai status OK! Moving on.
[2018-10-21 12:29:55] [E-Hentai] [info] Reverse Image Search Enabled, trying first.

[Difegue]

No problem at all! Line 311 is when we send back the tags to the main app, weird that it'd choke there and not anywhere else before.

Mind switching your installation to Debug mode? That'll make more logs available.

[AnyByte]

No problem at all! Line 311 is when we send back the tags to the main app, weird that it'd choke there and not anywhere else before.

Mind switching your installation to Debug mode? That'll make more logs available.

Debug mode was already turned on when I copied these logs, maybe Im supposed to get them somewhere inside the container? Because it seems like I don't get more logs with it turned on although I restarted container after changing mode to debug. Maybe I can PM you my instance so you could check it yourself.

[Difegue]

Woops, looks like debug logs don't show on the log view. I use the console output directly when debugging so I completely missed that - goes to show I don't get bug reports very often.

I've pushed a commit that fixes this issue - You should be getting the detailed debug logs now once Docker Cloud is done building and you update your image.

[AnyByte]

Welp now Im getting this: screenshot Although I cleared JSON cache and deleted redis database to let redis recreate it from scratch.

From docker logs:

lanraragi    | [Mojolicious] Template "exception.development.html.tt2" not found
lanraragi    | [Mojolicious] Template "exception.html.tt2" not found
lanraragi    | [Mojolicious] Rendering cached template "mojo/debug.html.ep"
lanraragi    | [Mojolicious] Rendering cached template "mojo/menubar.html.ep"
lanraragi    | [Mojolicious] 500 Internal Server Error (0.005389s, 185.563/s)
lanraragi    | [Mojolicious] GET "/" (c6bcd46f)
lanraragi    | [Mojolicious] Routing to controller "LANraragi::Controller::Index" and action "index"
lanraragi    | [Mojolicious] Rendering template "index.html.tt2"
lanraragi    | [Mojolicious] 200 OK (0.018275s, 54.720/s)
lanraragi    | [Mojolicious] GET "/api/archivelist" (89e210a1)
lanraragi    | [Mojolicious] Routing to controller "LANraragi::Control
lanraragi    | 2018-10-23 17:15:40,581 DEBG 'lanraragi' stdout output:
lanraragi    | ler::Api" and action "serve_archivelist"
lanraragi    | [Mojolicious] Wide character in subroutine entry at /usr/local/share/perl/5.24.1/Mojo/JSON.pm line 39.

[Difegue]

The dreadful wide character error -- I guess the archive API commit that I did earlier today isn't very unicode-compliant. I'd only tried it briefly with the demo (which doesn't have japanese names) and assumed mojo::json did utf8 seamlessly. At least now I know the error handling works properly. ✌

I have fixed it! Again!
Sorry for making you pull repeated Docker images. That's the price when living on the edge™. 🎌

[AnyByte]

Im glad to help. And now I got why it couldn't get tags from e-hentai (thanks to cloudflare): Im guessing captcha was not intended to show up like that in logs lol. screenshot Any way to show some modal with this captcha and allow server to grab cookies? Cause when I click on my PC it just reloads the page. Full log:

Please enable cookies.

One more step

Please complete the security check to access e-hentai.org

Why do I have to complete a CAPTCHA?

Completing the CAPTCHA proves you are a human and gives you temporary access to the web property.

What can I do to prevent this in the future?

If you are on a personal connection, like at home, you can run an anti-virus scan on your device to make sure it is not infected with malware.

If you are at an office or shared network, you can ask the network administrator to run a scan across the network looking for misconfigured or infected devices.

Cloudflare Ray ID: 46e7ff4fc7c58d8f
•
Your IP: 123.123.123.123
•
Performance & security by Cloudflare

[Difegue]

Oh, that one's pretty good. We dump the full HTML response in debug logs, so it makes sense that it'd appear this way.

I'm sorry to say that captchas are a bit out of spec for me though. The e-hentai forum login form can also ask for a captcha on initial loading (which we recognize and emit a warning about), but cloudflare captchas are more of an issue with your IP address/network.

I remember you saying e-h was blocked for you -- We use its forums to login and gain the cookies for ex access, so that might be your issue.

[AnyByte]

Well my server is connected to a VPN and Im guessing that cloud flare detects it. Maybe theres a way to paste cookies so that server will have same ones as my PC? Or maybe open some kind of iframe with a login page.

[Difegue]

I can try adding cookie values as parameters, yeah. (ipb_user, ipb_hash and igneous iirc)
They would be used instead of the login/password pair if provided.

Iframes or any kind of user interaction mid-execution are a no-go: Plugins are meant to be a black box, "data goes in tags come out" kind of deal.

[Difegue]

Added an alternative way to login using cookies only. I don't have my ex cookies on hand so I couldn't fully test access, but it shouldn't error out at least. More on this later when I run testing.

[Difegue]

Alright, here's a version that actually works. Turns out the online documentation about the cookie magic wasn't really up to date - luckily it wasn't too far off the mark so I was able to guess the rest.