search

Difegue / LANraragi

Web application for archival and reading of manga/doujinshi. Lightweight and Docker-ready for NAS/servers.
https://lrr.tvc-16.science
MIT License
2.18k stars 154 forks source link

Issues with unicode, Image unknown file types #83

Closed CtahSaot closed 5 years ago

CtahSaot commented 5 years ago

First, I want to say that LRR is a gem. This is the most polished personal manga gallery manager out there and makes life so much easier. The UI and intuitiveness of it is also extremely pleasing. I'm so glad to be using it.

LRR Version and OS LRR 0.5.7, DSM/Linux using Docker

Bug Details

Issue 1: Since using 0.5.7 I've noticed that some archives with certain unicode characters keep getting pruned from the database. The LRR log says the archives can't be found. When I restore the database or re-add the archives, it will process them as normal and then spit out the same error afterwards.

Issue 2: On previous versions of LRR I never had trouble with images or thumbnailing. With 0.5.7 I am getting errors about corrupt archives with the following errors in logs. I checked the images and they appear to be fine and no other application seems to have issue with them either so I'm unsure of what the problem is.

Matching Logs

Logs for Issue 2: Image::Scale unknown file type (./public/temp/ac422e52736e44f2a14940551082e2c4333af2db/xxx(Final Fantasy VII)/image [01].jpg), first 8 bytes were: 00 00 00 00 00 00 00 00

Misc Mojo log that might be related, maybe unrelated though: [2019-02-01 06:13:33.26734] [6] [error] Parsing filters is unsupported. at /usr/local/lib/perl5/site_perl/Archive/Peek/Libarchive.pm line 20. [2019-02-01 06:13:33.33549] [6] [error] Parsing filters is unsupported. at /usr/local/lib/perl5/site_perl/Archive/Peek/Libarchive.pm line 20. [2019-02-01 06:13:33.49855] [6] [error] Parsing filters is unsupported. at /usr/local/lib/perl5/site_perl/Archive/Peek/Libarchive.pm line 20.

Screenshots

Screenshot for Issue 1: screenshot_2019-02-01 lanraragi - logs

Difegue commented 5 years ago

Thanks a lot for the support! It's what keeps me going. 🎌

I was entirely expecting issues like those to prop up when switching to Libarchive, so it doesn't come up as much of a surprise. Let's walk through this in detail:

CtahSaot commented 5 years ago

Bear with me as I try to document my goose chase findings with detail. I'm sure not all of it is necessary, but I'm not sure what is necessary, so there's plenty of spaghetti. 🍝

Do you have other archives with Unicode characters stored and working fine?

My archive gallery is quite small, so that's why the list in the screenshot above seems limited. Unfortunately from what I'm seeing, it's having trouble with all unicode characters, × and · are a couple more examples in the titles.

If you use the built-in upload tool to add one of those failing files, does it work?

I tried to upload an archive with unicode characters with the built-in tool and still ran into trouble. The first time I tried to manually add the archive, the tool displays a successful upload. Reviewing the gallery and logs shows the same problem about the archive not being found.

Additionally, I found a couple of other things worth noting, I think.

If a duplicate file is uploaded through the tool, a cache loop will occur endlessly until the related files are deleted by hand.

LRR log: [2019-02-03 13:38:59] [Shinobu] [warn] This ID exists in the Redis Database but doesn't have the same file! You might be having duplicate files! [2019-02-03 13:43:26] [Shinobu] [warn] Our file: /home/koyomi/lanraragi/content/Yomikiri/archive name with unicode.zip [2019-02-03 13:43:26] [Shinobu] [warn] Cached file: /home/koyomi/lanraragi/content/archive name with unicode.zip

When I upload the duplicate file through the tool, the Mojolicious log gives this message: [2019-02-03 13:44:44.12827] [302] [error] [hexists] ERR Protocol error: invalid bulk length, at /usr/local/share/perl5/site_perl/Redis.pm line 268.

I cleared the temp cache (not sure if that affects anything), deleted the newly uploaded file, and attempted to upload again.

After upload completion the tool displayed this message:
A file bearing this name already exists in the Library.

I went back to check the gallery and logs and I see that it's looping over and over again, trying to add the same archives with unicode, saying they can't be found, deleting the IDs. It displays the same message about the single archive I tried to upload with the tool, about it being cached. It does not say the same about the already existing archives I never moved out of the folders since this all began. Even though it says it's deleting the archives, the IDs remain the same in the loops afterwards.

Unicode archives cannot be found -> Deleting ID XXXXX -> Done! -> Detects manually uploaded file as already cached -> Archive count changes by -1 -> Rebuild forced -> Adding new file (same files) -> Auto-tagger -> Unicode archives cannot be found | Endless loop here

So this loop seems to be triggered by that duplicate file I uploaded through the tool, which crops up because of the old cached file still sitting in a nested directory where I originally put it. I'm only able to get this loop to stop when I delete the newly uploaded archive itself.

Even if I delete the old file that's nested in a folder along with the newly uploaded file, it never gets added to the gallery when reuploading.

* For **Issue 2**, Libarchive seems to be **way** more anal about corrupt archives than the previous solution, even if they seemingly work fine. I've noticed it a few times as well.

If you try extracting the archive and rebuilding it, does that version pass?

I extracted the archive and rebuilt. The same thing happens. I also tried to shorten the length of the archive name, tried changing the file names themselves. Same issue. I even opened the files in the archive in Photoshop to view the raw data and didn't see anything unusual there either.

I'll probably have to dig into the Perl bindings to see if this can be made more permissive. To that order, it'd be great if I had an archive that triggers the error on hand to have a reproducible case.

I could email you one of the 'corrupted' archives if you'd like. It's soft yaoi, so I don't think it's suitable for Github. 😂Let me know and I'll send it over.

Difegue commented 5 years ago

Damn, that's a lotta reading.

Unicode archives cannot be found -> Deleting ID XXXXX -> Done! -> Detects manually uploaded file as already cached -> Archive count changes by -1 -> Rebuild forced -> Adding new file (same files) -> Auto-tagger -> Unicode archives cannot be found | Endless loop here

Yes! It's a bit obscure here because your uploaded file is adding extra complexity, but this is basically normal.
The upload worked since the file was put at the root of the content folder, but you still have the original file in content/Yomikiri. That's the first thing.

The second part is that the IDs for the unicode files are pruned every time (since the filenames got mangled and it can't find them again), then re-created from the files that...never really went away. It mangles the filenames again on commiting to the database, and we're back to square one.

Basically, it's the good old unicode mismatch bug again. I solved it once already by enforcing utf8 locale within the Docker container, but the switch from Debian to Alpine miiight have broken that. You can send me one of the corrupt archives, but my guess is that it'll work OK on my side.

Additional search on my end shows that locale support in Alpine kinda sucks, so I might very well have to do some extra Dockerfile plumbing to fix this.

I hope to have a nightly ready for testing with those three fixes in a few days. ✌

Difegue commented 5 years ago

Good to go! 🙌 I don't have access to my full Docker test environment right now to make sure everything works fine, but the nightly images now properly build musl UTF8 locales.

Difegue commented 5 years ago

I've tested the latest nightly image on my rig without any outstanding issues -- You should be able to pull it and check if it solves your issue.

CtahSaot commented 5 years ago

Alright, got a moment to pull nightly. I'm now seeing this pop up in the logs before it says the same archives above can't be found.

Wide character in say at /home/koyomi/lanraragi/lib/../lib/LANraragi/Utils/Generic.pm line 95.

Difegue commented 5 years ago

Dang, this blows. I might have fumbled up the locale install a bit.
I'll set up a DSM VM on my side to try and reproduce this on my end.

Update: I got a DSM 5.2 VM running the nightly Docker container and didn't encounter issues related to Unicode archives. (except a small hiccup in their name in logs but that didn't harm setup or reading in any way)
So I guess you're stuck being my guinea pig for now on this. 🙇‍♂️ I've pushed a new commit, and henceforth a new image to the nightlies that corrects a fuckup with the musl locale environment variable.
If that doesn't fix the issue, I still have the glibc compatibility route to try.

CtahSaot commented 5 years ago

Sorry for taking so long to get back to you.

Pulled the new nightly and am still able to reproduce the problem.

LRR Log:

[2019-02-13 02:44:09] [LANraragi] [info] LANraragi 0.5.7 started. (Production Mode)
[2019-02-13 02:44:09] [LANraragi] [info] Plugin Detected: Chaika.moe
[2019-02-13 02:44:09] [LANraragi] [info] Plugin Detected: Tag Copier
[2019-02-13 02:44:09] [LANraragi] [info] Plugin Detected: E-Hentai
[2019-02-13 02:44:09] [LANraragi] [info] Plugin Detected: eze
[2019-02-13 02:44:09] [LANraragi] [info] Plugin Detected: Hdoujin
[2019-02-13 02:44:09] [LANraragi] [info] Plugin Detected: nHentai
[2019-02-13 02:44:10] [Shinobu] [info] Shinobu Background Worker started -- Content folder will be scanned every 5 seconds.
[2019-02-13 02:44:10] [Shinobu] [info] Working dir is /home/koyomi/lanraragi
[2019-02-13 02:45:00] [Shinobu] [info] Archive count (0) has changed since last cached value (0) OR rebuild has been forced (flag value = 1), rebuilding...
[2019-02-13 02:45:00] [Shinobu] [info] Done!
[2019-02-13 02:45:06] [LANraragi] [info] LANraragi 0.5.7 (re-)started. (Debug Mode)
[2019-02-13 02:45:06] [LANraragi] [info] Plugin Detected: Chaika.moe
[2019-02-13 02:45:06] [LANraragi] [info] Plugin Detected: Tag Copier
[2019-02-13 02:45:06] [LANraragi] [info] Plugin Detected: E-Hentai
[2019-02-13 02:45:06] [LANraragi] [info] Plugin Detected: eze
[2019-02-13 02:45:06] [LANraragi] [info] Plugin Detected: Hdoujin
[2019-02-13 02:45:06] [LANraragi] [info] Plugin Detected: nHentai
[2019-02-13 02:45:06] [LANraragi] [info] Terminating previous Shinobu Worker if it exists... (PID is 13)
[2019-02-13 02:45:06] [LANraragi] [debug] Shinobu Worker new PID is 13
[2019-02-13 02:45:06] [LANraragi] [debug] Routing done! Ready to receive requests.
[2019-02-13 02:45:07] [Shinobu] [info] Shinobu Background Worker started -- Content folder will be scanned every 5 seconds.
[2019-02-13 02:45:07] [Shinobu] [info] Working dir is /home/koyomi/lanraragi
[2019-02-13 02:45:42] [Shinobu] [info] Adding new file /home/koyomi/lanraragi/content/Doujins/(Convention) [Circle (Artist)] Title (Pokémon).rar with ID f4f29636fb80467394dbff0b4f54e0da876963dd
[2019-02-13 02:45:42] [Archive] [debug] Pushing to redis on ID f4f29636fb80467394dbff0b4f54e0da876963dd:
[2019-02-13 02:45:42] [Archive] [debug] File Name: (Convention) [Circle (Artist)] Title (Pokémon)
[2019-02-13 02:45:42] [Archive] [debug] Filesystem Path: /home/koyomi/lanraragi/content/Doujins/(Convention) [Circle (Artist)] Title (Pokémon).rar
[2019-02-13 02:45:42] [Shinobu] [info] Archive count (1) has changed since last cached value (0) OR rebuild has been forced (flag value = 0), rebuilding...
[2019-02-13 02:45:42] [Shinobu] [warn] Deleting ID f4f29636fb80467394dbff0b4f54e0da876963dd - File /home/koyomi/lanraragi/content/Doujins/(Convention) [Circle (Artist)] Title (Pokémon).rar cannot be found.
[2019-02-13 02:45:42] [Shinobu] [debug] Treated 1 archives out of 1 .
[2019-02-13 02:45:42] [Shinobu] [info] Done!

Mojo Log:

[2019-02-13 02:45:06.80698] [6] [info] Listening at "http://*:3000"
[2019-02-13 02:45:13.87761] [6] [debug] GET "/" (6932ece3)
[2019-02-13 02:45:13.88467] [6] [debug] Routing to controller "LANraragi::Controller::Index" and action "index"
[2019-02-13 02:45:13.94613] [6] [debug] Rendering template "index.html.tt2"
[2019-02-13 02:45:13.99311] [6] [debug] 200 OK (0.115471s, 8.660/s)
[2019-02-13 02:45:14.23264] [6] [debug] GET "/api/archivelist" (48d533ac)
[2019-02-13 02:45:14.24438] [6] [debug] Routing to controller "LANraragi::Controller::Api" and action "serve_archivelist"
[2019-02-13 02:45:14.24626] [6] [debug] 200 OK (0.013575s, 73.665/s)
[2019-02-13 02:45:14.61165] [6] [debug] GET "/css/webfonts/fa-solid-900.woff2" (f06bf96c)
[2019-02-13 02:45:14.61401] [6] [debug] Template "not_found.development.html.tt2" not found
[2019-02-13 02:45:14.61483] [6] [debug] Template "not_found.html.tt2" not found
[2019-02-13 02:45:14.61650] [6] [debug] Rendering template "mojo/debug.html.ep"
[2019-02-13 02:45:14.65853] [6] [debug] 404 Not Found (0.046839s, 21.350/s)
[2019-02-13 02:45:45.20701] [6] [debug] GET "/" (1d46acd0)
[2019-02-13 02:45:45.20752] [6] [debug] Routing to controller "LANraragi::Controller::Index" and action "index"
[2019-02-13 02:45:45.24296] [6] [debug] Rendering template "index.html.tt2"
[2019-02-13 02:45:45.24660] [6] [debug] 200 OK (0.039532s, 25.296/s)
[2019-02-13 02:45:45.49559] [6] [debug] GET "/css/webfonts/fa-solid-900.woff2" (9a39be41)
[2019-02-13 02:45:45.49762] [6] [debug] Template "not_found.development.html.tt2" not found
[2019-02-13 02:45:45.49853] [6] [debug] Template "not_found.html.tt2" not found
[2019-02-13 02:45:45.49890] [6] [debug] Rendering cached template "mojo/debug.html.ep"
[2019-02-13 02:45:45.50854] [6] [debug] 404 Not Found (0.012912s, 77.447/s)
[2019-02-13 02:45:45.51182] [6] [debug] GET "/api/archivelist" (0d6974d2)
[2019-02-13 02:45:45.51242] [6] [debug] Routing to controller "LANraragi::Controller::Api" and action "serve_archivelist"
[2019-02-13 02:45:45.51420] [6] [debug] 200 OK (0.002339s, 427.533/s)
[2019-02-13 02:45:48.84952] [6] [debug] GET "/logs" (10032d23)
[2019-02-13 02:45:48.85213] [6] [debug] Routing to controller "LANraragi::Controller::Login" and action "logged_in"
[2019-02-13 02:45:48.85679] [6] [debug] Routing to controller "LANraragi::Controller::Logging" and action "index"
[2019-02-13 02:45:48.85936] [6] [debug] Rendering template "logs.html.tt2"
[2019-02-13 02:45:48.86318] [6] [debug] 200 OK (0.01354s, 73.855/s)
[2019-02-13 02:45:49.08271] [6] [debug] GET "/logs/general" (bf6d3078)
[2019-02-13 02:45:49.08395] [6] [debug] Routing to controller "LANraragi::Controller::Login" and action "logged_in"
[2019-02-13 02:45:49.08464] [6] [debug] Routing to controller "LANraragi::Controller::Logging" and action "print_general"
[2019-02-13 02:45:49.08745] [6] [debug] 200 OK (0.004699s, 212.811/s)
[2019-02-13 02:49:08.25467] [6] [debug] GET "/logs/plugins" (2754a20a)
[2019-02-13 02:49:08.25583] [6] [debug] Routing to controller "LANraragi::Controller::Login" and action "logged_in"
[2019-02-13 02:49:08.25652] [6] [debug] Routing to controller "LANraragi::Controller::Logging" and action "print_plugins"
[2019-02-13 02:49:08.25721] [6] [debug] 200 OK (0.002509s, 398.565/s)
[2019-02-13 02:49:08.43760] [6] [debug] GET "/logs/plugins" (ee0a00b0)
[2019-02-13 02:49:08.43814] [6] [debug] Routing to controller "LANraragi::Controller::Login" and action "logged_in"
[2019-02-13 02:49:08.43886] [6] [debug] Routing to controller "LANraragi::Controller::Logging" and action "print_plugins"
[2019-02-13 02:49:08.43951] [6] [debug] 200 OK (0.001884s, 530.786/s)
[2019-02-13 02:49:09.48138] [6] [debug] GET "/logs/mojo" (1e246fb5)
[2019-02-13 02:49:09.48252] [6] [debug] Routing to controller "LANraragi::Controller::Login" and action "logged_in"
[2019-02-13 02:49:09.48327] [6] [debug] Routing to controller "LANraragi::Controller::Logging" and action "print_mojo"
Difegue commented 5 years ago

Alright, still the same error it seems.
I'll try out the libc route and see where that gets us. If that fails, I'm probably going back to the drawing board to figure out a better way to handle file detection.

Difegue commented 5 years ago

I've propped up a new nightly, tell me how it goes. 👍

CtahSaot commented 5 years ago

Pulled the new nightly, still no success with my cursed archives. If there's anything specific you'd like me to try, feel free to instruct.

LRR Logs:

[2019-02-16 16:17:45] [LANraragi] [info] LANraragi 0.5.7 started. (Production Mode)
[2019-02-16 16:17:45] [LANraragi] [info] Plugin Detected: Chaika.moe
[2019-02-16 16:17:45] [LANraragi] [info] Plugin Detected: Tag Copier
[2019-02-16 16:17:45] [LANraragi] [info] Plugin Detected: E-Hentai
[2019-02-16 16:17:45] [LANraragi] [info] Plugin Detected: eze
[2019-02-16 16:17:45] [LANraragi] [info] Plugin Detected: Hdoujin
[2019-02-16 16:17:45] [LANraragi] [info] Plugin Detected: nHentai
[2019-02-16 16:17:46] [Shinobu] [info] Shinobu Background Worker started -- Content folder will be scanned every 5 seconds.
[2019-02-16 16:17:46] [Shinobu] [info] Working dir is /home/koyomi/lanraragi
[2019-02-16 16:18:41] [Shinobu] [info] Archive count (0) has changed since last cached value (0) OR rebuild has been forced (flag value = 1), rebuilding...
[2019-02-16 16:18:41] [Shinobu] [info] Done!
[2019-02-16 16:19:03] [LANraragi] [info] LANraragi 0.5.7 (re-)started. (Debug Mode)
[2019-02-16 16:19:03] [LANraragi] [info] Plugin Detected: Chaika.moe
[2019-02-16 16:19:03] [LANraragi] [info] Plugin Detected: Tag Copier
[2019-02-16 16:19:03] [LANraragi] [info] Plugin Detected: E-Hentai
[2019-02-16 16:19:03] [LANraragi] [info] Plugin Detected: eze
[2019-02-16 16:19:03] [LANraragi] [info] Plugin Detected: Hdoujin
[2019-02-16 16:19:03] [LANraragi] [info] Plugin Detected: nHentai
[2019-02-16 16:19:04] [LANraragi] [info] Terminating previous Shinobu Worker if it exists... (PID is 12)
[2019-02-16 16:19:04] [LANraragi] [debug] Shinobu Worker new PID is 14
[2019-02-16 16:19:04] [LANraragi] [debug] Routing done! Ready to receive requests.
[2019-02-16 16:19:04] [Shinobu] [info] Shinobu Background Worker started -- Content folder will be scanned every 5 seconds.
[2019-02-16 16:19:04] [Shinobu] [info] Working dir is /home/koyomi/lanraragi
[2019-02-16 16:20:20] [Shinobu] [info] Adding new file /home/koyomi/lanraragi/content/[Circle (Artist)] Title  (Pokémon Sun and Moon).rar with ID bc839f0d0af0f9ec05257c879fd413fa4b9c7ebe
[2019-02-16 16:20:20] [Archive] [debug] Pushing to redis on ID bc839f0d0af0f9ec05257c879fd413fa4b9c7ebe:
[2019-02-16 16:20:20] [Archive] [debug] File Name: [Circle (Artist)] Title  (Pokémon Sun and Moon)
[2019-02-16 16:20:20] [Archive] [debug] Filesystem Path: /home/koyomi/lanraragi/content/[Circle (Artist)] Title  (Pokémon Sun and Moon).rar
[2019-02-16 16:20:20] [Auto-Tagger] [info] Executing enabled plugins on archive with id bc839f0d0af0f9ec05257c879fd413fa4b9c7ebe.
[2019-02-16 16:20:20] [Shinobu] [info] Archive count (1) has changed since last cached value (0) OR rebuild has been forced (flag value = 0), rebuilding...
[2019-02-16 16:20:20] [Shinobu] [warn] Deleting ID bc839f0d0af0f9ec05257c879fd413fa4b9c7ebe - File /home/koyomi/lanraragi/content/[Circle (Artist)] Title  (Pokémon Sun and Moon).rar cannot be found.
[2019-02-16 16:20:20] [Shinobu] [debug] Treated 1 archives out of 1 .
[2019-02-16 16:20:20] [Shinobu] [info] Done!

Mojo Logs:

[2019-02-16 16:19:04.16083] [7] [info] Listening at "http://*:3000"
[2019-02-16 16:19:09.45411] [7] [debug] GET "/" (49289eb0)
[2019-02-16 16:19:09.46215] [7] [debug] Routing to controller "LANraragi::Controller::Index" and action "index"
[2019-02-16 16:19:09.54329] [7] [debug] Rendering template "index.html.tt2"
[2019-02-16 16:19:09.59487] [7] [debug] 200 OK (0.140795s, 7.103/s)
[2019-02-16 16:19:09.85352] [7] [debug] GET "/css/webfonts/fa-solid-900.woff2" (1055daec)
[2019-02-16 16:19:09.85698] [7] [debug] Template "not_found.development.html.tt2" not found
[2019-02-16 16:19:09.85836] [7] [debug] Template "not_found.html.tt2" not found
[2019-02-16 16:19:09.86031] [7] [debug] Rendering template "mojo/debug.html.ep"
[2019-02-16 16:19:09.90621] [7] [debug] 404 Not Found (0.052642s, 18.996/s)
[2019-02-16 16:19:09.91039] [7] [debug] GET "/api/archivelist" (b2e60399)
[2019-02-16 16:19:09.92300] [7] [debug] Routing to controller "LANraragi::Controller::Api" and action "serve_archivelist"
[2019-02-16 16:19:09.92508] [7] [debug] 200 OK (0.014644s, 68.287/s)
[2019-02-16 16:20:24.28943] [7] [debug] GET "/" (f53aef8c)
[2019-02-16 16:20:24.29003] [7] [debug] Routing to controller "LANraragi::Controller::Index" and action "index"
[2019-02-16 16:20:24.33234] [7] [debug] Rendering template "index.html.tt2"
[2019-02-16 16:20:24.33577] [7] [debug] 200 OK (0.046281s, 21.607/s)
[2019-02-16 16:20:24.61369] [7] [debug] GET "/api/archivelist" (a5199844)
[2019-02-16 16:20:24.93482] [7] [debug] Routing to controller "LANraragi::Controller::Api" and action "serve_archivelist"
[2019-02-16 16:20:24.93724] [7] [debug] 200 OK (0.323497s, 3.091/s)
[2019-02-16 16:20:24.94105] [7] [debug] GET "/css/webfonts/fa-solid-900.woff2" (7f299cff)
[2019-02-16 16:20:24.94377] [7] [debug] Template "not_found.development.html.tt2" not found
[2019-02-16 16:20:24.94486] [7] [debug] Template "not_found.html.tt2" not found
[2019-02-16 16:20:24.94534] [7] [debug] Rendering cached template "mojo/debug.html.ep"
[2019-02-16 16:20:24.95717] [7] [debug] 404 Not Found (0.016127s, 62.008/s)
[2019-02-16 16:20:26.47334] [7] [debug] GET "/logs" (a9cd59e2)
[2019-02-16 16:20:26.47600] [7] [debug] Routing to controller "LANraragi::Controller::Login" and action "logged_in"
[2019-02-16 16:20:26.54929] [7] [debug] Routing to controller "LANraragi::Controller::Logging" and action "index"
[2019-02-16 16:20:26.55185] [7] [debug] Rendering template "logs.html.tt2"
[2019-02-16 16:20:26.55531] [7] [debug] 200 OK (0.081934s, 12.205/s)
[2019-02-16 16:20:26.80627] [7] [debug] GET "/logs/general" (dc42e2fc)
[2019-02-16 16:20:26.80745] [7] [debug] Routing to controller "LANraragi::Controller::Login" and action "logged_in"
[2019-02-16 16:20:26.80815] [7] [debug] Routing to controller "LANraragi::Controller::Logging" and action "print_general"
[2019-02-16 16:20:26.80965] [7] [debug] 200 OK (0.003503s, 285.470/s)
[2019-02-16 16:21:14.72770] [7] [debug] GET "/logs/mojo" (6f7fd439)
[2019-02-16 16:21:14.72896] [7] [debug] Routing to controller "LANraragi::Controller::Login" and action "logged_in"
[2019-02-16 16:21:14.72965] [7] [debug] Routing to controller "LANraragi::Controller::Logging" and action "print_mojo"
Difegue commented 5 years ago

Well fuck me, I guess this shows the limitations of the current file detection system.
I have a few other related issues in store, so I think I'm due for a big ol' overhaul.

For the time being, I guess your only option is to rollback to the previous release and its Debian-based Docker images. I'll keep you updated when I have something new to try out.

Difegue commented 5 years ago

I think I have a pretty good grasp on the issue now, it's actually not unicode at all. It's ISO-8859.
Filenames encoded in 8859 have different codes for some characters than UTF-8(typically é/è, × and ·) and since we assume UTF-8 everywhere it led to failures in re-recognizing the file. Files with Japanese characters are usually full UTF-8 and function without any issues here.

I still think the system locale is somehow at fault, since this mismatch doesn't seem to happen on full UTF-8 systems -- whatever. A good chunk of the app has been rewritten to be way less dependent on filenames (#89), so right now they're not used to check for file presence anymore. At the very least you should be seeing the archives in your Index again.

I still use the filename stored in the database to extract archives for reading though, so tell me how opening up a file goes on your end. It should work(does on my test case), but at this point I shouldn't be surprised at anything.

Edit: scratch that, seems reading is still broken on the latest nightly even for me.

Difegue commented 5 years ago

Just removing any form of encoding/decoding when handling filenames in the DB (including replacing File::Find::utf8 which did encoding behind my back) basically gets rid of this entire headache. Sometimes it's just simpler to remove code.

I think this nightly build should be the one! Tell me how it goes for you.

koma-cute commented 5 years ago

I would implore that the changes introduced in 7b4c90d be reverted. Tainting your (and all users of your Docker image) builds with random untrusted code from a GitHub repo is a bad idea. Just use the Debian base.

Difegue commented 5 years ago

If I were to not trust random code from the internet, I probably would get rid of 75% of my dependencies...
Then again the changes in this Dockerfile have been reverted already, I recommend you look at the version from 89d5d79b289f92347e8f6512dee3d9a78e35041d which is the latest one.

I can probably remove the musl-locales stuff and it'll work fine now considering the changes I've made, but I can't get rid of the patched Crypt::Rijndael yet -- and that also falls under the "random code from a Git repo" umbrella. 🤷‍♀️

koma-cute commented 5 years ago

I suppose that's true. Just seeing the packages installed that way set off my "bad idea" alarm. I'd say as long as you can pin it to a known good version through signing or hashes, that would be ideal.

Difegue commented 5 years ago

Sure, that I can do. Hopefully this stuff will eventually be available in the alpine repos directly.

Difegue commented 5 years ago

I'll be unavailable for the rest of the week and therefore released 0.5.8 today to alleviate other issues such as the nH plugin.

@CtahSaot, I'll be closing this issue for the time being as I'm pretty darn sure it's fixed, but if you still encounter the problem, make sure to re-open this or make a new issue. ✌