search

riderkick / FMD

Forked from https://sf.net/p/fmd/
GNU General Public License v2.0
720 stars 213 forks source link

MangaDex Problems #884

Closed archet-VIII closed 6 years ago

archet-VIII commented 6 years ago

Upgraded to the newest version to try it out on the newest site, but it's not going well.

First, updating the manga list 'from server' returns a 404. Using the first update option works, although with nearly 20k manga on the list it does take quite a while.

Second, program can freeze or crash when loading the manga info. This particularly seems to happen when trying to load manga from past the first third or so of the list; I first ran into it when trying to load one that started with a Z, and later when testing it letter by letter ran into it again in the Ks. The first instance it froze the program and monopolized around 50% of the CPU for ~15 minutes before I terminated it, the second it just froze while using negligible resources.

The third time Windows came up with its 'unresponsive program' box twice, first over the manga list pane and then over the info pane. Clicking past those brought up a 'program exception' dialog with the following in the 'more info' area:

Program exception! Application : Free Manga Downloader Version : 0.9.131.0 Product Version : Win32 Debug Host Machine : Windows 8.1 64-bit FPC Version : 3.0.4 LCL Version : 1.9.0.0 WidgetSet : Win32/Win64 Target CPU-OS : i386-Win32 Build Time : 2018/01/27 02:26:03 Path : C:[Archives][Portable Programs]\fmd_0.9.88.0\fmd.exe Process ID : 8288 MainThread ID : 7880 Thread ID : 10780 Time : 27/01/2018 14:37:01.715 Sender Class : TGetMangaInfosThread Exception Class : EInvalidPointer Message : Invalid pointer operation $004431ED $004432CE $00442CD5 $00412A1E $764E919F $773CA8CB $773CA8A1

The fourth attempt just brought up the 'unresponsive program' boxes without the 'exception' dialog.

For the manga that do load, download does seem to be functioning properly. There are some chapters where the images don't download, but that's a known problem on MangaDex's side affecting the first ~5000 or so files uploaded to the site. The more recent ones work fine.

nille02 commented 6 years ago

i have non of your problems, but only problems with downloading. E.g. he download only Page 1,2,3,34 from Tsuki Tsuki Chapter 1. The other 4 Chapters got similar problems. https://mangadex.com/manga/775

Similar issues i got with this Series.

https://mangadex.com/manga/4280 https://mangadex.com/manga/9750

TakashiOkada23 commented 6 years ago

I'm also having problems downloading manga from mangadex. it downloaded fine for a while but then it started giving me errors. I kept pressing retry but then I realized nothing was downloading even though FMD was using all my bandwidth. I was able to download 1.13 GB before it stopped working. my guess is that it has a bandwidth limit per IP or something along those lines.

alkasul commented 6 years ago

I have the first issue which downloading from server gives a 404, in fact I had it since the previous update but I was lazy to report it since I can just use the update manga list.

TakashiOkada23 commented 6 years ago

sorry. I don't know what to do with that link(?) you gave me @kmvi I don't find a download button for that file you linked

nille02 commented 6 years ago

@alkasul the reason is, that there is no prebuild index online. you can ignore this. Update the List Create a new List that is complete.

archet-VIII commented 6 years ago

i have non of your problems, but only problems with downloading. E.g. he download only Page 1,2,3,34 from Tsuki Tsuki Chapter 1. The other 4 Chapters got similar problems.

There's some persistent issues on MangaDex's side around how it handles the uploaded archive files. Aside from the first couple days worth of uploads completely breaking and random chapters only having some of their images uploading more recently, they've had various problems getting it to work with archives that had both pngs and jpgs in them. They initially were not even able to be uploaded because the reader couldn't parse them, and while they have since gotten them mostly working on the site itself I would guess that there's still some issue with them that causes problems for FMD, as the jpg images from that chapter you refer to are the only ones that downloaded. The chapters I was able to download were all uploaded solely as jpgs, so I hadn't run into that.

kmvi commented 6 years ago

@archet-VIII just finished updating the list (~5 hours), no errors happened. Db attached Mangadex.zip @punpun3 this is a correction for future release

leoteesh commented 6 years ago

When i try to download mangadex, i get an error message saying file not found image

archet-VIII commented 6 years ago

@kmvi Afraid I'm still getting the same result, even with the .db file you provided. Trying to open manga info pages in the later half of the list consistently freezes up the program. Got another 'program exception' message in the process.

Program exception! Application : Free Manga Downloader Version : 0.9.131.0 Product Version : Win32 Debug Host Machine : Windows 8.1 64-bit FPC Version : 3.0.4 LCL Version : 1.9.0.0 WidgetSet : Win32/Win64 Target CPU-OS : i386-Win32 Build Time : 2018/01/27 02:26:03 Path : C:[Archives][Portable Programs]\fmd_0.9.88.0\fmd.exe Process ID : 11632 MainThread ID : 9808 Thread ID : 9808 Time : 28/01/2018 05:58:21.897 Exception Class : EInvalidPointer Message : Invalid pointer operation $00411E9B $00411D07 $004C709A $004C71E8 $004CD00F $004CCEF2 $0071F22B $00528903 $00519059 $0051BB2D $0052885D $00519E82 $00721C9B $00596404 $00501938 $0050446F $00504FBB

riderkick commented 6 years ago

@kmvi give me your sf username. I'll add you to admin group so you can add/update db files.

kmvi commented 6 years ago

@riderkick https://sf.net/u/kmvi

riderkick commented 6 years ago

Done. Data files is in '/home/pfs/project/newfmd/data'. Just use sftp to update. Use 7z with highest compression level if possible.

nille02 commented 6 years ago

@archet-VIII maybe you are right, but on the website all pages are working.

riderkick commented 6 years ago

@archet-VIII you should post the url of series you try to load. I don't see any reasonable line in mangadex's getinfo source that can cause EInvaldPointer. Invalid pointer usually causing by a code that try to access uninitialized/freed object.

karehaqt commented 6 years ago

Just to say I'm also finding issues with downloading from Mangadex, it's like the download gets a little way in then it starts again and does that over and over again.

nille02 commented 6 years ago

With the new Program Version the downloads are working fine for me.

karehaqt commented 6 years ago

Nice @nille02 I'll give it a try when I get home from work tonight :+1:

archet-VIII commented 6 years ago

@riderkick I first encountered issues when trying to open Zannen Jokanbu Black General-san; https://mangadex.com/manga/19657 which was actually the first thing I did after the list had finally been built. The next couple tries eventually produced that first exception error I posted, and subsequent attempts picking random entries on the list produced the second.

I updated to the new version this morning as was initially able to open Zannen Jokanbu's info without any problems, but upon trying to open a second random manga in the Zs the same issues immediately reoccurred. I can apparently freely open manga in the first third of the list, but trying to open anything past J or so causes crashes. Sometimes with the program exception message, but more often without. Here's some of the ones I picked up in the last little while:

Youkai Apaato no Yuuga na Nichijou; https://mangadex.com/manga/3757

Program exception! Application : Free Manga Downloader Version : 0.9.132.0 Product Version : Win32 Debug Host Machine : Windows 8.1 64-bit FPC Version : 3.0.4 LCL Version : 1.9.0.0 WidgetSet : Win32/Win64 Target CPU-OS : i386-Win32 Build Time : 2018/01/29 15:55:46 Path : C:[Archives][Portable Programs]\fmd_0.9.88.0\fmd.exe Process ID : 6912 MainThread ID : 8272 Thread ID : 8272 Time : 29/01/2018 08:20:00.915 Exception Class : EInvalidPointer Message : Invalid pointer operation $00411E9B $00411D07 $004C732A $004C7478 $004CD29F $004CD182 $00727A0B $00529D03 $0051A459 $0051CF2D $00529C5D $0051B282 $0072A47B $0059E9E4 $00502D38 $0050586F $005063BB

Zannen Jokanbu Black General-san; https://mangadex.com/manga/19657

Program exception! Application : Free Manga Downloader Version : 0.9.132.0 Product Version : Win32 Debug Host Machine : Windows 8.1 64-bit FPC Version : 3.0.4 LCL Version : 1.9.0.0 WidgetSet : Win32/Win64 Target CPU-OS : i386-Win32 Build Time : 2018/01/29 15:55:46 Path : C:[Archives][Portable Programs]\fmd_0.9.88.0\fmd.exe Process ID : 11320 MainThread ID : 5444 Thread ID : 10176 Time : 29/01/2018 08:31:28.816 Sender Class : TGetMangaInfosThread Exception Class : EInvalidPointer Message : Invalid pointer operation $0044307D $0044315E $00442B65 $00412A1E $764E919F $773CA8CB $773CA8A1

Juukinzoku Kanojo; https://mangadex.com/manga/520

Program exception! Application : Free Manga Downloader Version : 0.9.132.0 Product Version : Win32 Debug Host Machine : Windows 8.1 64-bit FPC Version : 3.0.4 LCL Version : 1.9.0.0 WidgetSet : Win32/Win64 Target CPU-OS : i386-Win32 Build Time : 2018/01/29 15:55:46 Path : C:[Archives][Portable Programs]\fmd_0.9.88.0\fmd.exe Process ID : 11272 MainThread ID : 4088 Thread ID : 7848 Time : 29/01/2018 08:44:28.675 Sender Class : TGetMangaInfosThread Exception Class : EInvalidPointer Message : Invalid pointer operation $0044307D $0044315E $00442B65 $00412A1E $764E919F $773CA8CB $773CA8A1

Doing a 'new' install in a different folder returns pretty much the same results, and in the course of that I found that whatever this issue is, it's not confined to MangaDex. I'm getting similar freezes and crashes while trying to open info pages from KissManga, although in this case it seems to be less constrained to a section of the alphabet and instead crashes on the third or fourth one I try to open. It's much less consistent, however, and I haven't yet gotten a program exception message from it.

One other thing that happens when this occurs: the list pane itself bugs out. Either the selected manga itself will vanish from the list right before the program freezes, or ALL visible manga names and counts will vanish and be replaced with a column of "(0)". Could something be going wrong while it's trying to update the chapter counts?

kmvi commented 6 years ago

Ok, I also got a freeze. Steps to reproduce:

  1. Select any site with ~20000 titles (mangafox, mangadex)
  2. Scroll titles list to the end
  3. Click twice on any row
  4. Click twice on any row again 
#0 SYSTEM_$$_WAITFREE_FIXED$PMEMCHUNK_FIXED$POSCHUNK at :0
#1 SYSTEM_$$_SYSFREEMEM_FIXED$PFREELISTS$PMEMCHUNK_FIXED$$LONGWORD at :0
#2 SYSTEM_$$_FINISH_WAITFIXEDLIST$PFREELISTS at :0
#3 SYSTEM_$$_ALLOC_OSCHUNK$PFREELISTS$LONGWORD$LONGWORD$$POINTER at :0
#4 SYSTEM_$$_SYSGETMEM_FIXED$LONGWORD$$POINTER at :0
#5 ?? at :0
#6 SQLDB$_$TCUSTOMSQLQUERY_$__$$_LOADBLOBINTOBUFFER$TFIELDDEF$PBUFBLOBFIELD at :0
#7 BUFDATASET$_$TCUSTOMBUFDATASET_$__$$_LOADBUFFER$PCHAR$$TGETRESULT at :0
#8 BUFDATASET$_$TCUSTOMBUFDATASET_$__$$_GETNEXTPACKET$$LONGINT at :0
#9 ?? at :0
#10 TDBDATAPROCESS__GETVALUE(19574, 1, <error reading variable>, 0x0) at D:\user\src\FMD\baseunits\DBDataProcess.pas:484
#11 TMAINFORM__VTMANGALISTGETTEXT(0x2e736e6f, 0x4aa4fc8, 156620572, 28, 0xa8e15c '0'#2, <error reading variable>) at .\forms\frmMain.pas:4388
#12 TCUSTOMVIRTUALSTRINGTREE__DOGETTEXT(0x94c2030, -1, TTSTATIC, 0x0, <error reading variable>) at VirtualTrees.pas:33450
#13 TCUSTOMVIRTUALSTRINGTREE__GETSTATICTEXT(0x94c2030, -1, <error reading variable>, 0x0) at VirtualTrees.pas:32904
#14 TCUSTOMVIRTUALSTRINGTREE__DOGETNODEEXTRAWIDTH(0x94c2030, -1, 0x18b168, <error reading variable>) at VirtualTrees.pas:33400
#15 TBASEVIRTUALTREE__GETMAXRIGHTEXTEND(<error reading variable>) at VirtualTrees.pas:21588
#16 TBASEVIRTUALTREE__UPDATEHORIZONTALRANGE(<error reading variable>) at VirtualTrees.pas:32056
#17 TBASEVIRTUALTREE__UPDATEHORIZONTALSCROLLBAR(false, <error reading variable>) at VirtualTrees.pas:32067
#18 TBASEVIRTUALTREE__UPDATESCROLLBARS(false, <error reading variable>) at VirtualTrees.pas:32145
#19 TBASEVIRTUALTREE__ENDUPDATE(<error reading variable>) at VirtualTrees.pas:26310
#20 TGETMANGAINFOSTHREAD__MAINTHREADSHOWINFOS(<error reading variable>) at D:\user\src\FMD\baseunits\uGetMangaInfosThread.pas:188
#21 CLASSES_$$_EXECUTETHREADQUEUEENTRY$TThread.PTHREADQUEUEENTRY at :0
#22 CLASSES_$$_CHECKSYNCHRONIZE$LONGINT$$BOOLEAN at :0
#23 TWINDOWPROCHELPER__DOWINDOWPROC(<error reading variable>) at win32callback.inc:1998
#24 WINDOWPROC(3608958, 0, 0, 0) at win32callback.inc:2657
#25 USER32!CallNextHookEx at :0
#26 USER32!CallNextHookEx at :0
#27 USER32!GetMessageW at :0
#28 USER32!DispatchMessageW at :0
#29 TWIN32WIDGETSET__APPPROCESSMESSAGES(<error reading variable>) at win32object.inc:407
#30 TAPPLICATION__HANDLEMESSAGE(<error reading variable>) at .\include\application.inc:1276
#31 TAPPLICATION__RUNLOOP(<error reading variable>) at .\include\application.inc:1418
#32 TWIDGETSET__APPRUN(0x426780 <TAPPLICATION__RUNLOOP>, <error reading variable>) at .\include\interfacebase.inc:54
#33 TAPPLICATION__RUN(<error reading variable>) at .\include\application.inc:1401
#34 main at md.lpr:97
kmvi commented 6 years ago

Hmm, I run FMD on other machine and cannot reproduce this bug...

Tmp341 commented 6 years ago

Program exception! Application : Free Manga Downloader Version : 0.9.132.0 Product Version : Win64 Debug Host Machine : Windows 10 64-bit FPC Version : 3.0.4 LCL Version : 1.9.0.0 WidgetSet : Win32/Win64 Target CPU-OS : x86_64-Win64 Build Time : 2018/01/29 15:54:43 Path : D:\Program Files\Portable\Fmd\fmd.exe Process ID : 8256 MainThread ID : 11416 Thread ID : 11416 Time : 29.01.2018 18:59:32.945 Exception Class : EInvalidPointer Message : Invalid pointer operation $0000000100013DF3 $00000001000EDEC3 $00000001000EB5B0 $00000001000EE013 $00000001000F58E7 $00000001000F56FD $00000001003DEB6D $00000001003D45E1 $0000000100167C9F $00000001001543DA $0000000100157A13 $0000000100167BB7 $00000001003C710F $0000000100010485 $000000010015FC69 $0000000100155535 $00000001003D7900

Tried what @kmvi said. I've updated mangadex through manual update, but it was showing all mangas with zero chapter. Scrolled down to end, randomly double clicked a manga starting with Z. Fmd crashed, with the error above. Opened up again, this time, i right clicked one around S and tried "View manga". It was loading, tried another one and it feezed. Not responding almost 1 minute later. Not showing an error either.

riderkick commented 6 years ago

vtmangalis is not virtual, its read all the data directly from dataset. Since it will be stored in memory anyway. I don't thing it will cause any issue. Maybe I need to make it virtual. But it may doubled the memory used.

karehaqt commented 6 years ago

Seems Mangadex are now using Cloudflare which has stopped it all from working now.

riderkick commented 6 years ago

@kmvi please test. I remove most pointer check. So test with empty cells too. Select, Filter etc. Also test the memory leaks. Use "xx Debug Leaks" build mode.

nille02 commented 6 years ago

@karehaqt did you has a link? i downloaded some chapters right now (Absolute Duo and Overloard) and they are working well.

karehaqt commented 6 years ago

@nille02 I'm using version 0.9.132.0 and these links just stay on loading

https://mangadex.com/manga/13108 https://mangadex.com/manga/9600 https://mangadex.com/manga/1900

I'm just putting these directly into the address bar in FMD and hitting the blue arrow, that's correct yes? I tried Absolute Duo and that doesn't work either. Is there a way to bypass Cloudflare in FMD that I'm missing?

Also, I can no longer update the manga list, again, down to Cloudflare.

kmvi commented 6 years ago

@riderkick trunk version of lazarus adds to image lists something named "BitmapAdv". FMD is unable to start when compiled using lazarus 1.8.0. I removed those "bitmapadv"s and it seems nothing is broken

kmvi commented 6 years ago

@riderkick I think I found another solution, without changing the main form. The idea is to cache current row index between GetValue/GetValueInt calls. Also, lags are slightely reduced while scrolling the list. The bug is also gone. Could you take a look? kmvi/FMD@713d1370

riderkick commented 6 years ago

There is usually a check in SetRecNo to not do any further if its not changed. But, TCustomSqliteDataset.SetRecNo seems to not cache current index. But still, there is a lot of things happen when changing RecNo. When scrolling or select it will change RecNo back and forth a few times (GetText, GetHint, BeforeCellpaint, etc). This is causing overhead to the Dataset. My test show hundreds of RecNo changed in simple scroll (with caching RecNo). I think make it virtual is the best solution for db with a lot of record. Lags when scrolling is gone when all node initialized. Even with millions record. Memory used seems not too high. 1:1 with db file size when fully loaded. This should not be a problem for usual works, node only initialize once when scrolling, so its not fully loaded. Download list and favorite list is kinda virtual, so they don't suffer this issue.

kmvi commented 6 years ago

I tested your version, the bug is gone, and no memory leaks detected. But there is another bug:

  1. Select any N >= 1 rows and delete it
  2. Last N rows disappears from the list (but actually the selected rows are removed from the db)
riderkick commented 6 years ago

@kmvi I implement your code with some changes. I believe using next prior is faster. https://github.com/riderkick/FMD/commit/4058a67ad67630ba2a7edefbe8851f29dedbfaed

riderkick commented 6 years ago

@kmvi https://github.com/riderkick/FMD/commit/3ade68bf3534f358aadca40f08f70a3d1540a2aa should fix the issue. Please test.

kmvi commented 6 years ago

@riderkick looks good for me 👍

karehaqt commented 6 years ago

Will the next update get around Cloudflare on Mangadex then?

nille02 commented 6 years ago

@karehaqt even with your links its working for me. maybe its only a region limited activation of cloudflare.

karehaqt commented 6 years ago

@nille02 If thats the case then this software is no use to me now 😞

nille02 commented 6 years ago

maybe @riderkick or @kmvi can tell more about it. cloudflare is also used by some other sites that are supported

karehaqt commented 6 years ago

Looks like MangaDLR on Android works fine with Mangadex so I can use that for now.

karehaqt commented 6 years ago

Decided to make a new FMD folder to start from scratch and when trying to view manga info for any title on Mangadex, I'm getting the following Error. It works for any non-Mangadex based title.

riderkick commented 6 years ago

@kmvi I added cloudflare check to all modules. https://github.com/riderkick/FMD/commit/33d2a7236ae411a5a96ea9b43356acf98894166f Please check affected website: FoOlSlide, GameofScanlation, KissManga, MangaAe, MangaGo, MyReadingManga, Tapas, Tumangaonline. I intercept THTTPSendThread.GET/POST/HEAD/XHR to always check for cloudflare's challenge. It will merge the cookies with cached cloudflare's cookies. So check also modules that change the cookies at runtime. It may affect those modules. Especially the one that use account. We can toggle CloudflareEnabled when adding the modules. The default is true for now. To see its performance.

riderkick commented 6 years ago

Other users can test and report. fmd_win32.7z.txt

kmvi commented 6 years ago

@riderkick should GetPageNumber also checks CF (also see comment in bc4b357c)?

--- a/baseunits/WebsiteModules.pas
+++ b/baseunits/WebsiteModules.pas
@@ -593,9 +593,12 @@ function TWebsiteModules.GetPageNumber(const DownloadThread: TDownloadThread;
 begin
   Result := False;
   if (ModuleId < 0) or (ModuleId >= FModuleList.Count) then Exit;
-  if Assigned(TModuleContainer(FModuleList[ModuleId]).OnGetPageNumber) then
-    Result := TModuleContainer(FModuleList[ModuleId]).OnGetPageNumber(
-      DownloadThread, AURL, TModuleContainer(FModuleList[ModuleId]));
+  with TModuleContainer(FModuleList[ModuleId]) do
+    if Assigned(OnGetPageNumber) then
+    begin
+      CheckCloudflareEnabled(DownloadThread.FHTTP);
+      Result := OnGetPageNumber(DownloadThread, AURL, TModuleContainer(FModuleList[ModuleId]));
+    end;
 end;

 function TWebsiteModules.GetPageNumber(const DownloadThread: TDownloadThread;

I also suggest to set Referer header before downloading cover image (for example, manga go requires it):

--- a/baseunits/uGetMangaInfosThread.pas
+++ b/baseunits/uGetMangaInfosThread.pas
@@ -150,7 +150,8 @@ begin
       // If there's cover then we will load it to the TPicture component.
       if OptionEnableLoadCover and (Trim(FInfo.mangaInfo.coverLink) <> '') then
         try
-          FInfo.FHTTP.Document.Clear;
+          FInfo.FHTTP.Reset;
+          FInfo.FHTTP.Headers.Values['Referer'] := FInfo.mangaInfo.url;
           if FInfo.FHTTP.GET(FInfo.mangaInfo.coverLink) then
           begin
             FCover.LoadFromStream(FInfo.FHTTP.Document);
riderkick commented 6 years ago

@kmvi Yes, I forgot to push my commit. Every event with THTTPSendThread should check for cloudflare. https://github.com/riderkick/FMD/commit/eeca711eb1911147700bfd781fa6d3981a29d9c7 Download cover image share the same THTTPSendThread with GetInfo. So you can set the referer there. https://github.com/riderkick/FMD/commit/441a00c327943b2c5c02387a7fd1998b8405c84c

kmvi commented 6 years ago

There is also problem here. If OnDownloadImage is not set then download request will be sent without CF cookies.

riderkick commented 6 years ago

Well, I think the best way is to set cf hook right after creation of THTTPSendThread. I remember looking up a module id for every thread. Maybe we can set it there.

riderkick commented 6 years ago

@kmvi please test https://github.com/riderkick/FMD/commit/cce1979fdab41b0caa675898eb8b6fcb9c8eb4ca .

riderkick commented 6 years ago

fmd_win32.7z.txt

kmvi commented 6 years ago

@riderkick 👍

kmvi commented 6 years ago

@riderkick I was fixing 'emailprotected' in Mangadex titles (e666179) and found the following:

  1. Set filter to "email"
  2. Click twice on any row in manga list
  3. Text of the clicked row is replaced with the text of next (?) row
  4. Some rows are replaced with '(0)'

untitled untitled2

riderkick commented 6 years ago

Cloudflare content replace @ character with javascript. You can try to find the title on other tag. Usually cloudflare content has <meta title=>.