Closed archet-VIII closed 6 years ago
i have non of your problems, but only problems with downloading. E.g. he download only Page 1,2,3,34 from Tsuki Tsuki Chapter 1. The other 4 Chapters got similar problems. https://mangadex.com/manga/775
Similar issues i got with this Series.
https://mangadex.com/manga/4280 https://mangadex.com/manga/9750
I'm also having problems downloading manga from mangadex. it downloaded fine for a while but then it started giving me errors. I kept pressing retry but then I realized nothing was downloading even though FMD was using all my bandwidth. I was able to download 1.13 GB before it stopped working. my guess is that it has a bandwidth limit per IP or something along those lines.
I have the first issue which downloading from server gives a 404, in fact I had it since the previous update but I was lazy to report it since I can just use the update manga list.
sorry. I don't know what to do with that link(?) you gave me @kmvi I don't find a download button for that file you linked
@alkasul the reason is, that there is no prebuild index online. you can ignore this. Update the List Create a new List that is complete.
i have non of your problems, but only problems with downloading. E.g. he download only Page 1,2,3,34 from Tsuki Tsuki Chapter 1. The other 4 Chapters got similar problems.
There's some persistent issues on MangaDex's side around how it handles the uploaded archive files. Aside from the first couple days worth of uploads completely breaking and random chapters only having some of their images uploading more recently, they've had various problems getting it to work with archives that had both pngs and jpgs in them. They initially were not even able to be uploaded because the reader couldn't parse them, and while they have since gotten them mostly working on the site itself I would guess that there's still some issue with them that causes problems for FMD, as the jpg images from that chapter you refer to are the only ones that downloaded. The chapters I was able to download were all uploaded solely as jpgs, so I hadn't run into that.
@archet-VIII just finished updating the list (~5 hours), no errors happened. Db attached Mangadex.zip @punpun3 this is a correction for future release
When i try to download mangadex, i get an error message saying file not found
@kmvi Afraid I'm still getting the same result, even with the .db file you provided. Trying to open manga info pages in the later half of the list consistently freezes up the program. Got another 'program exception' message in the process.
Program exception! Application : Free Manga Downloader Version : 0.9.131.0 Product Version : Win32 Debug Host Machine : Windows 8.1 64-bit FPC Version : 3.0.4 LCL Version : 1.9.0.0 WidgetSet : Win32/Win64 Target CPU-OS : i386-Win32 Build Time : 2018/01/27 02:26:03 Path : C:[Archives][Portable Programs]\fmd_0.9.88.0\fmd.exe Process ID : 11632 MainThread ID : 9808 Thread ID : 9808 Time : 28/01/2018 05:58:21.897 Exception Class : EInvalidPointer Message : Invalid pointer operation $00411E9B $00411D07 $004C709A $004C71E8 $004CD00F $004CCEF2 $0071F22B $00528903 $00519059 $0051BB2D $0052885D $00519E82 $00721C9B $00596404 $00501938 $0050446F $00504FBB
@kmvi give me your sf username. I'll add you to admin group so you can add/update db files.
@riderkick https://sf.net/u/kmvi
Done. Data files is in '/home/pfs/project/newfmd/data'. Just use sftp to update. Use 7z with highest compression level if possible.
@archet-VIII maybe you are right, but on the website all pages are working.
@archet-VIII you should post the url of series you try to load. I don't see any reasonable line in mangadex's getinfo source that can cause EInvaldPointer. Invalid pointer usually causing by a code that try to access uninitialized/freed object.
Just to say I'm also finding issues with downloading from Mangadex, it's like the download gets a little way in then it starts again and does that over and over again.
With the new Program Version the downloads are working fine for me.
Nice @nille02 I'll give it a try when I get home from work tonight :+1:
@riderkick I first encountered issues when trying to open Zannen Jokanbu Black General-san; https://mangadex.com/manga/19657 which was actually the first thing I did after the list had finally been built. The next couple tries eventually produced that first exception error I posted, and subsequent attempts picking random entries on the list produced the second.
I updated to the new version this morning as was initially able to open Zannen Jokanbu's info without any problems, but upon trying to open a second random manga in the Zs the same issues immediately reoccurred. I can apparently freely open manga in the first third of the list, but trying to open anything past J or so causes crashes. Sometimes with the program exception message, but more often without. Here's some of the ones I picked up in the last little while:
Youkai Apaato no Yuuga na Nichijou; https://mangadex.com/manga/3757
Program exception! Application : Free Manga Downloader Version : 0.9.132.0 Product Version : Win32 Debug Host Machine : Windows 8.1 64-bit FPC Version : 3.0.4 LCL Version : 1.9.0.0 WidgetSet : Win32/Win64 Target CPU-OS : i386-Win32 Build Time : 2018/01/29 15:55:46 Path : C:[Archives][Portable Programs]\fmd_0.9.88.0\fmd.exe Process ID : 6912 MainThread ID : 8272 Thread ID : 8272 Time : 29/01/2018 08:20:00.915 Exception Class : EInvalidPointer Message : Invalid pointer operation $00411E9B $00411D07 $004C732A $004C7478 $004CD29F $004CD182 $00727A0B $00529D03 $0051A459 $0051CF2D $00529C5D $0051B282 $0072A47B $0059E9E4 $00502D38 $0050586F $005063BB
Zannen Jokanbu Black General-san; https://mangadex.com/manga/19657
Program exception! Application : Free Manga Downloader Version : 0.9.132.0 Product Version : Win32 Debug Host Machine : Windows 8.1 64-bit FPC Version : 3.0.4 LCL Version : 1.9.0.0 WidgetSet : Win32/Win64 Target CPU-OS : i386-Win32 Build Time : 2018/01/29 15:55:46 Path : C:[Archives][Portable Programs]\fmd_0.9.88.0\fmd.exe Process ID : 11320 MainThread ID : 5444 Thread ID : 10176 Time : 29/01/2018 08:31:28.816 Sender Class : TGetMangaInfosThread Exception Class : EInvalidPointer Message : Invalid pointer operation $0044307D $0044315E $00442B65 $00412A1E $764E919F $773CA8CB $773CA8A1
Juukinzoku Kanojo; https://mangadex.com/manga/520
Program exception! Application : Free Manga Downloader Version : 0.9.132.0 Product Version : Win32 Debug Host Machine : Windows 8.1 64-bit FPC Version : 3.0.4 LCL Version : 1.9.0.0 WidgetSet : Win32/Win64 Target CPU-OS : i386-Win32 Build Time : 2018/01/29 15:55:46 Path : C:[Archives][Portable Programs]\fmd_0.9.88.0\fmd.exe Process ID : 11272 MainThread ID : 4088 Thread ID : 7848 Time : 29/01/2018 08:44:28.675 Sender Class : TGetMangaInfosThread Exception Class : EInvalidPointer Message : Invalid pointer operation $0044307D $0044315E $00442B65 $00412A1E $764E919F $773CA8CB $773CA8A1
Doing a 'new' install in a different folder returns pretty much the same results, and in the course of that I found that whatever this issue is, it's not confined to MangaDex. I'm getting similar freezes and crashes while trying to open info pages from KissManga, although in this case it seems to be less constrained to a section of the alphabet and instead crashes on the third or fourth one I try to open. It's much less consistent, however, and I haven't yet gotten a program exception message from it.
One other thing that happens when this occurs: the list pane itself bugs out. Either the selected manga itself will vanish from the list right before the program freezes, or ALL visible manga names and counts will vanish and be replaced with a column of "(0)". Could something be going wrong while it's trying to update the chapter counts?
Ok, I also got a freeze. Steps to reproduce:
#0 SYSTEM_$$_WAITFREE_FIXED$PMEMCHUNK_FIXED$POSCHUNK at :0
#1 SYSTEM_$$_SYSFREEMEM_FIXED$PFREELISTS$PMEMCHUNK_FIXED$$LONGWORD at :0
#2 SYSTEM_$$_FINISH_WAITFIXEDLIST$PFREELISTS at :0
#3 SYSTEM_$$_ALLOC_OSCHUNK$PFREELISTS$LONGWORD$LONGWORD$$POINTER at :0
#4 SYSTEM_$$_SYSGETMEM_FIXED$LONGWORD$$POINTER at :0
#5 ?? at :0
#6 SQLDB$_$TCUSTOMSQLQUERY_$__$$_LOADBLOBINTOBUFFER$TFIELDDEF$PBUFBLOBFIELD at :0
#7 BUFDATASET$_$TCUSTOMBUFDATASET_$__$$_LOADBUFFER$PCHAR$$TGETRESULT at :0
#8 BUFDATASET$_$TCUSTOMBUFDATASET_$__$$_GETNEXTPACKET$$LONGINT at :0
#9 ?? at :0
#10 TDBDATAPROCESS__GETVALUE(19574, 1, <error reading variable>, 0x0) at D:\user\src\FMD\baseunits\DBDataProcess.pas:484
#11 TMAINFORM__VTMANGALISTGETTEXT(0x2e736e6f, 0x4aa4fc8, 156620572, 28, 0xa8e15c '0'#2, <error reading variable>) at .\forms\frmMain.pas:4388
#12 TCUSTOMVIRTUALSTRINGTREE__DOGETTEXT(0x94c2030, -1, TTSTATIC, 0x0, <error reading variable>) at VirtualTrees.pas:33450
#13 TCUSTOMVIRTUALSTRINGTREE__GETSTATICTEXT(0x94c2030, -1, <error reading variable>, 0x0) at VirtualTrees.pas:32904
#14 TCUSTOMVIRTUALSTRINGTREE__DOGETNODEEXTRAWIDTH(0x94c2030, -1, 0x18b168, <error reading variable>) at VirtualTrees.pas:33400
#15 TBASEVIRTUALTREE__GETMAXRIGHTEXTEND(<error reading variable>) at VirtualTrees.pas:21588
#16 TBASEVIRTUALTREE__UPDATEHORIZONTALRANGE(<error reading variable>) at VirtualTrees.pas:32056
#17 TBASEVIRTUALTREE__UPDATEHORIZONTALSCROLLBAR(false, <error reading variable>) at VirtualTrees.pas:32067
#18 TBASEVIRTUALTREE__UPDATESCROLLBARS(false, <error reading variable>) at VirtualTrees.pas:32145
#19 TBASEVIRTUALTREE__ENDUPDATE(<error reading variable>) at VirtualTrees.pas:26310
#20 TGETMANGAINFOSTHREAD__MAINTHREADSHOWINFOS(<error reading variable>) at D:\user\src\FMD\baseunits\uGetMangaInfosThread.pas:188
#21 CLASSES_$$_EXECUTETHREADQUEUEENTRY$TThread.PTHREADQUEUEENTRY at :0
#22 CLASSES_$$_CHECKSYNCHRONIZE$LONGINT$$BOOLEAN at :0
#23 TWINDOWPROCHELPER__DOWINDOWPROC(<error reading variable>) at win32callback.inc:1998
#24 WINDOWPROC(3608958, 0, 0, 0) at win32callback.inc:2657
#25 USER32!CallNextHookEx at :0
#26 USER32!CallNextHookEx at :0
#27 USER32!GetMessageW at :0
#28 USER32!DispatchMessageW at :0
#29 TWIN32WIDGETSET__APPPROCESSMESSAGES(<error reading variable>) at win32object.inc:407
#30 TAPPLICATION__HANDLEMESSAGE(<error reading variable>) at .\include\application.inc:1276
#31 TAPPLICATION__RUNLOOP(<error reading variable>) at .\include\application.inc:1418
#32 TWIDGETSET__APPRUN(0x426780 <TAPPLICATION__RUNLOOP>, <error reading variable>) at .\include\interfacebase.inc:54
#33 TAPPLICATION__RUN(<error reading variable>) at .\include\application.inc:1401
#34 main at md.lpr:97
Hmm, I run FMD on other machine and cannot reproduce this bug...
Program exception! Application : Free Manga Downloader Version : 0.9.132.0 Product Version : Win64 Debug Host Machine : Windows 10 64-bit FPC Version : 3.0.4 LCL Version : 1.9.0.0 WidgetSet : Win32/Win64 Target CPU-OS : x86_64-Win64 Build Time : 2018/01/29 15:54:43 Path : D:\Program Files\Portable\Fmd\fmd.exe Process ID : 8256 MainThread ID : 11416 Thread ID : 11416 Time : 29.01.2018 18:59:32.945 Exception Class : EInvalidPointer Message : Invalid pointer operation $0000000100013DF3 $00000001000EDEC3 $00000001000EB5B0 $00000001000EE013 $00000001000F58E7 $00000001000F56FD $00000001003DEB6D $00000001003D45E1 $0000000100167C9F $00000001001543DA $0000000100157A13 $0000000100167BB7 $00000001003C710F $0000000100010485 $000000010015FC69 $0000000100155535 $00000001003D7900
Tried what @kmvi said. I've updated mangadex through manual update, but it was showing all mangas with zero chapter. Scrolled down to end, randomly double clicked a manga starting with Z. Fmd crashed, with the error above. Opened up again, this time, i right clicked one around S and tried "View manga". It was loading, tried another one and it feezed. Not responding almost 1 minute later. Not showing an error either.
vtmangalis is not virtual, its read all the data directly from dataset. Since it will be stored in memory anyway. I don't thing it will cause any issue. Maybe I need to make it virtual. But it may doubled the memory used.
Seems Mangadex are now using Cloudflare which has stopped it all from working now.
@kmvi please test. I remove most pointer check. So test with empty cells too. Select, Filter etc. Also test the memory leaks. Use "xx Debug Leaks" build mode.
@karehaqt did you has a link? i downloaded some chapters right now (Absolute Duo and Overloard) and they are working well.
@nille02 I'm using version 0.9.132.0 and these links just stay on loading
https://mangadex.com/manga/13108 https://mangadex.com/manga/9600 https://mangadex.com/manga/1900
I'm just putting these directly into the address bar in FMD and hitting the blue arrow, that's correct yes? I tried Absolute Duo and that doesn't work either. Is there a way to bypass Cloudflare in FMD that I'm missing?
Also, I can no longer update the manga list, again, down to Cloudflare.
@riderkick trunk version of lazarus adds to image lists something named "BitmapAdv". FMD is unable to start when compiled using lazarus 1.8.0. I removed those "bitmapadv"s and it seems nothing is broken
@riderkick I think I found another solution, without changing the main form. The idea is to cache current row index between GetValue/GetValueInt calls. Also, lags are slightely reduced while scrolling the list. The bug is also gone. Could you take a look? kmvi/FMD@713d1370
There is usually a check in SetRecNo
to not do any further if its not changed. But, TCustomSqliteDataset.SetRecNo
seems to not cache current index.
But still, there is a lot of things happen when changing RecNo. When scrolling or select it will change RecNo back and forth a few times (GetText, GetHint, BeforeCellpaint, etc). This is causing overhead to the Dataset. My test show hundreds of RecNo changed in simple scroll (with caching RecNo).
I think make it virtual is the best solution for db with a lot of record. Lags when scrolling is gone when all node initialized. Even with millions record. Memory used seems not too high. 1:1 with db file size when fully loaded. This should not be a problem for usual works, node only initialize once when scrolling, so its not fully loaded.
Download list and favorite list is kinda virtual, so they don't suffer this issue.
I tested your version, the bug is gone, and no memory leaks detected. But there is another bug:
@kmvi I implement your code with some changes. I believe using next prior is faster. https://github.com/riderkick/FMD/commit/4058a67ad67630ba2a7edefbe8851f29dedbfaed
@kmvi https://github.com/riderkick/FMD/commit/3ade68bf3534f358aadca40f08f70a3d1540a2aa should fix the issue. Please test.
@riderkick looks good for me 👍
Will the next update get around Cloudflare on Mangadex then?
@karehaqt even with your links its working for me. maybe its only a region limited activation of cloudflare.
@nille02 If thats the case then this software is no use to me now 😞
maybe @riderkick or @kmvi can tell more about it. cloudflare is also used by some other sites that are supported
Looks like MangaDLR on Android works fine with Mangadex so I can use that for now.
Decided to make a new FMD folder to start from scratch and when trying to view manga info for any title on Mangadex, I'm getting the following Error. It works for any non-Mangadex based title.
@kmvi I added cloudflare check to all modules. https://github.com/riderkick/FMD/commit/33d2a7236ae411a5a96ea9b43356acf98894166f
Please check affected website: FoOlSlide, GameofScanlation, KissManga, MangaAe, MangaGo, MyReadingManga, Tapas, Tumangaonline.
I intercept THTTPSendThread.GET/POST/HEAD/XHR
to always check for cloudflare's challenge. It will merge the cookies with cached cloudflare's cookies. So check also modules that change the cookies at runtime. It may affect those modules. Especially the one that use account.
We can toggle CloudflareEnabled
when adding the modules. The default is true for now. To see its performance.
Other users can test and report. fmd_win32.7z.txt
@riderkick should GetPageNumber
also checks CF (also see comment in bc4b357c)?
--- a/baseunits/WebsiteModules.pas
+++ b/baseunits/WebsiteModules.pas
@@ -593,9 +593,12 @@ function TWebsiteModules.GetPageNumber(const DownloadThread: TDownloadThread;
begin
Result := False;
if (ModuleId < 0) or (ModuleId >= FModuleList.Count) then Exit;
- if Assigned(TModuleContainer(FModuleList[ModuleId]).OnGetPageNumber) then
- Result := TModuleContainer(FModuleList[ModuleId]).OnGetPageNumber(
- DownloadThread, AURL, TModuleContainer(FModuleList[ModuleId]));
+ with TModuleContainer(FModuleList[ModuleId]) do
+ if Assigned(OnGetPageNumber) then
+ begin
+ CheckCloudflareEnabled(DownloadThread.FHTTP);
+ Result := OnGetPageNumber(DownloadThread, AURL, TModuleContainer(FModuleList[ModuleId]));
+ end;
end;
function TWebsiteModules.GetPageNumber(const DownloadThread: TDownloadThread;
I also suggest to set Referer header before downloading cover image (for example, manga go requires it):
--- a/baseunits/uGetMangaInfosThread.pas
+++ b/baseunits/uGetMangaInfosThread.pas
@@ -150,7 +150,8 @@ begin
// If there's cover then we will load it to the TPicture component.
if OptionEnableLoadCover and (Trim(FInfo.mangaInfo.coverLink) <> '') then
try
- FInfo.FHTTP.Document.Clear;
+ FInfo.FHTTP.Reset;
+ FInfo.FHTTP.Headers.Values['Referer'] := FInfo.mangaInfo.url;
if FInfo.FHTTP.GET(FInfo.mangaInfo.coverLink) then
begin
FCover.LoadFromStream(FInfo.FHTTP.Document);
@kmvi Yes, I forgot to push my commit. Every event with THTTPSendThread
should check for cloudflare. https://github.com/riderkick/FMD/commit/eeca711eb1911147700bfd781fa6d3981a29d9c7
Download cover image share the same THTTPSendThread
with GetInfo
. So you can set the referer there. https://github.com/riderkick/FMD/commit/441a00c327943b2c5c02387a7fd1998b8405c84c
There is also problem here. If OnDownloadImage is not set then download request will be sent without CF cookies.
Well, I think the best way is to set cf hook right after creation of THTTPSendThread
. I remember looking up a module id for every thread. Maybe we can set it there.
@kmvi please test https://github.com/riderkick/FMD/commit/cce1979fdab41b0caa675898eb8b6fcb9c8eb4ca .
@riderkick 👍
@riderkick I was fixing 'emailprotected' in Mangadex titles (e666179) and found the following:
Cloudflare content replace @ character with javascript. You can try to find the title on other tag. Usually cloudflare content has <meta title=>.
Upgraded to the newest version to try it out on the newest site, but it's not going well.
First, updating the manga list 'from server' returns a 404. Using the first update option works, although with nearly 20k manga on the list it does take quite a while.
Second, program can freeze or crash when loading the manga info. This particularly seems to happen when trying to load manga from past the first third or so of the list; I first ran into it when trying to load one that started with a Z, and later when testing it letter by letter ran into it again in the Ks. The first instance it froze the program and monopolized around 50% of the CPU for ~15 minutes before I terminated it, the second it just froze while using negligible resources.
The third time Windows came up with its 'unresponsive program' box twice, first over the manga list pane and then over the info pane. Clicking past those brought up a 'program exception' dialog with the following in the 'more info' area:
The fourth attempt just brought up the 'unresponsive program' boxes without the 'exception' dialog.
For the manga that do load, download does seem to be functioning properly. There are some chapters where the images don't download, but that's a known problem on MangaDex's side affecting the first ~5000 or so files uploaded to the site. The more recent ones work fine.