[BETA] Cloudflare Bypassing

[SDXC]

THIS IS ONLY FOR PEOPLE WHO WANT TO TEST EXPERIMENTAL FEATURES! If you want to help in finding bugs and by providing feedback, you can download this beta release of FMD (based on 0.9.163.0). Backup your old FMD Installation!

About this beta release

This beta package implements a new approach on how to bypass the Cloudflare "Please wait 5 seconds" page, that in many cases blocks FMD.

How it works

When preparing a request FMD will check, if the website is protected by Cloudflare (even if it is not active right now). If CF has been detected, FMD will check in the advanced settings, if there already is a cf_clearance cookie and a user agent defined. If not, FMD will call a 3rd party executable called cf_bypass.exe which tries to solve the CF challenge and exports the resulting cookies and user agent into a text file in the config folder. FMD reads the content and loads the values into the advanced settings. After that, FMD will retry to open the web connection by using the cookie and user agent.

Limitations

If you have manually set a cf_clearance cookie in THTTP, then FMD won't create a new cookie and probably fails if the cookie isn't valid anymore.

Combination of cookies

You can combine cookie strings by setting additional cookies in the THTTP-SubTree of the advanced settings (last entry mustn't have a tailing ";" at the end).

Manual workaround still possible

If you have problems with a website that uses CF and the bypass doesn't automatically work, you can still try by doing the manual workaround you had to up until now.

Cookie timespan

Everytime you close FMD, the cookie and user agent in TDynHTTP are being cleared. FMD doesn't check the timespan of the cookie validity, so if you have problems after some time, just restart FMD and retry.

About your tasks for this beta test

Please try to use this beta version in your everyday routines and report any bugs you encounter. Also if you think there could be some improvement, please let me know.

DON'T OPEN ISSUES FOR THIS BETA, REPORT EVERYTHING IN THIS ISSUE!

Download

fmd_0.9.163.1_Win64.zip fmd_0.9.163.1.zip

[ghost]

nice, Mangadex Cloudflare bypass fine and download working

[Slasar41]

If I pause the downloads, change the network connection, then resuming it without restarting FMD, the next chapter/manga on queue will fail. https://i.imgur.com/FXcRWTU.gifv

Maybe this options should became 'True' to avoid confusion.

[SDXC]

@Slasar41 That is more or less an expected behavior. I mean, did it ever work with the manual workaround, when you paused and changed the network connection?

DynHTTP is not part of the "user" settings, but the possibility should be given to clear these settings if there is a need to.

For example if I add proxy settings but i don't want to use them right now, I want to be able to disable them and still use the cf_bypass. If cf_bypass would enable the settings automatically, I would always have problems with settings I set before, etc.

So no, this won't be implemented like that. It's easier for users to understand that DynHTTP is not part of the user settings than if they can't use their settings anymore like they want to, etc.

[Slasar41]

I thought it will be fixed with the new implementation. 😅

So it should be separated, maybe adding a line between ProxyPassword and DynHTTP.

[Slasar41]

FMD freeze for about 3 seconds repeatedly in one minute and made the HDD's active time to spike.

[SDXC]

@Slasar41 Regarding the settings: Not possible without tinkering on things that shouldn't be tinkered with. It's a generated control, so I don't have much freedom with it. And it's also not really a necessity. It's there and it (theoretically) works. If I come up with a better idea, I'll let you know.

What exactly was FMD doing while you had that spike?

[Slasar41]

Okay, thanks.

Doing nothing or while downloading, so If I have constant transfer rate 500KB/s, it'll freeze at that speed then will became 1.5MB/s then go down to 500KB/s again. This also happened from when I still use riderkick version on my laptop with SSD. I thought it might be a laptop problem but it also happened on my desktop (I never use FMD on my desktop 😅).

[molitar]

Well I never used MangaDex so it was the perfect one to test with but I am not getting any cookies at all.

[SDXC]

@Slasar41 depending on the website it could be that's just some slow connections right now. even MangaDex is really slow atm.

@molitar Please don't try too much on MangaDex right now. If CF asks for recaptchas then you also won't get any cookies. You can test it on japscan.to if you just want to test.

[Slasar41]

@SDXC Sorry, it's not in one minute but in two minutes. Just FMD that froze up. See https://i.imgur.com/lGkVpra.gifv

[Mereck13579]

@molitar Please don't try too much on MangaDex right now. If CF asks for recaptchas then you also won't get any cookies. You can test it on japscan.to if you just want to test.

Tried both MangaDex and Japscan.to, neither worked (fyi I'm on a fresh install). Didn't get any recaptchas challenge when trying to connect with my browser.

[Josexv1]

Poeple in the forums released a software to download from MD and they setup some limitations which could help avoiding bans, maybe if we can implement it in the MD module it will improve our chances

[Slasar41]

For MangaDex, FMD doesn't got the cookies automatically so I put manual workaround and it works (needs to solve the captcha). For Japscan, FMD does got the cookies automatically but it's still does't works even if later I put manual workaround.

[molitar]

Japscans worked perfectly fine for me including testing a download.

[SDXC]

@molitar I think some websites have problems with some user agents. There are some mobile user agents, which for example JapScan can't handle or handles differently. In my case I could load the manga info, but not the chapters when I had a mobile user agent.

[devprojectid]

error

C:\FMD\cf_bypass.exe Traceback (most recent call last): File "cf_bypass.py", line 3, in <module> IndexError: list index out of range [13196] Failed to execute script cf_bypass

[SDXC]

@devprojectid Thanks but I need to know what FMD was doing at the time of the error, where this error was reported and for which website it happened.

[devprojectid]

@devprojectid Thanks but I need to know what FMD was doing at the time of the error, where this error was reported and for which website it happened.

it fixed now, it's comeout when cloudflare solve captcha show up

[Tmp341]

That exe can't detect recaptcha issue? fmd can't bypass cf now because recaptcha issue occured kinda error show up maybe.

[molitar]

Strange behavior with NineManga. It was reporting 0 chapters even when I tried update manga list. I had to restart FMD before it started to read chapters again for download.

[molitar]

One thing I noticed is that I have to go back into the advanced and clear out the cookies from the DynHTTP and HTTP then restart FMD to get it to work again with Kissmanga. The other option is to clear them both then hit apply and try again. So it appears right now that DynHTTP is also populating in the HTTP so the update is ignored when cookie is no longer valid.

[molitar]

Bug: Almost all the lua show date of 12/30/1899. Does not happen with the release version backup folder.

only a very of them show the correct date.

[jsoudeer]

Hello,

Japscan finally works for me. I change the url, using manga instead of mangas. (ex : https://www.japscan.to/manga/the-promised-neverland/ ) How can i change the db?

Johan

[molitar]

The cf workaround will stop looking for new cookies until I manually go into the DynHTTP cookies and delete them all.. Then I can go download the website info like for kissmanga and it will work. It seems if it is not working if there is already a DynHTTP it will not refresh.

[SDXC]

DynHTTP will now automatically be renewed, if the website couldn't load because of Cloudflare.

With this, I released a new version of FMD including the cf_bypass. This issue will be closed. You can report problems the regular way now, by opening new issues.