search

dteviot / WebToEpub

A simple Chrome (and Firefox) Extension that converts Web Novels (and other web pages) into an EPUB.
Other
740 stars 140 forks source link

Version 0.0.171 Now Fails on Foxaholic + Foxaholic18 for 403 Error #1447

Closed yuyu-cloud closed 2 months ago

yuyu-cloud commented 3 months ago

Describe the bug After updating to the latest experimental build of the extension (found on this repository), the extension no longer works on both Foxaholic and Foxaholic18 sites.

To Reproduce Steps to reproduce the behavior:

  1. Go to link: https://18.foxaholic.com/novel/political-marriage-abo/
  2. Click on Webtoepub
  3. See this error: image

Click OK. Tried to reload chapter 1 to test, no cloudflare verification showed so could not satisfy requirement.

See second error:

image

  1. Webtoepub fails after selecting "Retry" and the "Save As" box is prompted but no chapters were downloaded.

Expected behavior From a previous issue, Webtoepub should be able to bypass this problem, thereby pulling the content without seeing any error dialogue box about the 403 cloudflare issue.

Additional context The version I was using before for Webtoepub was 0.0.160 and there was no problem at all when encountering cloudflare protection for any sites. Actually, the strange thing was, after generating the epub using the old experimental build of 0.0.160, and then trying again with the new version 0.0.171, Webtoepub was able to proceed and generate the epub containing all chapters successfully. Though I'm clueless on why it suddenly worked after using the old version first before the new build.

gamebeaker commented 3 months ago

as reference #1445 bug: 18.foxaholic.com is the hostname but the domain for the cookie is .foxaholic.com

gamebeaker commented 3 months ago

@yuyu-cloud what is your chrome version? I think you have an old chrome version and because of this you didn't have any problems with 0.0.0.160 in the newest chrome version is a bug and to mitigate this bug code got changed in WebToEpub. This new code is apparently buggy thanks for the feedback.

gamebeaker commented 3 months ago

@yuyu-cloud Test versions for Firefox and Chrome have been uploaded to https://github.com/dteviot/WebToEpub/releases/tag/developer-build. Pick the one suitable for you, follow the "How to install from Source (for people who are not developers)" instructions at https://github.com/dteviot/WebToEpub/tree/ExperimentalTabMode#user-content-how-to-install-from-source-for-people-who-are-not-developers and let me know how it goes. Tested with:

gamebeaker commented 3 months ago

@MoonShadowIllusion i deleted your Comment because virustotal flags it and you didn't provide more info what this exe should be. https://www.virustotal.com/gui/file/b127de888f09ce23937c12b7fccfa47a8f48312b0e43eb59b6243f665c6d366a image

MoonShadowIllusion commented 3 months ago

@MoonShadowIllusion i deleted your Comment because virustotal flags it and you didn't provide more info what this exe should be. https://www.virustotal.com/gui/file/b127de888f09ce23937c12b7fccfa47a8f48312b0e43eb59b6243f665c6d366a image

sorry, my account seem to have been hacked

yuyu-cloud commented 3 months ago

@gamebeaker

Hallo!! Thank you sooooo much for your prompt response and speedy fix, it worked like a charm and fixed the problem better than ever~ And you were correct about my Chrome haha, I wasn't using the latest build, but just updated it :)

dteviot commented 3 months ago

@yuyu-cloud Reopen, so know to notify you when version in Chrome store is updated.

ssj4maiko commented 2 months ago

Reopening as I'm getting the same error on Foxaholic (not 18.foxaholic, don't have anything I read from that one, so I can't say for sure.)

I installed dev build 177 (the last one available at the time). This is the page I'm running it from: https://www.foxaholic.com/novel/i-became-the-strongest-with-the-failure-frame%e3%80%90abnormal-state-skill%e3%80%91as-i-devastated-everything/

Try to run it here with the lastest ~30 chapters (never had problems before), and I'm now getting the mentioned error and it's stuck there.

So I don't know if only 18.foxaholic had been fixed, or if something happened that broke it again.

Obs: I'm running Firefox if that makes a difference. Version 129.0a1 (Nightly)

dteviot commented 2 months ago

@ssj4maiko

I just tried the following chapters from supplied link

They downloaded with no problem. Note, I did have the Cloudflare "Are you a human" appear in my browser before I attempted the download.

ssj4maiko commented 2 months ago

@ssj4maiko

I just tried the following chapters from supplied link

* [0.1 - Volume 1 Illustrations (Contains Spoilers)](https://www.foxaholic.com/novel/i-became-the-strongest-with-the-failure-frame%e3%80%90abnormal-state-skill%e3%80%91as-i-devastated-everything/0-1_2/)

* https://www.foxaholic.com/novel/i-became-the-strongest-with-the-failure-frame%e3%80%90abnormal-state-skill%e3%80%91as-i-devastated-everything/0-2_2/

* https://www.foxaholic.com/novel/i-became-the-strongest-with-the-failure-frame%e3%80%90abnormal-state-skill%e3%80%91as-i-devastated-everything/394/

They downloaded with no problem. Note, I did have the Cloudflare "Are you a human" appear in my browser before I attempted the download.

thank you for the reply.

I was able to get it to work, but not in the expected way, so I guess if others are still having the problem, this could help.

First, the attempt that didn't work. (Everything was done in the same browser/profile) 1 - When I first opened the link to the index, it asked me for the "are you human" question (It had also happened yesterday too). 2 - I then opened the Web2Epub page, and I couldn't import the chapters, as it kept on reporting the same errors (And between yesterday and today, I did close the browser/restart the machine, so there is also that) 3 - Tried to close and open the web2epub page again, it did not work.

So I used another profile where I never entered the site before (You probably could reproduce it by cleaning the cookies on the website) 1 - Opened about:debugging#/runtime/this-firefox 2 - Imported the zip file 3 - Opened the foxaholic novel index (and pressed the Are You Human button again. 4 - Opened Web2Epub page 5 - It works

So I'm guessing that maybe the page set a cookie or something when there is an error. So try cleaning all cookies. (I don't have a foxaholic account, so I tried everything without it).

I'm guessing the trick is related to having the cf_clearance cookie. I got it on the profile that worked, I did not get it on the one that didn't work.

I did try cleaning the cookies hoping to get the Are you human question again, but it doesn't appear at all, so it doesn't seem like I will be able to get it to work in this profile. But it did work on the other profile, so having different profiles or trying in another browser is a possible solution.

gamebeaker commented 2 months ago

@ssj4maiko could you try to open the first Chapter and than try to download? I have seen multiple sites that have one cloudflare challange for the novel Index and a different one for the Chapters. edit: i just had a site where i had to deactivate ublock origin to solve the cloudflare challange.

It could be something with the CHIPS they are at the moment in the firefox nightly build new. That is the stuff with cookies partition key. image

Later i am going to test if the fix is simple to change the function in HttpClient.js -> setPartitionCookies(url) to handle firefox. @dteviot is there somewhere logic to handle the change from browser. -> chrome. or do i have to copy the "same" code and just have to change chrome.cookies -> browser.cookies ?

ssj4maiko commented 2 months ago

@ssj4maiko could you try to open the first Chapter and than try to download? I have seen multiple sites that have one cloudflare challange for the novel Index and a different one for the Chapters. edit: i just had a site where i had to deactivate ublock origin to solve the cloudflare challange.

I did consider that ublock Origin may have been the cause, but both profiles had it, and I did face the "Are you human" challenge on both, but for some reason, that cookie was not set in the one that didn't work.

If anything, I could argue that the profile that worked is the one I use for dev, so it could be that the browser was better prepared to handle cloudflare stuff.

Regardless, I did try disabling ublock origin, cleaned the cookies, and keep on reloading, but I never get the cloudflare challenge again (instead, I got rate limited by foxaholic), so it doesn't seem like that would lead to a fix.

gamebeaker commented 2 months ago

@ssj4maiko Test versions for Firefox and Chrome have been uploaded to https://github.com/dteviot/WebToEpub/releases/tag/developer-build. Pick the one suitable for you, follow the "How to install from Source (for people who are not developers)" instructions at https://github.com/dteviot/WebToEpub/tree/ExperimentalTabMode#user-content-how-to-install-from-source-for-people-who-are-not-developers and let me know how it goes. Tested with:

ssj4maiko commented 2 months ago

@ssj4maiko Test versions for Firefox and Chrome have been uploaded to https://github.com/dteviot/WebToEpub/releases/tag/developer-build. Pick the one suitable for you, follow the "How to install from Source (for people who are not developers)" instructions at https://github.com/dteviot/WebToEpub/tree/ExperimentalTabMode#user-content-how-to-install-from-source-for-people-who-are-not-developers and let me know how it goes. Tested with:

* https://mtlarchive.com/novel/traveling-through-thousands-of-realms-starting-from-obtaining-the-ten-marvel-rings

Yup, it worked straight away 1st try. Tried on the same profile that was always failing before.

gamebeaker commented 2 months ago

I just found an error in the code. I am going to fix it in the next days.

gamebeaker commented 2 months ago

It wasn't an error.

dteviot commented 2 months ago

@ssj4maiko @yuyu-cloud Updated version (1.0.0.0) has been submitted to Firefox and Chrome stores. Firefox version is available now. Chrome might be available in a few hours to 21 days.