dteviot / WebToEpub

A simple Chrome (and Firefox) Extension that converts Web Novels (and other web pages) into an EPUB.
Other
719 stars 136 forks source link

Mtlarchive is having some 403 issues again. #1439

Closed Franzeeee closed 1 month ago

Franzeeee commented 2 months ago

The mtlarchive.com is having some 403 errors again. Even if i already did the captcha and already logged in, it's still giving me the 403 error.

Kiradien commented 2 months ago

Looks like we've hit an issue with sending the cf cookies again; not entirely sure why this is hitting specifically on mtlarchive. I'll try to look into this in the evening if no one picks it up. The default request cookie in the domain appears to have cf_clearance after a new line - might be the reason. image

cf_clearance is completely missing.

gamebeaker commented 2 months ago

@Franzeeee Firefox works, Chrome is broken.

gamebeaker commented 2 months ago

Looks to me like it is going to be a bigger problem. I just tested https://www.scribblehub.com/ and it also stopped working. At the moment i am not sure what the problem is. It looks like something changed with the cookies: let name1 = "cf_clearance"; cookies = await chrome.cookies.getAll({name: name1}); returns all "old" wesites with cloudflare cookie. Example i get a cookie for discord -> visit discord -> execute js again now the cookie for discord isn't returned anymore. It looks to me like cloudflare changed some flags for the cookie cf_clearance: old: image new: image edit: I just visited an old issue: #1306 and in the comment i uploaded a screenshot where the flags are the same as in the "new" screenshot:

I think the problem is, that the cloudflare cookie is not sent with the request. If you open Inspect -> Network click the failed request -> cookies there is this message: Screenshot 2024-05-21 230645 Link from message: https://developers.google.com/privacy-sandbox/3pcd/chips?utm_source=devtools For fanfiction.net the missing cookies name is cf_clearance: grafik WebToEpub in firefox sends this cookie: grafik edit: @Kiradien i missed your comment sry.

Originally posted by @gamebeaker in https://github.com/dteviot/WebToEpub/issues/1306#issuecomment-2123450414

Kiradien commented 2 months ago

It looks like something changed with the cookies:

It looks like the problem is in Chrome rather than the cookies themselves; I've been digging through the setting method and the following request returns an empty array. let cookies = await chrome.cookies.getAll({partitionKey: {topLevelSite: topLevelSite}});

It shouldn't be a permission issue, as the old standby does still return cookies... however it does not include the ones we need. await chrome.cookies.getAll({url: 'https://mtlarchive.com'})

Edit: After fiddling around with the cookie itself, the parameter Partition Key Site seems to be the source of issues. The moment this value is set, all requests seem to ignore the cookies very existence. Doing a basic getAll with no properties shows how it appears/disappears depending on how it's set. image

This is either a permissions thing, in which I can't find the setting required, or Google has completely cut out access to these items.

No idea where to go from here -_-

gamebeaker commented 2 months ago

I have opened an issue with chromium: https://issues.chromium.org/issues/361862746

gamebeaker commented 2 months ago

@Kiradien @dteviot would be cool if you guys could vote "I am impacted" in the issue tracker: https://issues.chromium.org/issues/361862746

gamebeaker commented 2 months ago

@Franzeeee Test versions for Firefox and Chrome have been uploaded to https://github.com/dteviot/WebToEpub/releases/tag/developer-build. Pick the one suitable for you, follow the "How to install from Source (for people who are not developers)" instructions at https://github.com/dteviot/WebToEpub/tree/ExperimentalTabMode#user-content-how-to-install-from-source-for-people-who-are-not-developers and let me know how it goes. Tested with:

gamebeaker commented 1 month ago

Chrome code fix got merged https://issues.chromium.org/issues/361862746.

dteviot commented 1 month ago

@Franzeeee Updated version (1.0.0.0) has been submitted to Firefox and Chrome stores. Firefox version is available now. Chrome might be available in a few hours to 21 days.