search

Nandaka / PixivUtil2

Download images from Pixiv and more!
http://nandaka.devnull.zone/
BSD 2-Clause "Simplified" License
2.4k stars 254 forks source link

CF_Clearance Cookie Missing #1379

Open Hi-FiMan opened 1 month ago

Hi-FiMan commented 1 month ago

Prerequisites

Description

Fanbox is no longer giving out cf_clearance cookie so I can no longer login with PixivUtil.

Versions

Latest release. Running from source does not work as it keeps saying demjson3 module is missing even though I built it with pip and all that.

prowlguru commented 1 month ago

I encountered the same issue. You use Firefox to get the cookies, right? I think a new version of Firefox might have changed how some cookies are handled, because the cf_clearance cookie disappears after login. It doesn't seem to happen with Chromium browsers, but I can't get the cookies from the those to work at all.

I think this happened with Firefox 130 or 131. The current solution that works for me is using a slightly older Firefox to get the cookies, like the ESR version.

So, as a summary for what is currently needed for things to work: The current release of PixivUtil2 (v20240703). EmerladCoder's fix to two files that you have to replace: https://github.com/Nandaka/PixivUtil2/issues/1361#issuecomment-2273943969 Copy the relevant cookies from a Firefox earlier than version 130 or 131.

With this, Fanbox works for me. It might be time to simply use the full cookie, like gallery-dl does?

Nandaka commented 1 month ago

I don't really use Fanbox, so if you can send me a PR, then I'll merge it to master.

Hi-FiMan commented 1 month ago

I encountered the same issue. You use Firefox to get the cookies, right? I think a new version of Firefox might have changed how some cookies are handled, because the cf_clearance cookie disappears after login. It doesn't seem to happen with Chromium browsers, but I can't get the cookies from the those to work at all.

I think this happened with Firefox 130 or 131. The current solution that works for me is using a slightly older Firefox to get the cookies, like the ESR version.

So, as a summary for what is currently needed for things to work: The current release of PixivUtil2 (v20240703). EmerladCoder's fix to two files that you have to replace: #1361 (comment) Copy the relevant cookies from a Firefox earlier than version 130 or 131.

With this, Fanbox works for me. It might be time to simply use the full cookie, like gallery-dl does?

Yeah Chromium based browsers don't even see a cookie to export. I'm not sure what the long term solution would be if newer versions of Firefox are "broken".

It would be REALLY nice if I could just pass the cookie file to the program like most other downloaders out there. It gets really annoying having to do this song and dance every other day because cloudflare changed.

biggestsonicfan commented 1 month ago

It would be REALLY nice if I could just pass the cookie file to the program like most other downloaders out there. It gets really annoying having to do this song and dance every other day because cloudflare changed.

I've been thinking of working on a PR to handle this, but I feel like we're getting to a point where people are forgetting how to use computers and expect things on a desktop to magically work like a phone app...

Hi-FiMan commented 1 month ago

The entire point of this program is to make things easier...

I work in IT and I hate phones and silly apps, but I also hate repetitive mundane tasks and that's why I automate stuff at work and that's why people created this program.

biggestsonicfan commented 1 month ago

I could argue that with you to death, however, I think I have found the problem. If the data I am looking at is correct, Pixiv is creating cf_clearance, reading it, then it immediately expires. There will be no cf_clearance cookie to read because it has already expired.

Hi-FiMan commented 1 month ago

Why does the cookie not immediately expire on firefox 130 and older?

biggestsonicfan commented 1 month ago

I don't know, but I'm on 131 now...

EDIT: As a temporary fix, I've installed Firefox portable 130 and logged in via that to get the cookie. This is so odd...

EDIT2: I have confirmed cf_clearance remains in the cookies.sqlite file of a Firefox profile, but it appears that both Firefox dev tools and extensions do not show expired cookies. I have also confirmed that the cf_clearance once extracted will work with fanbox.

biggestsonicfan commented 1 month ago

It was discovered by mikf that Firefox 131 forward can hide these kinds of cookies:

Firefox 131 introduced support for "partitioned" cookies: CHIPS. This prevents addons like Export Cookies from including the cf_clearance cookie in their output... This feature can be controlled with Firefox's network.cookie.CHIPS.enabled setting in about:config.

So the solution for now is to change the boolean value of network.cookie.CHIPS.enabled in your about:config for now. image Change this value to false