KeyError - "name='YII_CSRF_TOKEN', domain=None, path=None"

[Butterfly-Dragon]

Tried to download something from Henaifoundry because i'm a perv because i'm a perv.

I am getting this error:

error   hentaifoundry: An unexpected error occurred: KeyError - "name='YII_CSRF_TOKEN', domain=None, path=None". Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .

the since the log is already verbose i went there and it states this:

[gallery-dl][debug] Starting DownloadJob for 'https://www.hentai-foundry.com/pictures/user/thekite/'
[hentaifoundry][debug] Using HentaifoundryPicturesExtractor for 'https://www.hentai-foundry.com/pictures/user/thekite/'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): www.hentai-foundry.com:443
[urllib3.connectionpool][debug] https://www.hentai-foundry.com:443 "HEAD /?enterAgree=1 HTTP/1.1" 200 0
[hentaifoundry][error] An unexpected error occurred: KeyError - "name='YII_CSRF_TOKEN', domain=None, path=None". Please run gallery-dl again with the --verbose flag, copy its output and report this issue on https://github.com/mikf/gallery-dl/issues .
[hentaifoundry][debug] 
Traceback (most recent call last):
  File "C:\Python39\lib\site-packages\gallery_dl\job.py", line 67, in run
    for msg in self.extractor:
  File "C:\Python39\lib\site-packages\gallery_dl\extractor\hentaifoundry.py", line 34, in items
    self._init_site_filters()
  File "C:\Python39\lib\site-packages\gallery_dl\extractor\hentaifoundry.py", line 151, in _init_site_filters
    response.cookies["YII_CSRF_TOKEN"], "%22", "%22")[0]),
  File "C:\Python39\lib\site-packages\requests\cookies.py", line 328, in __getitem__
    return self._find_no_duplicates(name)
  File "C:\Python39\lib\site-packages\requests\cookies.py", line 399, in _find_no_duplicates
    raise KeyError('name=%r, domain=%r, path=%r' % (name, domain, path))
KeyError: "name='YII_CSRF_TOKEN', domain=None, path=None"

Might be an issue elsewhere too that uses cross site request forgery tokens to avoid unauthorized bots. I just found out by chance.

[mikf]

There have been some small changes regarding CSRF tokens and setting content filters in the last version (1.15.2, commit 783e0af2), but it works on CI and on my machine without problems.

$ gallery-dl https://www.hentai-foundry.com/pictures/user/thekite/
/tmp/hentaifoundry/thekite/hentaifoundr…4170_HMGBAPS Short: Clash of Wills 9.jpg
/tmp/hentaifoundry/thekite/hentaifoundr…WANDERLUST CYOA_ The Light of Ida 10.png^C
KeyboardInterrupt

Does this error also happen for you on an older version, e.g. 1.15.1? Does it happen for all hentaifoundry URLs, only some of them, or only after going through a few user profiles first?

[Butterfly-Dragon]

Happens to all hentaifoundry urls and from the start. 🤔 And this is the first version where it happened. But it could be a coincidence. Sometimes people change something but the error is due to the site changing how it works not the original code.

EDIT: Also i tried to go to some hentaifoundry pages and look around for stuff with my normal browser and then export the cookie... yup, same error. so it's not really something that happens because of that.

[mikf]

Could you run gallery-dl with a hentai-foundry URL and -o write-pages=all as cmdline option and post the files created in your current working directory here?

$ gallery-dl -o write-pages=all https://www.hentai-foundry.com/pictures/user/thekite/

[Butterfly-Dragon]

🤔

gallery-dl -o write-pages=all https://www.hentai-foundry.com/pictures/user/thekite/
* .\gallery-dl\hentaifoundry\thekite\hentaifoundry_854170_HMGBAPS Short_ Clash of Wills 9.jpg
* .\gallery-dl\hentaifoundry\thekite\hentaifoundry_853638_WANDERLUST CYOA_ The Light of Ida 10.png
* .\gallery-dl\hentaifoundry\thekite\hentaifoundry_852770_WANDERLUST CYOA_ The Light of Ida 9.png
* .\gallery-dl\hentaifoundry\thekite\hentaifoundry_852230_HMGBAPS Short_ Clash of Wills 8.jpg
* .\gallery-dl\hentaifoundry\thekite\hentaifoundry_851676_WANDERLUST CYOA_ The Light of Ida 8.png
* .\gallery-dl\hentaifoundry\thekite\hentaifoundry_851314_HMGBAPS Shorts 30_ Clash of Wills 7.jpg
* .\gallery-dl\hentaifoundry\thekite\hentaifoundry_850911_WANDERLUST CYOA_ The Light of Ida 7.png
* .\gallery-dl\hentaifoundry\thekite\hentaifoundry_850129_HMGBAPS Shorts 30_ Clash of Wills 6.jpg
* .\gallery-dl\hentaifoundry\thekite\hentaifoundry_849592_WANDERLUST CYOA_ The Light of Ida 6.png
* .\gallery-dl\hentaifoundry\thekite\hentaifoundry_849307_HMGBAPS Shorts 30_ Clash of Wills 5.jpg
* .\gallery-dl\hentaifoundry\thekite\hentaifoundry_848726_WANDERLUST CYOA_ The Light of Ida 5.png
* .\gallery-dl\hentaifoundry\thekite\hentaifoundry_847886_How My Trainer Became My Plaything (for a day!).jpg
* .\gallery-dl\hentaifoundry\thekite\hentaifoundry_847335_HMGBAPS Shorts 30_ Clash of Wills 4.jpg
* .\gallery-dl\hentaifoundry\thekite\hentaifoundry_846672_WANDERLUST CYOA_ The Light of Ida 4.png
  .\gallery-dl\hentaifoundry\thekite\hentaifoundry_846050_WANDERLUST CYOA_ The Light of Ida 3.png
KeyboardInterrupt

that... actually gave no problem whatsoever.

Files.zip

i normally use a custom config which goes like this:

{
    "extractor":
    {
        "base-directory": "D:/Downloads/Downloader/Manga/",
        "directory": ["Files"],
        "filename": "{filename}.{extension}",
        "archive": "D:/Downloads/Downloader//!Downloader/SQL/gallery-dl-archive.sqlite3",
        "cache.file": "D:/Downloads/Downloader//!Downloader/SQL/tmp/cache.sqlite3",
        "skip": "abort:3",
        "retries": -1,
        "sleep": 0,
        "restrict-filenames": "auto",
        "timeout": null,
        "postprocessors": null,
        "cookies": "D:/Downloads/Downloader//!Downloader/cookies.txt",
        "category-transfer": true,
        "parent-directory": true,
        "adjust-extensions": true,
        "refresh-token": "cache",

        "oauth":
        {
            "browser": true
        },
        "artstation":
        {
            "filename": "{user[full_name]} - {title} - {filename} {asset[id]} {asset[position]:03}.{extension}",
            "external": true
        },
        "deviantart":
        {
            "filename": "{author[username]} - {index} {title}.{extension}",
            "extra": true,
            "flat": true,
            "folders": false,
            "journals": "none",
            "mature": true,
            "metadata": true,
            "original": true,
            "quality": 100,
            "wait-min": 0,
            "wait-max": 8
        },
        "furaffinity":
        {
            "include": "gallery,scraps",
            "filename": "{filename} - {title}.{extension}"
        },
        "hentaifoundry":
        {
            "filename": "{user} - {title} - {filename}.{extension}"
        },
        "instagram":
        {
            "filename": "{username} {fullname} - {date} {filename}.{extension}"
        },
        "patreon":
        {
            "filename": "{creator[full_name]} {creator[vanity]} - {date} {title} - {filename}.{extension}"
        },
        "tumblr":
        {
            "filename": "{blog[name]} - {timestamp} {num:03} {slug}.{extension}",
            "avatar": false,
            "external": false,
            "inline": true,
            "posts": "all",
            "reblogs": false
        },
        "twitter":
        {
            "filename": "{user[name]} - {date} {tweet_id} {num:03}.{extension}",
            "skip": "abort:5",
            "netrc": false,
            "content": false,
            "retweets": false,
            "videos": true
        }
    },

    "downloader":
    {
        "part-directory": null,
        "rate": null,
        "retries": -1,
        "timeout": 30.0,
        "part": true,

        "http":
        {
            "mtime": true,
            "rate": null,
            "retries": -1,
            "timeout": 30.0,
            "verify": true
        },

        "ytdl":
        {
            "forward-cookies": true,
            "mtime": true,
            "rate": null,
            "retries": -1,
            "timeout": 30.0,
            "verify": true,
            "format": "bestvideo+bestaudio/best",
            "outtmpl": null
        }
    },

    "output":
    {
        "mode": "auto",
        "progress": true,
        "shorten": true,
        "log": {
            "level": "info",
            "format": {
                "debug"  : "\u001b [debug   {name}: {message}\u001b ]",
                "info"   : "\u001b [info    {name}: {message}\u001b ]",
                "warning": "\u001b [warning {name}: {message}\u001b ]",
                "error"  : "\u001b [error   {name}: {message}\u001b ]"
            }
        },
        "logfile": {
            "path": "D:/Downloads/Downloader//!Downloader/log.txt",
            "mode": "w",
            "level": "debug"
        }
    },

    "cache": {
        "file": "D:/Downloads/Downloader//!Downloader/SQL/cache.sqlite3"
    },

    "netrc": false
}

[Butterfly-Dragon]

uhm... sorry about the previous edits but i realized only after posting that i did not anonymize the file because i thought i had put everything in the cookies file... and that was not the case. 😅

[mikf]

Well, thank you for this, but I would've hoped for a crash and the HTTP requests leading to it, to figure out what's going on on your end and why you don't get a YII_CSRF_TOKEN cookie, etc.

It would be really helpful if you could record, with -o write-pages=all, what happens before a crash. Otherwise all I can do is guesswork and maybe roll some of the changes from https://github.com/mikf/gallery-dl/commit/783e0af26db186863ef817165490bf77a0de5e37 back to how it was before.

[Butterfly-Dragon]

Yeah. That's the problem. If i run it normally i... don't even get to the first downloaded image. 😅

It's not something that happens "after a while" or... "after downloading a few images" it's something that starts immediately. 😅

That is why i thought the problem was in the config file. Because i did not load the config file to do what you asked me.

[Butterfly-Dragon]

currently running it until i see if it creates a problem.

[Butterfly-Dragon]

yup, can confirm, went through the entire gallery without a single hitch.

[Butterfly-Dragon]

this time i'm uploading without the images because i doubt you need the actual 1 GB of smut, just the logs Files.zip

[mikf]

I've figured out what's causing it: Your global cookies.txt file already contains a YII_CSRF_TOKEN cookie for hentai-foundry.com. Unset the cookies file for hentaifoundry and it'll work without crashing:

        "hentaifoundry":
        {
            "cookies": null,
            "filename": "{user} - {title} - {filename}.{extension}"
        },

edit: or you use the changes from https://github.com/mikf/gallery-dl/commit/0211af7ca84552623ef95f8bc7799859fabcc6da