search

grrttedwards / wow-addon-updater

Python utility for updating World of Warcraft addons
GNU General Public License v3.0
71 stars 14 forks source link

curse download fails again - possible cloudscraper fix #198

Open ilu33 opened 1 year ago

ilu33 commented 1 year ago

Describe the bug All curse downloads fail with "403 None"

If you run pipenv update on the command line and try again, does the issue go away? No

Log output All curse downloads fail with "403 None" - no need to post the same log entry over and over.

I see that requests are made with HTTP/1.1. According to https://stackoverflow.com/questions/65604551/cant-bypass-cloudflare-with-python-cloudscraper it might help to

Sadly I'm not good enough to know where exactly to place these things in your code.

ghost commented 1 year ago

I just found a github user who mirrors curseforge addons. They also have a fork of cloudscraper with 34 new commits since the last update on upstream. This may be a solution, but I have not tried it. https://github.com/curseforge-mirror/cloudscraper

DUTNO1 commented 1 year ago

It's the hardcoded UserAgent that's mucking things up, in updater/site/__init__.py change CURSE_UA from: CURSE_UA = {'custom': 'Chrome/80'} to: CURSE_UA = {'browser': 'chrome', 'platform': 'windows', 'mobile': False} - as mentioned in the SO thread and you should be good to go.

grrttedwards commented 1 year ago

@DUTNO1 Thanks. Yeah the user agent trick was a random discovery to fix Curse blocking years ago.

I may use Dragonflight as a reason to come and update this code. But it's tough for me to find the time. I appreciate everyone's dedication to this tool!

ilu33 commented 1 year ago

@DUTNO1 Changing that did not help. Does it work for you?

@omicron-o Do you have any idea how one could include the modified library you mentioned instead of the on provided by pip? Is there a way to create a temporary local override to test it?

DUTNO1 commented 1 year ago

@ilu33 welp, it seems like they "fixed" it - doesn't work for me anymore either.

grrttedwards commented 1 year ago

Looks like it's spontaneously working fine again:

Installing/updating addon: bagnon to version: 10.0.3...
...
Installing/updating addon: advancedinterfaceoptions to version: 1.8.1...

Name                        Prev. Version               New Version
────                        ─────────────               ───────────
hear-kitty                  -----                       1.10.0
claw                        -----                       9.0.2-6
...
deadly-boss-mods            -----                       DBM-10.0.0
Press any key to continue . . .
ghost commented 1 year ago

I suspect that sometimes it breaks because curse/cloudflare sometimes puts the site into a higher level of protection, causing some "v2" captcha to pop up that the library does not support. I'm not super sure though, just guessing based on log/exception output.

arsirantala commented 1 year ago

Would love that this issue is solved somehow - I checked where the addon files are actually downloaded (in curseforge's case), and they are hosted in mediafilez.forgecdn.net.

The url's suffix follows the addon name and its version.. I think it should be pretty straight forward to form a valid url in the script for the addon file from forgecdn to be downloaded (without headers/cookies been set).