search

AlphaSlayer1964 / kemono-dl

A simple kemono.party downloader using python.
504 stars 81 forks source link

Downloader fails to get creator info #64

Closed bog4t closed 2 years ago

bog4t commented 2 years ago

NSFW

Version

Version: 2022.02.26

Your Command


python kemono-dl.py --cookies kemono.party_cookies.txt --output ".downloads/%(service)s/%(username)s" --links "https://kemono.party/patreon/user/18701935, https://kemono.party/patreon/user/3535362, https://kemono.party/fanbox/user/13651304" --verbose

Description of bug

Script seems to fail when attempting to get all creators, thus the rest of the program also fails.

How To Reproduce

Get cookies while logged into a consumer account, run command as shown above.

Error messages and tracebacks


DEBUG: Given command: python kemono-dl.py --cookies kemono.party_cookies.txt --output .downloads/%(service)s/%(username)s --links https://kemono.party/patreon/user/18701935, https://kemono.party/patreon/user/3535362, https://kemono.party/fanbox/user/13651304 --verbose
ERROR: Failed to get creator info from kemono.party
ERROR: Failed to find user for service:patreon user_id:18701935
ERROR: Failed to find user for service:patreon user_id:3535362
ERROR: Failed to find user for service:fanbox user_id:13651304
DEBUG: Completed in 00:00:00
INFO: Completed

Additional comments

AlphaSlayer1964 commented 2 years ago

Are you using a vpn? Can you access https://kemono.party/api/creators in your browser? I just ran a test and was able to download normally and get the creators api.

anon567321 commented 2 years ago

Hello! I have the exact same problem. And what's more, all of a sudden the last version stopped working, I didn't change anything in the system, but now it gives this error. I can access https://kemono.party/api/creators in my browser. 1 .

bog4t commented 2 years ago

Are you using a vpn? Can you access https://kemono.party/api/creators in your browser? I just ran a test and was able to download normally and get the creators api.

Nope, I am not using a vpn and I'm able to get a response via my browser but not through the program. I did manage to get some more insight into the problem though.

After changing line 77 in src/main.py to: 

except requests.ConnectionError as e:

I was able to receive this:

Traceback (most recent call last):
  File "kemono-dl\venv\lib\site-packages\requests\models.py", line 910, in json
    return complexjson.loads(self.text, **kwargs)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.2800.0_x64__qbz5n2kfra8p0\lib\json\__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.2800.0_x64__qbz5n2kfra8p0\lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.2800.0_x64__qbz5n2kfra8p0\lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "kemono-dl\kemono-dl.py", line 4, in <module>
    main()
  File "kemono-dl\src\main.py", line 807, in main
    downloader(urls)
  File "kemono-dl\src\main.py", line 38, in __init__
    self.all_creators = self.get_all_creators()
  File "kemono-dl\src\main.py", line 76, in get_all_creators
    all_creators += self.session.get(url=creators_api_url, headers=headers, timeout=TIMEOUT).json()
  File "kemono-dl\venv\lib\site-packages\requests\models.py", line 917, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: [Errno Expecting value] <!DOCTYPE html><html><head><title>DDOS-GUARD</title><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1"><script>var DOMReady=function(t){var e=document,d="addEventListener";e[d]?e[d]("DOMContentLoaded",t):window.attachEvent("onload",t)},loadScript=function(t,e){var d=document.createElement("script");d.type="text/javascript",d.src=t,"string"==typeof e&&""!==e&&(d.id=e),(document.getElementsByTagName("head")[0]||document.body).appendChild(d)};DOMReady(function(){loadScript("https://check.ddos-guard.net/check.js"),loadScript("/.well-known/ddos-guard/check?context=free_splash","ddg_script_f"),setTimeout(function(){document.location.reload(!0)},3e3)});</script><style>*{margin:0;padding:0}body,html{font-family:Open Sans,Arial,Helvetica,sans-serif;height:100%;background:linear-gradient(to top,#e5f3fb,#fff)}.logo{width:120px;margin-bottom:35px}#title{font-size:32px;font-weight:900;margin-bottom:40px}#description{color:#8f9390;margin-bottom:30px}#link-ddg{white-space:nowrap;margin-bottom:30px;font-size:18px}#link-ddg a:focus:active:hover:visited{color:#00adee}#link-ddg a:active{color:#00adee}#link-ddg a:hover{color:#00adee}#link-ddg a:visited{color:#00adee}.container{display:flex;flex-direction:column;justify-content:center;align-items:center;height:100%;text-align:center;background:url(data:image/svg+xml;base64,PHN2ZyBpZD0i0KHQu9C+0LlfMSIgZGF0YS1uYW1lPSLQodC70L7QuSAxIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjAgMCAxNjYuNzUgNDcuOTciPjxkZWZzPjxzdHlsZT4uY2xzLTF7ZmlsbDojZmZmO308L3N0eWxlPjwvZGVmcz48dGl0bGU+0JzQvtC90YLQsNC20L3QsNGPINC+0LHQu9Cw0YHRgtGMIDE8L3RpdGxlPjxwYXRoIGNsYXNzPSJjbHMtMSIgZD0iTTEzNS41NSwzOS44MmMtLjM4LDAtLjc1LDAtMS4xMywwYTE4LjkxLDE4LjkxLDAsMCwwLTM1LjMzLTQuNTNBMjUuMjYsMjUuMjYsMCwwLDAsNzcuMjEsMjIuN2MtLjgyLDAtMS42MywwLTIuNDMuMTJBMjkuMjYsMjkuMjYsMCwwLDAsMTcsMjkuMjZhMjkuNTksMjkuNTksMCwwLDAsLjE2LDNBMTguNTIsMTguNTIsMCwwLDAsMCw0OEgxNTAuNDNBMTcuNjQsMTcuNjQsMCwwLDAsMTM1LjU1LDM5LjgyWiIvPjxwYXRoIGNsYXNzPSJjbHMtMSIgZD0iTTE1OS40OSw0NS40NUE5LjU4LDkuNTgsMCwwLDAsMTUzLDQ4aDEzQTkuNTgsOS41OCwwLDAsMCwxNTkuNDksNDUuNDVaIi8+PC9zdmc+) center bottom no-repeat}.lds-spin{width:80px;height:80px}@media screen and (max-width:1200px){.logo{width:100px}#title{font-size:29px}#link-ddg{font-size:15px}#link-ddg{font-size:20px}}@media screen and (max-width:770px){.logo{width:70px;margin-bottom:10px}#title{font-size:15px;margin-bottom:10px}#description{margin-bottom:10px;font-size:13px}#link-ddg{margin-bottom:10px;font-size:13px}.lds-spin{width:40px;height:40px}#link-ddg{font-size:18px}}</style></head><body><div class="container"><div class="logo"><svg id="dl" xmlns="http://www.w3.org/2000/svg" viewbox="0 0 6.62 7.69"><defs><style>.cls-1{isolation:isolate}.cls-2{fill:#00adee}.cls-3{fill:#fff}</style></defs><title>ddos_3</title><g id="_2560" data-name="2560" class="cls-1"><g id="G2" data-name="Gr2"><path class="cls-2" d="M3.3,0A6,6,0,0,1,0,1.91C.13,4.46,1.6,7.49,3.3,7.65,5,7.49,6.47,4.46,6.6,1.91A6,6,0,0,1,3.3,0Z"/><polygon class="cls-3" points="4.55 4.73 4.55 2.74 3.29 1.79 2.03 2.71 2.03 2.91 3.25 2.2 3.25 2.59 2.03 3.16 2.03 3.4 3.25 2.95 3.25 3.26 2.03 3.66 2.03 3.91 3.25 3.66 3.25 4.01 2.03 4.18 2.03 4.43 3.25 4.36 3.25 4.73 1.89 4.73 1.89 5.09 4.71 5.09 4.71 4.73 4.55 4.73"/></g></g></svg></div><div id="title"></div><div id="description"></div><div id="link-ddg"><a href="https://ddos-guard.net" target="_blank" id="link"></a></div><div class="lds"><svg class="lds-spin" width="100px" height="100px" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" viewbox="0 0 100 100" preserveaspectratio="xMidYMid" style="background: none;"><g transform="translate(80,50)"><g transform="rotate(0)"><circle cx="0" cy="0" r="10" fill="#00adee" fill-opacity="1" transform="scale(0.7525 0.7525)"><animatetransform attributename="transform" type="scale" begin="-0.875s" values="0.7 0.7;1 1" keytimes="0;1" dur="1s" repeatcount="indefinite"/><animate attributename="fill-opacity" keytimes="0;1" dur="1s" repeatcount="indefinite" values="1;0" begin="-0.875s"/></circle></g></g><g transform="translate(71.21320343559643,71.21320343559643)"><g transform="rotate(45)"><circle cx="0" cy="0" r="10" fill="#00adee" fill-opacity="0.875" transform="scale(0.715 0.715)"><animatetransform attributename="transform" type="scale" begin="-0.75s" values="0.7 0.7;1 1" keytimes="0;1" dur="1s" repeatcount="indefinite"/><animate attributename="fill-opacity" keytimes="0;1" dur="1s" repeatcount="indefinite" values="1;0" begin="-0.75s"/></circle></g></g><g transform="translate(50,80)"><g transform="rotate(90)"><circle cx="0" cy="0" r="10" fill="#00adee" fill-opacity="0.75" transform="scale(0.9775 0.9775)"><animatetransform attributename="transform" type="scale" begin="-0.625s" values="0.7 0.7;1 1" keytimes="0;1" dur="1s" repeatcount="indefinite"/><animate attributename="fill-opacity" keytimes="0;1" dur="1s" repeatcount="indefinite" values="1;0" begin="-0.625s"/></circle></g></g><g transform="translate(28.786796564403577,71.21320343559643)"><g transform="rotate(135)"><circle cx="0" cy="0" r="10" fill="#00adee" fill-opacity="0.625" transform="scale(0.94 0.94)"><animatetransform attributename="transform" type="scale" begin="-0.5s" values="0.7 0.7;1 1" keytimes="0;1" dur="1s" repeatcount="indefinite"/><animate attributename="fill-opacity" keytimes="0;1" dur="1s" repeatcount="indefinite" values="1;0" begin="-0.5s"/></circle></g></g><g transform="translate(20,50.00000000000001)"><g transform="rotate(180)"><circle cx="0" cy="0" r="10" fill="#00adee" fill-opacity="0.5" transform="scale(0.9025 0.9025)"><animatetransform attributename="transform" type="scale" begin="-0.375s" values="0.7 0.7;1 1" keytimes="0;1" dur="1s" repeatcount="indefinite"/><animate attributename="fill-opacity" keytimes="0;1" dur="1s" repeatcount="indefinite" values="1;0" begin="-0.375s"/></circle></g></g><g transform="translate(28.78679656440357,28.786796564403577)"><g transform="rotate(225)"><circle cx="0" cy="0" r="10" fill="#00adee" fill-opacity="0.375" transform="scale(0.865 0.865)"><animatetransform attributename="transform" type="scale" begin="-0.25s" values="0.7 0.7;1 1" keytimes="0;1" dur="1s" repeatcount="indefinite"/><animate attributename="fill-opacity" keytimes="0;1" dur="1s" repeatcount="indefinite" values="1;0" begin="-0.25s"/></circle></g></g><g transform="translate(49.99999999999999,20)"><g transform="rotate(270)"><circle cx="0" cy="0" r="10" fill="#00adee" fill-opacity="0.25" transform="scale(0.8275 0.8275)"><animatetransform attributename="transform" type="scale" begin="-0.125s" values="0.7 0.7;1 1" keytimes="0;1" dur="1s" repeatcount="indefinite"/><animate attributename="fill-opacity" keytimes="0;1" dur="1s" repeatcount="indefinite" values="1;0" begin="-0.125s"/></circle></g></g><g transform="translate(71.21320343559643,28.78679656440357)"><g transform="rotate(315)"><circle cx="0" cy="0" r="10" fill="#00adee" fill-opacity="0.125" transform="scale(0.79 0.79)"><animatetransform attributename="transform" type="scale" begin="0s" values="0.7 0.7;1 1" keytimes="0;1" dur="1s" repeatcount="indefinite"/><animate attributename="fill-opacity" keytimes="0;1" dur="1s" repeatcount="indefinite" values="1;0" begin="0s"/></circle></g></g></svg></div></div><script type="text/javascript">var lang=navigator.language,host=window.location.hostname;"ru"==lang||"ru-RU"==lang?(document.getElementById("title").innerHTML="Проверка браузера перед переходом на сайт "+host,document.getElementById("description").innerHTML="Это автоматический процесс. Вы будете перенаправлены на запрашиваемый ресурс в ближайшее время.<br>Пожалуйста ожидайте...",document.getElementById("link").innerHTML="Защита от DDoS-атак DDoS-GUARD"):(document.getElementById("title").innerHTML="Checking your browser accessing "+host,document.getElementById("description").innerHTML="This process is automatic. Your browser will redirect to your requested content shortly.<br>Please allow up to 5 seconds...",document.getElementById("link").innerHTML="DDoS protection by DDoS-GUARD");</script></body></html>: 0
AlphaSlayer1964 commented 2 years ago

Weird the ddos guard shouldn't be set up for their api. It also doesn't seem to be set up when I make a request. Well since you have to pass cookies in anyways I guess I will make that request take in the cookies as well.

AlphaSlayer1964 commented 2 years ago

@bog4t in main.py line 79 can you change it to this: 

all_creators = self.session.get(url=creators_api_url, headers=headers, timeout=TIMEOUT, cookies=args['cookies'])

And see if that fixes the issue? I am unable to get the DDOS guard to show up when making api calls. I do know it shows when accessing files, that's what the cookies are for.

anon567321 commented 2 years ago

I have some kind of error

Edit: Wait, I am stupid. First I changed line 79 in kemono-dl-2022.02.26 and got the error. Now I changed line 79 in kemono-dl-2022.02.10, and everything works as it should.

AlphaSlayer1964 commented 2 years ago

I changed line 79 in kemono-dl-2022.02.10, and everything works as it should.

Ok then it looks like they have DDOS guard for even the api calls now. I will have to fix that.

bog4t commented 2 years ago

I cloned the repository and it seems I'm not getting the DDOS guard anymore and the downloader's working again.

AlphaSlayer1964 commented 2 years ago

There should never have been DDOS guard checks on the api calls so that's weird, but it seems it happened to a few people. I am going to go ahead and add sending the cookies with the api calls in future releases just in case.