wwyaiykycnf / e621dl

The automated e621.net downloader
40 stars 23 forks source link

Cant get config.txt and print.txt #43

Closed twaalfkraaien closed 2 years ago

twaalfkraaien commented 6 years ago

When I run e621dl.py using cmd it gives me this

C:\Users\andol\Downloads\e621dl-2.4.6\e621dl-2.4.6>python e621dl.py File "e621dl.py", line 145 print '' ^ SyntaxError: Missing parentheses in call to 'print'. Did you mean print(print '')?

Wulfre commented 6 years ago

You need to use python 2 to run this version of the script.

twaalfkraaien commented 6 years ago

Well Ive gotten it to run now. But instead its not able to do anything. Even with python 2(2.7). It doesnt do anything. It does check the date. But after that It just closes instantly. image

Wulfre commented 6 years ago

The user agent that wwyaiykycnf is using probably got banned again. I can suggest either changing it yourself:

Edit support.py and downloader.py Replace version = 'e621dl ' + VERSION + ' (by wwyaiykycnf) with something else, such as version = 'e621dl/' + VERSION + ' (by [YOUR USERNAME HERE])'

Or just use my fork because this one seems to get banned every time that the user agent is changed after a short while. Do note that my version is very different from this one now.

DatDraggy commented 6 years ago

I created a new pull request for this issue. https://github.com/wwyaiykycnf/e621dl/pull/45 You can either wait for it to get approved (could take a while) or download it directly already from my profile. https://github.com/DatDraggy/e621dl

Instead of having to edit support and downloader you can just choose a username in the config.txt

Wulfre commented 6 years ago

A side note, I'm glad you keep coming back to try and fix these little things Draggy. I hate redirecting people to my fork but there is something that e621 doesn't like about how this one is written, otherwise it wouldn't keep getting banned.

DatDraggy commented 6 years ago

Nono it's literally only the useragent. Read their guidelines. It doesn't allow emulated agents and you have to use your own username in it. Otherwise they'll throttle it if you use it too much on one agent (might be that they perma banned wwyaiykycnf as username)

I even asked the e621 team/devs about it: https://puu.sh/yH3yk/29dd47c83c.png

Wulfre commented 6 years ago

I read that as the project should have the username of the creator so that if there is an issue they can contact the developer.

I think that the reason why the user agent keeps getting banned is because there are no HTTP error checks in the code, so it will keep requesting data even after it has been denied access. This is why even after wwyaiykycnf changed the useragent to the TOS's format it keeps getting banned.

DatDraggy commented 6 years ago

They never contacted wwyaiykycnf about this. It should be the users name, not developer because cloudflare might rate limit it if there is too much on one user agent.

Since I changed my version to my own username I've never been blocked again. I'm gonna ask e621 about it again if we receive more banned agent issues

EDIT: Re-read their guidelines. I'm gonna ask why its still being blocked

DatDraggy commented 6 years ago

Okay, so it's because the max-posts-per-page is only at 100. Therefor, more pages (=> requests) are needed to fetch all posts. This results in cloudflare rate limiting.

They said we can use my new pull request to prevent everyone from being limited.

I'm gonna play around with it a little more and then probably make a new request.

So, I guess, apologies for being too naive and not looking into what e621_api.py does exactly.

Wulfre commented 6 years ago

There are two solutions to this. Increase the post limit to 320 and then either limit the pages to 750 (there is a page limit also) use use the before_id style request which is what I use on my fork. There is no limit to before_id because it starts from whatever id it is given, rather than always starting from the latest post and shifting down by a number of pages.

DatDraggy commented 6 years ago

The second part sounds very smart, but I doubt that I can do that myself, lol. You could merge your edit with the original one if you like. I'm gonna see what's the highest max-posts I can set before it stops working properly and then ask for a pull again.

Wulfre commented 6 years ago

The max posts is 320. I would do a merge but my fork is a completely different program at this point.

DatDraggy commented 6 years ago

That sucks. What do you think about adding a small pause/timeout for every x pages to relax cloudflare a bit?