New Site Update Broke e621dl

[RedFoxNightfox]

Here is all it does now.

e621dl INFO Running e621dl version 4.4.1.

e621dl INFO Checking for partial downloads.

e621dl INFO Parsing config. Traceback (most recent call last): File "e621dl.py", line 74, in File "e621dl.py", line 74, in File "lib\remote.py", line 71, in get_tag_alias File "site-packages\requests\models.py", line 935, in raise_for_status requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://e621.net/tag/index.json [15384] Failed to execute script e621dl Press any key to continue . . .

[grug-son-of-thog]

Seeing the same issue on my end. Which is weird, because that page does exist still.

[RedFoxNightfox]

https://e621.net/tag/index.json redirects to https://e621.net/tags?page= I don't see that index.json page actually existing.

[grug-son-of-thog]

Ah, right you are. I didn't notice that the wget redirected. Boo.

[Wulfre]

The new API is still being documented so I'll update this as soon as I get all the information I need.

[lurkbbs]

Some good news. According to this post, new tag limit is 40.

[Wulfre]

Nice. Thanks for the info. I'll change that too when I fix everything else.

[Wulfre]

I ran into an issue where some posts are hidden unless you are logged in. Would it be a bad idea to have users log in to their accounts through the script, or is it preferred to skip over hidden posts?

EDIT: I realized after posting this that some people browse without an account, so I think I'll have to skip over hidden posts by default and leave a disclaimer that some posts will be missed unless the user creates and logs in to an account.

[grug-son-of-thog]

My first thought is that it should be able to work both ways. Give the user the option to sign in and scrape the hidden files, or let them remain logged out and just skip them. Perhaps functionality could also be included that would encrypt the user's credentials and store them locally so they don't need to log in every time?

[lurkbbs]

Perhaps functionality could also be included that would encrypt the user's credentials and store them locally so they don't need to log in every time?

I don't think encryption is that needed. API is already uses api key instead of password, and it can be changed with two clicks. And unless you want to enter password to encrypt/decrypt said key, any default password is trivially deducible with code analysis.

[Wulfre]

Commit 1ae15b0 fixes the largest issues that the API changes caused. Downloading posts works again.

Tag verification is broken for now, so I might use logging in to check a user's blacklist as well since I can't confirm if tags are spelled, aliased, and ultimately blacklisted, correctly anymore.

[RedFoxNightfox]

seeing as I have no idea how to use commit source code, any chance of updating the exe release to the current commit?

[lurkbbs]

seeing as I have no idea how to use commit source code, any chance of updating the exe release to the current commit?

You can install python and then just doubleclick e621dl.py. Reading of the code is not necessary. And you can it install easily with chocolatey, that is also quite easy to install.

1. Open PowerShell as an Admin. On Win10, you can right-click on Start Button, then select PowerShell (Admin)
2. Paste this and press Enter:

Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))

And confirm everything if needed. This installs chocolatey.

3. Paste this and press Enter:

choco install python

And also confirm everything. Obviously, this installs python. If it doesn't work, close PowerShell then open it and paste-enter again.

4. Download source and place everything in a folder with e621dl.exe.

All done! To run, just double-click e621dl.py instead of the executable.

[RedFoxNightfox]

Yeah, I've tried that, but it just gives me errors,

H:\E621\e621dl-3.1.1>e621dl.py Traceback (most recent call last): File "H:\E621\e621dl-3.1.1\e621dl.py", line 11, in <module> from e621dl import local File "H:\E621\e621dl-3.1.1\e621dl\local.py", line 9, in <module> import yaml File "C:\Python38\lib\site-packages\yaml\__init__.py", line 2, in <module> from error import * ModuleNotFoundError: No module named 'error'

this is why I just stick with the exe.... working with commit source code is just a big headache for me.

[lurkbbs]

it just gives me errors,

Sorry. Totally forgot about external modules. From PowerShell or command line (with or without Admin privileges), run this

pip install pyyaml requests

working with commit source code is just a big headache for me.

And yet the only way to work with non-release version. I work with pyinstaller, a tool to assemble executables, and the only reliable way to make executable to run on Win 7 is to assemble it on Win 7 in some Virtual Machine. Passing code to and executable from is such a pain.

Also, pyinstaller doesn't like when e621dl.py file imports from e621dl folder. I tried different ways but eventually gave up and just renamed e621dl to e621dl_lib.

[RedFoxNightfox]

that just gave errors.

Traceback (most recent call last): File "c:\python38\lib\runpy.py", line 193, in _run_module_as_main return _run_code(code, main_globals, None, File "c:\python38\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "C:\Python38\Scripts\pip.exe__main.py", line 5, in File "c:\python38\lib\site-packages\pip_internal__init.py", line 40, in from pip._internal.cli.autocompletion import autocomplete File "c:\python38\lib\site-packages\pip_internal\cli\autocompletion.py", line 8, in from pip._internal.cli.main_parser import create_main_parser File "c:\python38\lib\site-packages\pip_internal\cli\main_parser.py", line 11, in from pip._internal.commands import ( File "c:\python38\lib\site-packages\pip_internal\commands\init__.py", line 6, in from pip._internal.commands.completion import CompletionCommand File "c:\python38\lib\site-packages\pip_internal\commands\completion.py", line 6, in from pip._internal.cli.base_command import Command File "c:\python38\lib\site-packages\pip_internal\cli\base_command.py", line 26, in from pip._internal.index import PackageFinder ImportError: cannot import name 'PackageFinder' from 'pip._internal.index' (c:\python38\lib\site-packages\pip_internal\index\init__.py)

-.- this is why I was requesting an update to the release version.

[Wulfre]

Sorry for my absence. What @lurkbbs said is true, and I lost the Win 7 virtual machine that I had set up to make executables, which is why there hasn't been one in over a year. I also use pyinstaller and it's not exactly the most reliable way to turn a python script into an exe. I refactored the code to be easier to work with in python and since then the exe hasn't been coming out correctly. Following the instructions in the README should get it running in Python correctly, though I'm assuming that you have tried that already.

The only difference is that the instructions assume you install from the installer on the official python website, but I wouldn't expect the chocolatey version to be configured too differently.

[RedFoxNightfox]

I did.

[lurkbbs]

Oh, dear. I swear, I just updated python from 3.7 to 3.8 via chocolatey and everything worked. But just at the wrong moment pip choose to be broken. OK, please upgrade pip, just download it from here and double-click (or python get-pip.py from a console). Then try pip install pyyaml requests again.

@Wulfre, README is kinda outdated and doesn't mention neither pyyaml nor requirements.txt.

[RedFoxNightfox]

that worked, but when I ran the thing, after renaming the other directory to e621dl_lib it now does this:

Traceback (most recent call last): File "e621dl.py", line 10, in from e621dl import constants File "H:\E621\e621dl-3.1.1\e621dl.py", line 10, in from e621dl import constants ImportError: cannot import name 'constants' from partially initialized module 'e621dl' (most likely due to a circular import) (H:\E621\e621dl-3.1.1\e621dl.py) Press any key to continue . . .

[Wulfre]

You don't need to rename the directory. That was just an attempt to make the exe compile correctly. The script references the directory exactly how it is in the repo.

@lurkbbs I could have sworn I updated the README when I changed to yaml. Thanks for reminding me.

[RedFoxNightfox]

then I get the following:

[i] Running e621dl version 5.0.0.

[i] Checking for partial downloads...

[i] Parsing config... Traceback (most recent call last): File "e621dl.py", line 34, in config = local.get_config() File "H:\E621\e621dl-3.1.1\e621dl\local.py", line 23, in get_config config = yaml.load(infile, Loader = yaml.SafeLoader) File "C:\Python38\lib\site-packages\yaml__init__.py", line 114, in load return loader.get_single_data() File "C:\Python38\lib\site-packages\yaml\constructor.py", line 49, in get_single_data node = self.get_single_node() File "C:\Python38\lib\site-packages\yaml\composer.py", line 36, in get_single_node document = self.compose_document() File "C:\Python38\lib\site-packages\yaml\composer.py", line 58, in compose_document self.get_event() File "C:\Python38\lib\site-packages\yaml\parser.py", line 118, in get_event self.current_event = self.state() File "C:\Python38\lib\site-packages\yaml\parser.py", line 193, in parse_document_end token = self.peek_token() File "C:\Python38\lib\site-packages\yaml\scanner.py", line 129, in peek_token self.fetch_more_tokens() File "C:\Python38\lib\site-packages\yaml\scanner.py", line 223, in fetch_more_tokens return self.fetch_value() File "C:\Python38\lib\site-packages\yaml\scanner.py", line 577, in fetch_value raise ScannerError(None, None, yaml.scanner.ScannerError: mapping values are not allowed here in "config.yaml", line 20, column 90 Press any key to continue . . .

[lurkbbs]

Some error in config.yaml, don't know what exactly, but it's on line 20, column 90. In Windows' Notepad and most other pads, you can press Ctrl+G and enter a line to go to. Alternatively, Edit -> Go To in the window menu.

[RedFoxNightfox]

okay, there were a few : marks in the file for some reason. fixed that and ran. now id does this:

[i] Running e621dl version 5.0.0.

[i] Checking for partial downloads...

[i] Parsing config... Traceback (most recent call last): File "e621dl.py", line 43, in default_days = config['default_search'].get('days', 1) TypeError: string indices must be integers Press any key to continue . . .

[Wulfre]

That one might actually be a bug. Did you leave the days option blank in your yaml file?

[RedFoxNightfox]

no, but I copy pasted everything from the old .ini did all the formatting change? let me get a blank one and see... oof. okay, yeah. that is actually on me for not checking that.

[Wulfre]

Okay. I didn't write the fallback option correctly. I'll have to fix that, but hopefully you're running for now.

[RedFoxNightfox]

how does the new blacklist work now? How do I format it?

[RedFoxNightfox]

also, logging into account does not fix hidden posts problem

[Wulfre]

blacklist:
   - tag
   - tag

[gotmilk0112]

I went through and installed all the stuff here, but it still just keeps giving some "yaml" error.

[Wulfre]

I'm doing more or less a rewrite to accommodate the features of the new API, rather than trying to patch together fixes. I'll be dedicating a good amount of time this weekend to getting it done.

[Wulfre]

After reading over the entire script again, I keep seeing how badly written it is. Hopefully the next version will be decent enough to maintain.