subfolder blacklist

[huskelord]

I was wondering if there was a way to make certain subfolders be blacklisted form others because something i tried to do was make my cyborg subfolder have it own subfolders for each type of species like alien, canine, etc. but unless I added the tag cyborg it would download the full species subfolders but if i did add the tag I would get ..Animated and ..Images added to it since it has a tag and thus the default subfolders would be added to it. bassicly what i was wondering is if you could ad a "blacklist_default_subfolders" section that i could add to get rid of that problem or just a "blacklist_subfolders =" to each search that I don't want to have the default subfolders added to even if I have a tag in the search if this is not possible or too much of a pain to do don't worry about it i currently just have the cyborg and any of the other ones i would like to have just be on their own without any subfolders

[lurkbbs]

Yes, I can, and that's not that hard. Just a reminder, if none of the subfolders are matched, images will be downloaded to a root folder. During this week it'll be done.

There is more general question. And as I see it, current config format is ill-equipped for hierarchical structures. Everything feels hacked.

So, I'm curious, what do you think of Hocon? It looks like a good alternative, with a bit more of braces but without excess quotes like in TOML

YAML also looks promising.

Also, maybe you have some task-specific format in mind. Or maybe you think everything is fine as it is.

Let me know what do you think. Also, can you give me some basic config that has everything you want in a format you want, and I'll convert it to YAML, TOML, HOCON and JSON. To be more specific, to HOCON, and then to all other using hocon tools. Just want to choose the most optimal config format

[lurkbbs]

Also, given that I want to make a quick fix solution but it's a pain to make an executable, can you install Python and all required modules? It's not that hard. Easiest way is with Chocolatey

Just open admin Powershell, and type this:

Set-ExecutionPolicy Bypass -Scope Process -Force; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))
choco install python

Then open Powershell not from admin, and type this

python -m pip install -U pip
pip install requests colorama

Line by line meaning:

Set-ExecutionPolicy Bypass -Scope Process -Force; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))

Installs Chocolatey.

choco install python

Installs python

python -m pip install -U pip

Upgrades python module installer

pip install requests colorama

Installs required modules.

I'll give you both executable and python script, with multiple_configs_py.bat and e621_noclose_py.bat to launch source code version. Try to install it and use "*_py.bat" versions, this way it will be a lot easier to provide hotfixes. e621dl.zip

to disable default subfolders, add blacklist_default_subfolders = true to a section, like this

[*subfolder]
...
blacklist_default_subfolders = true

Also, no more empty folders.

Finally, you may have a question why I even bother with hotfixes and trying to do it fast. It's simple: you are the only person who don't ignore bugs and who tells me what to improve and to add. And I want to make e621dl with less bugs and with more useful features. The fact that someone find it useful and want to see it better, it "fills me with determination".

[huskelord]

i dont know/understand what TOML YAML AND HOCON are so yeah but this is what i currently have for a configs is this (i use the multiple cofig)

3 Config Data (API).txt 4 Config Data (DB).txt

with the api downloaded first to get the images and the db to further sort the images and all that without taking up all the space i have by downloading to much as for the debugging stuff ill try and do that later but i wont be able to do it for a while since i have some finals stuff to take care of and all that.

Edit1: i should probably mention that i don't know code so i dont know how much me installing python will do since i have little no idea what to do with it. but i'll try my best to help

Edit2: bug time i run the program and this showes up Traceback (most recent call last): File "e621dl.py", line 594, in File "e621dl.py", line 476, in main File "e621dl_lib\local.py", line 540, in get_files_dict ValueError: invalid literal for int() with base 10: 'desktop' [16104] Failed to execute script e621dl

i don't know why it happens withh both the api and db configs

[lurkbbs]

For the error, there is some file that looks like "sometext.desktop.ext", most likely desktop.ini

For now, either delete downloads and recreate all from cache or open Powershell in downloads and run

Get-ChildItem * -Include *desktop.* -Recurse | Remove-Item

It is a bug, the app assumes that all files in all folders looks like "anything.number.ext" or "number.ext". I never thought it will be a problem, but I forgot about "thumbs.db" or "desktop.ini" and other Windows hidden files. I'll fix it sometime next week.

I'll look into configs, and my version of how it can be written in different formats.

[huskelord]

that first bug got fixed but i got a new one when i tried to add a new config to the group this is what i got when running it on noclose Exception in api iterator: Traceback (most recent call last): File "e621dl.py", line 145, in prefilter_build_index File "e621dl_lib\remote.py", line 226, in get_posts File "site-packages\requests\models.py", line 940, in raise_for_status requests.exceptions.HTTPError: 522 Server Error: Origin Connection Time-out for url: https://e621.net/post/index.json

and this is what i got when running it on the mulit congi Traceback (most recent call last): File "e621dl.py", line 594, in File "e621dl.py", line 303, in main File "e621dl.py", line 303, in File "e621dl_lib\remote.py", line 277, in get_tag_alias File "site-packages\requests\models.py", line 940, in raise_for_status requests.exceptions.HTTPError: 522 Server Error: Origin Connection Time-out for url: https://e621.net/tag/index.json [37680] Failed to execute script e621dl

these ar what i have for my configs right now. 3 Config Data Artist (API).txt this one is for downloading everything that an artist has posted and to have the downloaded pools when that gets added

4 Config Data Misc (API).txt this one is too download everyting else i want downloaded form the source

5 Config Data ALL (DB).txt this one I don't know if will be run by the multi config but will basicly sort everying in the database into its designated forlder outside of the ones used for the main download and tho prune anything that may need pruning when I feel it is necesarry

I don't know howmany configs there are in the multi config or how to change that so if you could do that or tell me how I would appreciate it.

Edit1: I'm not sure but i think this happens during the tag verification but im not possive

[lurkbbs]

Errors 5xx means problems on e621.net. This particular error means Cloudflare (protection from DDoS) could not connect to e621.net servers. Whenever 5xx error happens, try check the site first, maybe there's some maintenance at the time.

[huskelord]

Got it. thank you. Do you know if multi config will be able to go with three configs as is or will I need to change something.

[lurkbbs]

try open multiple_configs_py.bat with "right click --> edit" or "open with --> notepad", it should be like this:

@setlocal enableextensions
@cd /d "%~dp0"

copy /y config1.ini config.ini
e621dl.py

copy /y config2.ini config.ini
e621dl.py no_new_files_reset

pause

you can just add another lines, like this:

@setlocal enableextensions
@cd /d "%~dp0"

copy /y config1.ini config.ini
e621dl.py

copy /y config2.ini config.ini
e621dl.py no_new_files_reset

copy /y config3.ini config.ini
e621dl.py no_new_files_reset

pause

This script basically replaces config.ini with whatever config you like, config.ini in our case, and then starts e621dl after every replace. "no_new_files_reset" means list of newly downloaded files won't be reset. To skip pruning, you should only use it in the last config.

So, you can easily add new configs, you can even rename these configs.

[huskelord]

This should be all for this issue but i have one final question ran accross post 2050646 and a few others that gifs won't open unless i open it in internet explroer or that wont even move even though they are an animatioin Edit1: i also noticed that some animations seem to be slower or mor janky than thier originals would there be a way to fix that or a way to have it completly deleted from all files so it can be redownloaded

[lurkbbs]

Heh. That's just GIF being GIF and the ultimate reason why Google uses WebM whenever possible. So, first, not moving at all. That's because your image viewer either doesn't support gif animation at all or having problem with some files. Second, slowness, is also because animation speed is rather unclear and gif files uses different ways to set it, and some viewers don't support it correctly. But there may be another reason. Some viewers decode gifs on the fly for every frame, again and again, thus making fast animation hard or impossible. Others decode every frame first and then begin animations to play.

There are a lot of good viewers on the Net. Currently, I use HoneyViewer, but occasionally I use Imagine by Chun Sejin. Also, however strange it may sounds, some video players like SMPlayer or MPC-BE supports images as well. With ActiveX flash plugin, MPC-BE supports flash as well, but not as good as dedicated flash projector.

If you want folder view, XnView and Imagine are good enough. If not, HoneyView looks best.

[huskelord]

i have a question. does the purging happen before after or during the programs run? Edit1 also would there be a way to make a config or comand that I could put into a config that will make it only purge but not doanload anyting Edit2: i also think the blacklits_default_subfoldrs = True might be getting rid of the main blacklist for some posts but i can't be sure about that.

[lurkbbs]

does the purging happen before after or during the programs run?

Right after last post being processed and before cache pruning.

a config that will make it only purge but not doanload anyting

Why? I mean, pruning should be after last file is placed into folder, therefore it must be activated in the last config. If it's a question of convenience, you can make another config without any searches, just "Settings" sections with pruning options enabled. By the way, new multiple config solution will be available soon, this time pruning will be after all configs are processed if at least in one of them these options are enabled.

I also think the blacklits_default_subfoldrs = True might be getting rid of the main blacklist for some posts

Nope. Just checked, global blacklist adds unconditionally. If a post looks like it should be blocked, check manually if it has blocked tags, just go to https://e621.net/post/show/<post id> to see. I also recommend to make an account on e621 and appropriate tags by yourself. Also, you can PM me there, username is lurkbbs, just like here.

[huskelord]

doesn't pruning need serches to know what to get rid of or move because whenever i prune say with my artist config it will get rid of anything in my misc and db config since the serches are not in my artist config. the main reason im asking is because of how i have my config setup.

First I have a artist config wich downloads everyting (safe qestionable and explicit) that isn't blacklisted from the artist listed (i also plan to put whatever comand i need for the pool downlaods here and not anywhere else) 3 Config Data Artist (API).txt

then i have my misc which will download only explicit post of certain tags so i have a larger db to work with 4 Config Data Extra (API).txt

finaly I have my db which will seperate all explicit posts that i have downloaded by species and some other tags without downloading a whole bunch of other things that i don't have space for the thing is i dont't want the safe and qestionable add to this though only the explict. 5 Config Data Species + Misc (DB).txt

that's why i can't have my last one (the db) prune every time since i would have to redownlaod the safe and questionable post for the artists and i can't just make my db have safe and qestionable because i don't have that much room to work with considering the amount of species tags im working with.

Also could you explain what you mean about the new mulitple config solution like what is it going to be.

[lurkbbs]

First, about configs. Why do you not use cache?

Second, about needs to know. e621dl.py no_new_files_reset means e621dl won't forget previous downloads when it starts new config. But just wait a bit, it will all become irrelevant.

that's why i can't have my last one (the db) prune every time since i would have to redownlaod the safe and questionable post for the artists and i can't just make my db have safe and qestionable because i don't have that much room to work with considering the amount of species tags im working with.

If you tried and it didn't work, that's a bug. Luckily, this will be irrelevant now.

Also could you explain what you mean about the new mulitple config solution like what is it going to be.

Instead of one config file, there will be "configs" folder, and all files with '*.ini' extension will be used as configs, processed in Natural order. So, no more "multiple_configs.bat". prune_downloads and prune_cache will be activated after all configs are processed if set to true in at least one of them. Also, no long wait to check all those similar tags. Finally, downloading will be resumed right where it was interrupded if e621dl closed with x button on a window, or if Windows decides it want to reboot.

[huskelord]

................... ALL HAIL THE GENIUS

also i don't use cache because space but now that i think about it thats a stupid reason since alot of the images will be copied alot anyways so why not one more. plus it will help with cleaing space if I can go through all the animations at once. I'm an ideot. I thought i was being smart.

Also Found a bug Ok i don't know what the problem is exactly but i encounted somthing that worries me in my incest main folder i have images that should be in subfolders but aren't. post REDACTED should be in REDACTED but for some reason it's not in there but in the main folder the main thing that worries me is that the main forlder (REDACTED to be spacific) doesn't have a tag on it its just a placefolder for all the REDACTED tags. this happens in some other folders as well.

(this is what i made me think that blacklist_defaut_subfolders = true caused it to not be affected by the main blacklist because the images looked like they had some tags i blacklisted they probably didn't but i blacklisted them indavidualy so i can't check.)

I'm going to delet them and run the program agian to see if it will fix it.

[lurkbbs]

And done!

OK, one again about cache, copies and hardlinks.

Say, you have a 1MB file a.jpg and its copy, 1MB file b.jpg. They will take 2MB space. Now, say you have 1MB file a.jpg and its hardlink ah.jpg. They will take... 1.00001 MB of space. That is, still 1MB of space.

How does it works, for copies: Take our hard drive (or ssd, no matter). Files are stored like this:

[address to where bytes of file lies, filename]
.....
[actual data of the file]

So, for copies, it looks like this

[address of byte of a file "a.jpg", it's filename "a.jpg"]
[address of byte of a file "b.jpg", it's filename "b.jpg"]
.....
[actual data of a file named "a.jpg"]
[actual data of a file named "b.jpg"]

For hardlinks, it's like:

[address of byte of a file "a.jpg", it's filename "a.jpg" and it's another filename "ah.jpg"]
.....
[actual data of a file named "a.jpg" and also "ah.jpg"]

As you can see, with hardlinks there is no additional copies, an small size increase for storing additional filename.

And that means, as long as all your configs have make_hardlinks = true, all the copies have zero size and didn't take space at all.

As for post, please edit it to leave any NSFW details to e621 inbox, and post it all there. I'm not really comfortable with discussing it here on Github.

I'll look into this tomorrow.

[lurkbbs]

Oh, And while I'm here, give me configuration with the problem, and if you get a working solution, that with that. Just pack it into zip file and drop here.

[huskelord]

5 Config Data Species + Misc (DB).txt I know its not a zip but i don't know how to make them.

[lurkbbs]

For built-in Win 10 zip compressor you can select files and folders, then right-click --> send to --> compressed zip folder.

And thanks, if it's only with this file, I will find this bug and will kill it a bit more quickly

[huskelord]

thanks for the info as for the bug I have seen it in other places i think but I could be wrong i don't know I'm doing a complete redownload so i can have a complete cache again so if it shows up again ill know if its just a random bug or a repeat offender and i'll let you know. Edit1: I also REDACTED any nsfw stuff

[lurkbbs]

Thanks again!

[huskelord]

almost finished the redownload stuff but the images that are going into the wrong areas are aperarig again same ones in the same places Edit1: also how long would you say it will take for you to finish the config forlder update? not to sound impatiant but i'm really excited for it since it will help alot

[lurkbbs]

how long would you say it will take for you to finish the config forlder update?

About minus one day, I suppose. As always, my communication skills fails me, it was done yesterday, shortly before "And done!" message. So, hint: hit "watch" near "Star" button and select "Releases only"

So, problem stays with that post, for these three configs. I'll look into it starting now

[huskelord]

couple of questions that i have.

what happens if the program runs into an error will it just move on to the next config file like the multi config or will it just close (im assuming it will just stop and not move on with the noclose bat)

if im adding post to the blacklist folder while the program is running does it add those posts to the blacklist folder before pruning it or do i have to restart the program to get those files added to the blacklist program (i think this was ansered this before or you asked me what would be best for it but i forgot and don't know where to look)

will it be possible to have the program prun itself if it runs into a no space error and then attempt to run again. (this is something that i think might be helpful for people with little space to store stuff like me)

[lurkbbs]

what happens if the program runs into an error will it just move on to the next config file like the multi config or will it just close

Just closes. But, after the error is fixed, it will continue from the last uncompleted config.

if im adding post to the blacklist folder while the program is running does it add those posts to the blacklist folder before pruning it or do i have to restart the program to get those files added to the blacklist program

For simplicity, assume you need to restart. For now, post blacklist updates when a config changes, but not during the config being processed. That also means that on previous config posts are unblocked and could be downloaded anyway, and will not be removed after all configs are done. But this can be changed anyway.

will it be possible to have the program prun itself if it runs into a no space error and then attempt to run again. (this is something that i think might be helpful for people with little space to store stuff like me)

Nope. I don't know how choose what to delete and what not in the middle of a program run. By the way, how large is you drive? Maybe you should buy external 1TB HDD, you config really needs a lot of space.

[lurkbbs]

There are errors in at least 5.Config.Data.Species.+.Misc.DB.txt, details in e621 messages. I think you should check if there are some other errors.

[huskelord]

Got it thanks. As for the external harddrive it's something I'm saving up for to be sure. One of the reasons I asked for a the favorites function so I can have all my favorites on my main harddrive and the miscilanious on the external.

I see the favorites working like the blacklist does with dragging the posts I want to favorite and the names get added to a text document but instead of blacklisting anyting that got favorited can be downloaded specifically. Like in default setting there can be favorite_posts_only = true/false command with if it's true it'll only download posts named in the favorites document in the file format of the all config files in the folder like the prune command.

If this is something you think you would like to add let me know and I'll make a new issue for lit like the download pools issue so this one can be closed since the original issue has been fixed for a while now.

[lurkbbs]

I see the favorites working like the blacklist does with dragging the posts I want to favorite and the names get added to a text document but instead of blacklisting anyting that got favorited can be downloaded specifically. Like in default setting there can be favorite_posts_only = true/false command with if it's true it'll only download posts named in the favorites document in the file format of the all config files in the folder like the prune command.

I have another proposal. Since hardlinks cost zero space, how about I add favorites folder in addition to cache and downloads, and copy (Edit: not download) all your favorites there. With option like make_favorites = none/only_files/folder_tree to chooses if you need to create folder structure like in downloads or just plain files. And then use some of file sync software to sync your external drive with internal.

[huskelord]

that would work thanks. Edit1: im assuming there would be a folder like and text document that would work like the blacklist folder and text document but for favoriets. if not how would you designate what is a favorite or not.

[lurkbbs]

somewhat like with blocked posts, two folders: to_favorites to add favorites to favorites to store added files

one file favorites.txt to store favorite ids

[huskelord]

got it thank you.

[huskelord]

Opened a new issue that i think youll want to take a look at.

[lurkbbs]

Thanks for notifying, but there is no need in the future. Any new issues are sent to my email which I regularly check. Same for e621.net messages

[lurkbbs]

OK, I'm closing this Issue for now