Bionus / imgbrd-grabber

Very customizable imageboard/booru downloader with powerful filenaming features.
https://www.bionus.org/imgbrd-grabber/
Apache License 2.0
2.48k stars 216 forks source link

Crashes on Reload from .igl files (other stuff too) #473

Open SultrySamthepennanceman opened 8 years ago

SultrySamthepennanceman commented 8 years ago

For whatever reason, it seems downloading from a source, saving the session as a .IGL is fine, but it seems like when you download from the igl, save, then overwrite the .igl after your download is completed the program crashes.

Additionally I'm getting crashes from sources that seem have some sort of pattern with them either being 1 page and 1-any single number digit images, but mutiple ones aren't uncommon either.

Also, Can you make it a bit more obvious as to how to place login details to various sites the program uses? I can't seem to tell if my usage of Gelbooru or Sankaku accoutns acctualy work because I keep getting an unknown response or something along those lines to using the test button.

It's seemingly a bother, but can you also offer ways to order downloads on the link list other than move up/down, I feel that it needs to have the option of beign moved by holding the left cursor over the link so it can be moved up and down the table to the users desire, and it should also have basic arrangements like by name, or lowest to highest, images per page/on page whatever to highest to lowest, as something like that as an option for users could reduce crashes with links that have fewer pictures to download which seems so common now.

Additionally it's been botherign me for a while, but this thing needs to able to reallocate a destination fodler to a link without having to input the danm thing manually.

Lastly, when loading a .igl list, it seems that it will ALWAYS direct you to the grabber directory folder, and does not remember any other places you would save .igl files in, it doesn't have a memory.

Whilst the addition of this feature would be nice, what would be better is if in the event of loading and saving .igl files, or even setting the destination folder for images if we had a similar drop down menu for commonly, remembered or even saved locations for directory folders we wish to save content to, rather than having to manually change this everytime we start up a session of grabber.

Bionus commented 8 years ago

I'll try to answer paragraph by paragraph as there's a lot of content here.

  1. Indeed, thanks for reporting this.
  2. Is it easily reproducible? If so do you have some IGL or example so that I could? I haven't met any crash these days. Which source are you using?
  3. What do you mean unknown? "Unable to test"? If so it's because the test procedure hasn't been added for all sources. They are to be set in the last time of the source's settings (url, fields, cookie, etc.) Also, not all kind of login can be tested currently. For example, GB's settings are:

    [login]
    parameter=false
    method=post
    url="http://gelbooru.com/index.php?page=account&s=login&code=00"
    pseudo=user
    password=pass
    cookie=user_id
  4. You mean drag and drop and column sorting?
  5. I don't quite understand this part.
  6. What about remembering the last location, same as for the "Save as" button? Do you have many folders where you store igl files?
SultrySamthepennanceman commented 8 years ago

2.If you want, I can give you my sources, and you can test with them on grabber, so you can try to reproduce it.

Crash works by:

Search two tags, Get all, save it as .igl, download, get a crash, wait for a restart to reopen session, download from each source usind "download Selected" instead for each source until you spot the one that causes the crash, use "delete selected" to remove it, then save over the igl again. if you crash on downloaded the non working link, restart, then delete it, but basically it's about manually removing links that seem like they won't work based on the pattern of the crashes being from the larger default sources you offer with grabber that are usually rule34.booru, Phael, and (Not default but worth a mention) Rule34hentai.net where it seems to happen the most.

Repeat this, and you might get a crash.

You might also notice it happens with sources that my be false (Positives?) in that they have 1 page and what is either an absurd ammount of images, or too few around single digits.

4.Yes, but with the ability to arrange which order they are downloaded in when you press "download" other than "Move Up" and "Move Down" like how you would go to your folders toolbar in your documents, View, Sort By and chose the way they are arranged, such as alphabetical, numerical, and whatever else suits the purpose of prioritising downloads from sources that have the most pictures to offer to reduces crashes caused by sources added that are either not downloading due to some error, or because the program thinks it has images based on the tag you searched, but it really is just the entire site's pcitures which casues a crash, or something along those lines, (happens with rule34 Phael most of the time, but it's easily noticable and removable via using delete selection on the link so when you click "download" to batch download it won't pick up the source and crash because of this).

I also forgot to mention this, but the sources menu needs a search filter so people can find the sources they want to downlaod from with ease to select, and it could also do with some sort of source grouping feature, where people can load a list of active sources/deactive sources they place under a group to search from (I.E person wants to Download from only the defautl sites added to grabber sources but doesn't want to go through all the sources he's added to find them individually and select them, so he can press a "default" button to select the tick boxes for the default Grabber sources turning the other ones off.

The destination folder on the Downloads tab, no matter what starts of in Program files/Grabber, every session, it doesn't remember where you last saved a .IGL file, so you haveto navigate out of that constantly to where you saved your IGL.

I keep my .IGLs in the files they are coresspondant to, which often have the Common Structure of:

Subject/

Folder G Folder H Folder TS Folder L Folder S Folder B

And the .Igl will be here, but for the most part, I keep them In folders located in "My Pictures." I have to navigate to my pictures everytime because I start in The grabber Directory folder.

As for the Downloadthemall! thing: Downloadthemall has this feature where it seems to pretty much remember folder directories you've saved things to frequently, and keeps them without any type of expiry unless you download a buttload of things to a new directory you havn't before, which is done through it's dTAoneclick tool, but to to cut things short, It always remembers my Artifical Acdemy 2 folders for where I save Eyes, Male/Female Students anf Highlights.

Essentially, I want to be able to have a drop down menu that has a list of saved (And removable) directories that I designate As my destination folder for my pictures downloaded, as well as the same thing for saving .igl files.

Additionally, the Destination for downloaded pictures on the Downloads tab "C:\Users\User\Pictures(Destination folder) needs to be able to be changed both manually and by way of a choose folder window.

Just to get the across to you, when you go to the downloads tab and click folder and it all get's hilighted, one should be able to rewrite the filepath manually if they wish.

Additionally, (I think) doing this manually works or it just ends up whereever the default destination folder is regardless if you changed it, I forgot.

The other way is through a chose folder window where the user changes where he wants the links and their downloaded images to go to

Also, when I updated the Imageboard grabber, I forgot at what version this oocured but all images used for tags I favorited seemingly turned to red X's for some reason, might just go away if I delete and replace the tags, but you might want to test it by adding a bunch of favorites, and installing an update to see if it breaks them.

lastly, There seems to be an issue with the assignment of icons to sources on the sources window, as I could have sworn ages back when used this program a few of the sites I added to sources had their own icons, and I think this occured sometime when updating the program, but, all of my sources added all shapeshifted into Gelbooru icons.

Didn't break anything but it does hurt identification a little, It should be checked In possibly the same way as I suggested for favorite tags.

SultrySamthepennanceman commented 8 years ago

Also, I noticed that scroll-clicking on tags on the tags tab doesn't open it up as a new tab on the search results area, which should be a thing, because reasons.

SultrySamthepennanceman commented 8 years ago

Also, there seems to be some sort of odd Variation when usign the program with a proxy.

When using a proxy, Images that would normally show up on grabber to be clicked on for preview do not show up in merged view, unlike when you use a proxy, and it seems that most sources which I download from that normally crash when I'm not using my proxy miraculously download.

I have a small belief it has something to do with the download speed, as it's possible that crashes I've been getting have something to do with how the program interacts with the booru's it downloads from, and the proxy slows it down a little, (I also use privoxy for my web browser and on a few of the more used Booru sites I've noticed it can interfere with stuff like the ads for example.) It's just a guess though, but considering I'm testign this to the limite by using the tags, animated, animated_gif, webm, mp4, .mp4, sfm, and Source_filmaker and I appear to be in the process of sucessfully downloading what Danbooru has to offer with no crashes so far, I might be on to something here.

I have no idea why this occurs, but it does.

I use Privoxy, (Port 8118) 127.0.0.1 HTTP

Additionally, I've found out about the favorites thing not showing the images for them. It seems that when you right click on a tag shown on a previed image and save that tag to favorites, you get the image previewed associated with the one you saved, but this doesn't apply to favorites you might have added by putting them into the search bar and favoritign by highlighting them and adding them to favorites.

When it's done through the search bar or the tags tab (Soon to be anyway) it should pick either a random image, or most popular image under that tag (Making the image thumbnail for the favorited tag choice optional?)

Still doesn't fix the loss of the image on an update, but it solves a few things.

SultrySamthepennanceman commented 8 years ago

Also, found another problem, the use of the "Move Up" tool works fine with a selected group of sources, on the downloads tab, but the "Move Down" one doesn't move these down correctly, and leaves a space per every time it is clicked with a selected group of sources.

In case you didn't get that, search something, get the page, Hold Crtl and select a number of souces on the Downloads page, move them up and move them down, and you'll noticed they weren't moved in order.

Bionus commented 8 years ago

First, sorry for not replying earlier, I read your message but I was pretty busy and wanted to give a more complete reply.

the use of the "Move Up" tool works fine with a selected group of sources, on the downloads tab, but the "Move Down" one doesn't move these down correctly, and leaves a space per every time it is clicked with a selected group of sources

Indeed, the order in which the lines were being moved down was wrong (it was the same as the one for moving the items up, which is incorrect). I corrected it, thank you. :smile:

When using a proxy, Images that would normally show up on grabber to be clicked on for preview do not show up in merged view, unlike when you use a proxy, and it seems that most sources which I download from that normally crash when I'm not using my proxy miraculously download.

From what I understand from your previous comment (about proxy), when you're not using a proxy, it crashes? Could you give me a way to reproduce this? (a tag and source that usually cause it) Or at least your log after the crash? I never had any problems downloading from the main sources.

When it's done through the search bar or the tags tab (Soon to be anyway) it should pick either a random image, or most popular image under that tag (Making the image thumbnail for the favorited tag choice optional?)

What do you mean soon to be? From a search that has been done (and the thumbnails loaded) I could indeed take a random thumbnail in the results and use it as the favorite's image. For other cases, I'll add it to the backlog but I don't think it will be done soon since it's not really a major issue (from my point of view, feel free to tell me wrong).

As for the Downloadthemall! thing: Downloadthemall has this feature where it seems to pretty much remember folder directories you've saved things to frequently, and keeps them without any type of expiry unless you download a buttload of things to a new directory you havn't before, which is done through it's dTAoneclick tool, but to to cut things short, It always remembers my Artifical Acdemy 2 folders for where I save Eyes, Male/Female Students anf Highlights.

I thought it was a Firefox feature? Anyway, I believe it works by associating domains and extensions to their last saved location. For example, it will remember where you last saved PNGs from somesite.com and automatically save them there next time. This kind of behavior wouln't work with Grabber.

Essentially, I want to be able to have a drop down menu that has a list of saved (And removable) directories that I designate As my destination folder for my pictures downloaded, as well as the same thing for saving .igl files.

Isn't that overkill? From what I understand, they are all in the same location, what about remembering the last saved location and automatically go there when saving new IGL? (763336626940c1a8b0236d9008c42fa9cda72faf)

Additionally, the Destination for downloaded pictures on the Downloads tab "C:\Users\User\Pictures(Destination folder) needs to be able to be changed both manually and by way of a choose folder window. Just to get the across to you, when you go to the downloads tab and click folder and it all get's hilighted, one should be able to rewrite the filepath manually if they wish.

You can double click on any cell or start typing when a cell is highlighted to edit its contents. But there indeed won't be a folder window there, just plain text editing.

Additionally, (I think) doing this manually works or it just ends up whereever the default destination folder is regardless if you changed it, I forgot.

I'm not sure what you mean? I just tried changing a destination manually and start a download and it went to the right location.

Also, when I updated the Imageboard grabber, I forgot at what version this oocured but all images used for tags I favorited seemingly turned to red X's for some reason, might just go away if I delete and replace the tags, but you might want to test it by adding a bunch of favorites, and installing an update to see if it breaks them.

Seems like your favorites thumbnails got deleted by the update, I'll investgate this.

Also, I noticed that scroll-clicking on tags on the tags tab doesn't open it up as a new tab on the search results area, which should be a thing, because reasons.

Indeed, it's been done for next version (commit 605963de9a47fa8eeac0f292eae8e108ac288c9d).

lastly, There seems to be an issue with the assignment of icons to sources on the sources window, as I could have sworn ages back when used this program a few of the sites I added to sources had their own icons, and I think this occured sometime when updating the program, but, all of my sources added all shapeshifted into Gelbooru icons.

It's not normal, sources should have an icon reprensenting the kind of boards they are derevied from (shimmie, danbooru, gelbooru, etc.): sources Does all sources work properly except from that? Are the icon.png in the "%localappdata%/Bionus/Grabber/sites" correct?

SultrySamthepennanceman commented 8 years ago

From what I understand from your previous comment (about proxy), when you're not using a proxy, it crashes? Could you give me a way to reproduce this? (a tag and source that usually cause it) Or at least your log after the crash? I never had any problems downloading from the main sources.

Using a Proxy, (Or at least privoxy) makes it so that search results you'd be able to clearly see the thumbnails on the results page for cannot be seen, (Possibly not picked up from or loaded) and additionally, it seems that this occurs with some of the main default Grabber sources, but in some cases sources which cause the most crashes for me on either the spectrum of scarce or vastly large tags (how many images under the tag in question), and it seems that when using a proxy, I'm more likely to get sucessful downloads from scarce tags that commonly crash with the proxy on, compared to larger ones that crash without the proxy on.

I suspect that it isn't too responsive with the more popular sources because Sites like gelbooru have it's ads and other bits blacklisted/adblocked by Privoxy when it's in use, or something along those lines.

Basically, it seems that Downloads performed under Privoxy (Other proxies by extension?) seem to vary in sucess, which I found to be a bit odd.

When it's done through the search bar or the tags tab (Soon to be anyway) it should pick either a random image, or most popular image under that tag (Making the image thumbnail for the favorited tag choice optional?)

What do you mean soon to be? From a search that has been done (and the thumbnails loaded) I could indeed take a random thumbnail in the results and use it as the favorite's image. For other cases, I'll add it to the backlog but I don't think it will be done soon since it's not really a major issue (from my point of view, feel free to tell me wrong).

Try having a bunch of favorited tags that have no thumbnail, and after a while of non-use you come back to them and then can't remeber why you had them in the first place, you delete them, remember later, have to find them again, and then an update or something else gets's rid of the thumbnails again and you repeat the vicious cycle, etc Random is a good choice, but havign it choose the most popular image is better, because the thumbnail in question is more likly to be a memorable image for the tag if it's an artist for example.

I mean, with something like TheBoogie you'd always know what it was under your favorites via the thumbnail, but for things that are more obscure scarce or not so outstanding you'd recognise it anywhere, it becomes something of a mysters

Isn't that overkill? From what I understand, they are all in the same location, what about remembering the last saved location and automatically go there when saving new IGL? (763336626940c1a8b0236d9008c42fa9cda72faf)

Whilst that works too, keeping all the .igl files in the same place doesn't really work for organisational reasons, it works for ease of access though which is nice, (I mean you do have the option of opening the grabber source folder in My Programs which is a nice touch, but still)

I wanted the option of having the directory you save your content in to be saved so you could bring up the directory at leisure so you didn't have to hunt it down through a browse window because Some of the folders I use for Grabber Downloaded content are more or less there to stay to host the stuff I get, and It might be quicker to have a few saved locations I could easily click on on the Grabber program to switch to rather than hunt through the browse window to find them in view of vastly larger collections of grabber orientated files.

You can double click on any cell or start typing when a cell is highlighted to edit its contents. But there indeed won't be a folder window there, just plain text editing.

Additionally, (I think) doing this manually works or it just ends up whereever the default destination folder is regardless if you changed it, I forgot.

I'm not sure what you mean? I just tried changing a destination manually and start a download and it went to the right location.

Alright, sorry, might have worded that wrong, but basically text editing the source destination is nice and all, but you can't do it in a group like with folders on Windows Explorer and group rename, so yeah, being able to do that would be a godsend, AND doing it through a Browse: for folder window woud be a godsend for those files which are under longer and further to reach directories you want to save to when typing all the words for it just takes too long.

Also, when I updated the Imageboard grabber, I forgot at what version this oocured but all images used for tags I favorited seemingly turned to red X's for some reason, might just go away if I delete and replace the tags, but you might want to test it by adding a bunch of favorites, and installing an update to see if it breaks them.

Seems like your favorites thumbnails got deleted by the update, I'll investgate this.

Also, I noticed that scroll-clicking on tags on the tags tab doesn't open it up as a new tab on the search results area, which should be a thing, because reasons.

Indeed, it's been done for next version (commit 605963de9a47fa8eeac0f292eae8e108ac288c9d).

So will it show the same stuff as when you would right click on tag text in an open preview image window? (That's what I was getting at, not too sure if I got that across)

lastly, There seems to be an issue with the assignment of icons to sources on the sources window, as I could have sworn ages back when used this program a few of the sites I added to sources had their own icons, and I think this occured sometime when updating the program, but, all of my sources added all shapeshifted into Gelbooru icons.

It's not normal, sources should have an icon reprensenting the kind of boards they are derevied from (shimmie, danbooru, gelbooru, etc.)

Yeah, I'm missing the original icons for my default sources, most use Gelbooru's or Danbooru's at this point.

Also, its been bothering me, but E621 has a sister SFW Site, I was wondering why that hasn't been added to grabber yet? ( I tried, but failed in the attempt to do so manually)