search

Bionus / imgbrd-grabber

Very customizable imageboard/booru downloader with powerful filenaming features.
https://www.bionus.org/imgbrd-grabber/
Apache License 2.0
2.55k stars 216 forks source link

Current version downloading 0 byte files #630

Closed Falord closed 7 years ago

Falord commented 8 years ago

What steps will reproduce the problem?

Well, all I did was try to open a favorites or two, once this version of grabber was opened or even after waiting a couple minutes and trying to get atleast one to load, repeatedly if necessary.

What is the expected behavior? What do you get instead?

I did this up to version 4.7.0 since that was the version I last had working on my system and it caused no problems and would all I clicked near immediately. Once I clicked a tag in this version, in this case my favorites, the program would freeze for a few seconds and open it in a tab. Once opened, attempting to either reload that tab or open new tags well freeze the program again or just crash it.

What version of the program are you using? On what operating system?

Version 4.8.2 this happening on. I'm on Windows 8.

Please provide any additional information below

This is more off topic, but I've notice that the more recent installers no longer give the option to pick your location to install grabber and uses the last set location from my old installs. Can the option be re-implemented?

One1One1One commented 8 years ago

Just look at the RAM usage. At about 1.900MB it will crash. That happens if there are many image search results in the open tabs (in total) and if the download query is very long. It reminds me of the 1.900MB margin that crashed firefox in the past, when it was only available in its 32-bit version. But in this case 64-bit wouldn't help much. Somewhere there is a lot of memory wasted. Unfortunately I'm not a C++ dev so I can't help much.

If you want to pick a new location, just uninstall the old installation before installing the new version. If you didn't do the portable method, than the old settings should remain in C:\Users\UserName\AppData\Local\Bionus\Grabber after uninstalling (at least it did so for me).

Bionus commented 8 years ago

About crashes, as said on issue #618 this version introduced a few big leaks that I'm currently working on. I'll upload a new version as soon as I get satisfying results. About the freezes, it's also related to the leak issue and I have already a fix on develop, so it should also be fixed as soon as the new version is released.

Sorry about this, this new version introduces some big internal changes, which caused these leaks to appear. I'm currently working on it as fast as I can to get it fixed, so it shouldn't take more than a few days.

This is more off topic, but I've notice that the more recent installers no longer give the option to pick your location to install grabber and uses the last set location from my old installs. Can the option be re-implemented?

I didn't change anything in the installer, so I guess it just doesn't prompt for the install folder if you have it already installed on your system. So as MrCrackR said, uninstall first. If you want to have multiple versions installed at the same time, you'd be better off using the portable mode.

Just look at the RAM usage. At about 1.900MB it will crash. That happens if there are many image search results in the open tabs (in total) and if the download query is very long. It reminds me of the 1.900MB margin that crashed firefox in the past, when it was only available in its 32-bit version. But in this case 64-bit wouldn't help much. Somewhere there is a lot of memory wasted. Unfortunately I'm not a C++ dev so I can't help much.

That's also my conclusion for all these crashes, that's why I'm currently re-doing most of the program's memory management. There's currently a 5mo leak per page since version 4.8.0 (my bad), and there has been a 200ko leak per image since a long time ago (causing the crashes on big batch downloads).

Bionus commented 8 years ago

After a few fixes, I managed to get rid of the 5mo/page leak, and reduce the old 200ko/image one to around 60ko/image.

Bionus commented 8 years ago

With the new version, it's still not perfect but already a lot better, as you can see below (stats were done with the Windows task manager on windows 10, using version 4.8.3).

Sorry for the inconvenience! I just uploaded version 4.8.3. Please tell me if it fixed your issues, so that I can know if I still have to investigate or not.

Download of 2,000 images

10 pages and 2,000 images took 65 mo to store in memory (32 ko per image). Each image downloaded "leaked" 13 ko. In the end, 45 mo were leaked.

Download of 20,000 images

100 pages and 20,000 images took 440 mo to store in memory (22 ko per image). In the end, 220 mo were leaked.

What can be easily noticed is that except for the initial storage in RAM of the images' data, the download itself did not need much more even though we had 10 times more images. The further in the download we were, the less it ate RAM (even getting some back each time it recycled the already downloaded images).

In conclusion, there shouldn't be issues as long as you don't go above 100,000 images for now (after which the storage itself of the data will eat all your RAM).

Falord commented 8 years ago

You didn't change the installer? That's strange. But from version 4.5.3 back, still ask you to pick an install location regardless of if it was already installed. Most be the older versions were bugged for still asking then!

Falord commented 8 years ago

Just checked version 4.8.3. It no longer freezes and moves like normal without crashes, but no longer downloads. It just ignores all in batch downloads. A matter of fact, nothing saves in the new version. Even using 'save' or 'save and close' for me on the image itself produces a file with 0 bytes.

One1One1One commented 8 years ago

For me it now works perfectly except for images from the source rule34.paheal.net, where it doesn't save the images but some HTML-files. just an example for such a file

Edit: I can confirm now as well, that there are some files, that end up as 0-byte-files on my HDD. Not all of them, but a few.

Bionus commented 8 years ago

For me it now works perfectly except for images from the source rule34.paheal.net, where it doesn't save the images but some HTML-files.

Yeah, they broke their RSS API and for some reason it's above the Regex API in the new version. A workaround is to change the site's source order to put the Regex on top (requires a restart).

version 4.8.3 [...] no longer downloads. It just ignores all in batch downloads. A matter of fact, nothing saves in the new version. Even using 'save' or 'save and close' for me on the image itself produces a file with 0 bytes.

&

I can confirm now as well, that there are some files, that end up as 0-byte-files on my HDD. Not all of them, but a few.

I'll investigate, but I think some logs may help for this, until I can reproduce the issue. Is it totally random? (all of Falord's image seem to break, while only a few of MrCrackR's have this issue)

Falord commented 8 years ago

The batch downloads are work now. After restarting my computer they started working. But all images that are saved using "save" and "save and close" all produce 0 byte files for me still on version 4.8.3. Hope these are the right logs.

main.zip

One1One1One commented 8 years ago

The only error I get is:

[17:33:33.254] Error parsing RSS file: letter is expected (4 - 53). [17:33:33.254] Loading using Rss failed. Retry using Html.

This log is written, when 0-byte files are created:

[17:41:13.950] Loading image from http://simg4.*booru.com//images/86/9d/*.jpg [17:41:14.156] Image received from http://simg4.*booru.com//images/86/9d/*.jpg [17:41:14.156] Saving image in D:\Files**booru.com\3192782.jpg

They are only created when directly viewing them inside Grabber (on every source), not when downloading them with the function in the Downloads-tab. They show up in the Grabber window, but aren't saved to the HDD correctly.

ghost commented 8 years ago

I'm also having this issue while testing with safebooru.

screen shot 2016-10-19 at 12 34 08 pm

Bionus commented 8 years ago

The issue should be fixed in the develop branch, and the fix will be included in the next release. :smile: