Problem with downloading BIG tag result.

[GoogleCodeExporter]

What steps will reproduce the problem?
1.search any tag producing lof of result (4000+), like 'naruto'
2.Click 'get all'
3.start downloading

What is the expected output? What do you see instead?
Grabber start some 'pre download operation', and get stuck on counting when 
going over 1000 images. Generally downloading nothing.

What version of the product are you using? On what operating system?
3.2.3+W7

Please provide any additional information below if neccessary.
on version 3.2.1, it grabber would simply crahs a few second after having 
started the downloads.

Please also attach the log, settings and dump if possible (remember to
remove your pseudo/password when uploading the settings files).

Original issue reported on code.google.com by ser...@hotmail.com on 5 Dec 2012 at 7:27

[GoogleCodeExporter]

Mmmmh, strange. I just checked and managed to start downloading all naruto's 
images (this step took less than one second), and checked a 100.000 images 
download which more or less worked.
Have you tried quitting and retrying?

Anyway, could I have your settings file as well as an .igl file for this 
download, in order to reproduce this? Thanks ^^

Original comment by bio.nus@hotmail.fr on 5 Dec 2012 at 8:49

• Added labels: Priority-Critical
• Removed labels: Priority-Medium

[GoogleCodeExporter]

yes I tried many times,
and with many different tags
once it is around 4000 images
it never go to the end
and I let my computer search a whole day already.
I just tried panty_&_stocking_with_garterbelt tag on danbooru (2955 hits)

it get stuck at 1125 images count, but none have been downloaded...

t remind me a very old bug of grabber...
the one you could not reproduce on your PC ^^
grabber getting slower and slower with each download...

might be something of the same kind!?

but why does it count 1125 images, but download nothing?
is it the new way that grabber work? or a danbooru bug (he does not download 
images from danbooru, but does from gelbooru)

NB: I got no problemif I try to download many images, if it is from many 
different tag searches.

Original comment by ser...@hotmail.com on 5 Dec 2012 at 10:38

[GoogleCodeExporter]

> I just tried panty_&_stocking_with_garterbelt tag on danbooru (2955 hits)
Same here, everything went smoothly. Could you send me your settings file? 
Maybe that's a setting you activated and that I didn't that cause this, or some 
particular mix of settings.

> t remind me a very old bug of grabber...
> the one you could not reproduce on your PC ^^
> grabber getting slower and slower with each download...
Yep, this reminds me of this bug too, but I don't think it's related, unless 
your computer is really at cause (but I don't think so).

> it get stuck at 1125 images count, but none have been downloaded...
> but why does it count 1125 images, but download nothing?
> is it the new way that grabber work? or a danbooru bug (he does not download 
images from danbooru, but does from gelbooru)
Yes, if it's stuck in the pre-download operation, that's normal that nothing's 
downloaded. Its main reason to be is to fill in the list, actually (and do some 
quick background work).
By the way, this operation is supposed to take no more than a few seconds with 
a linear complexity, so if it takes more than one minute you can just give up 
(unless it just goes slowly, of course). ^^
This operation is just filling the list and doing some light background work, 
there's nothing that should cause a freeze here. It worked the same way in 
previous version, but instead of showing the current status, you just had to 
wait without knowing when it would finish. ^^

> NB: I got no problemif I try to download many images, if it is from many 
different tag searches.
What do you mean? It doesn't bug when you do empty search downloads?

Original comment by bio.nus@hotmail.fr on 5 Dec 2012 at 11:39

[GoogleCodeExporter]

> NB: I got no problem if I try to download many images, if it is from many 
different tag searches.
What do you mean? It doesn't bug when you do empty search downloads?

I mean, 
If I download 4000+ image from a single tag search, it hang,

but if I download 4000+ images ,
but they are separated in many different tag search, most of them being around 
200 images, no problem occur.

Original comment by ser...@hotmail.com on 6 Dec 2012 at 2:09

[GoogleCodeExporter]

here is a nice ini file
for this bug and the Danbooru bug
Jouyeux noël avant l'heure!

Original comment by ser...@hotmail.com on 6 Dec 2012 at 7:27

Attachments:

• settings.ini

[GoogleCodeExporter]

joyeux, pas jouyeux ...

Original comment by ser...@hotmail.com on 6 Dec 2012 at 7:28

[GoogleCodeExporter]

Ok, so this proves me wrong as it doesn't come from your settings.

> If I download 4000+ image from a single tag search, it hang,
> but if I download 4000+ images ,
> but they are separated in many different tag search, most of them being 
around 200 images, no problem occur.
Ooh, interesting info, I'll check that in more detail, as it's not supposed to 
change anything (so it might be the source of this problem). :)

> Joyeux noël avant l'heure !
Merci, à toi aussi :)

Original comment by bio.nus@hotmail.fr on 6 Dec 2012 at 9:01

[GoogleCodeExporter]

yes, I was able to download up to 8000+ images
made of more than 20 tag search on gelbooru and danbooru at the same time (same 
tag, sharing 90% of the same images)
not a single problem

but a single search of 4000+ image does hang (and in 3.2.1 it was crashing)
it seem to happen also with just only 2000+ images.

and also, since 3.2.3(3.2.2?), any download from danbooru give a 'file already 
exist' for images never downloaded.

qu'est ce qu'il va t'apporter le perno'?

Original comment by ser...@hotmail.com on 6 Dec 2012 at 10:32

[GoogleCodeExporter]

oh, I just figured something ...
about 'pre-download operation' I mentionned.
it might not be what you think.
in reality, it is grabber skipping all the images on danbooru.
since it was going so fast, downloading nothing, I thought it was 'counting or 
scanning the images one by one' or something.
so, it was actually already trying to download (but it got mixed with the 
'danbooru- file already exist' bug).

I see that there is something completelly new in the download operation, 
apparently, grabber cut(or something) all the pages, that are too big, into 
smaller pages, just before downloading.
'pre download operation' had nothing to do with that.

Original comment by ser...@hotmail.com on 6 Dec 2012 at 10:40

[GoogleCodeExporter]

So, the program bugs in the middle of the download then, not the pre-download 
step.
If I understood correctly, it downloads the pages (for example 3 pages of 1000 
images for panty_&_stocking_with_garterbelt), then prepare them very quickly, 
and then tries to download them but fails (downloads nothing) and blocks in the 
middle?
That's "good" news ^^

> and also, since 3.2.3(3.2.2?), any download from danbooru give a 'file 
already exist' for images never downloaded.
Yes, that's because I broke the %search_n% token you're using in this update 
(3.2.2 and 3.2.3). I joined a version that should fix this (before I release a 
complete update).

> I see that there is something completelly new in the download operation, 
> apparently, grabber cut(or something) all the pages, that are too big, into 
smaller pages, just before downloading.
What do you mean? I don't think I have changed anything much in the way 
downloads work in this update (I just added indicators so you don't just stand 
in front of a meaningless "please wait" message that can stay here pretty long 
and therefore could make you think the program froze).

Pour Noël, une Cort EVL-Z4 (entre autres trucs) : 
http://www.mydukkan.com/media/Cort.CortEVLZ41796.jpg
Et toi ? :)

Original comment by bio.nus@hotmail.fr on 6 Dec 2012 at 10:49

Attachments:

• Grabber.exe

[GoogleCodeExporter]

I explain everything:
The new step (that is not the 'pre download operation' that I was talking 
about):
something that look like this 
X/8

and then, there is the download as itself (I was calling that 'pre download 
operation' because I was missleaded by the danbooru bug):
X/7458

IF I download on danbooru :
it hang at 11xx/7458
and downloaded nothing (issue 164 'files already exist')

If I use gelbooru :
it hang at 11xx/7458
but did download all the 11xx images.

habituellement, pour noel, je ne fait rien de spécial, je suis pas tres fêtes 
et traditions... ^^
je m'offre suffisement de choses quand je veux tout le temps pour que les 
fêtes soient qqch de spécial ^^
mais cette année est spéciale, je me suis offert un voyageau japon, pour 3 
semaines!
chez ma p'tit' copine qui habite par là!

Original comment by ser...@hotmail.com on 6 Dec 2012 at 10:59

[GoogleCodeExporter]

Actually, there's three steps (with two new steps), that's why there's been a 
misunderstanding from the beginning I think.

- First step, "pages download": X/8
- Second step, "images preparation", very fast (less than one second): X/7458
- Third and last step, "download": X/7458 (stuck for you at 11xx/7458)

Ok, I understood the problem. I guess I'll have to start a huge download to 
check that then.

Y'en a qui ont de la chance de partir au Japon. Même si je suis pas vraiment 
à plaindre, partant pour 5 mois en Corée, de mars à juillet xD
Dans quelle partie du Japon ? :)

Original comment by bio.nus@hotmail.fr on 6 Dec 2012 at 11:12

[GoogleCodeExporter]

osaka et alentoures

Original comment by ser...@hotmail.com on 6 Dec 2012 at 11:12

[GoogleCodeExporter]

le grabber 3.2.3,5 fais marcher danbooru sans problèmes ^^

Original comment by ser...@hotmail.com on 6 Dec 2012 at 11:13

[GoogleCodeExporter]

Aaaah, I'm glad to hear it :)

Original comment by bio.nus@hotmail.fr on 6 Dec 2012 at 11:28

[GoogleCodeExporter]

aaaaaaah! XD XD XD

I try to download a simple tag (princess_peach, 700+ images on danbooru, 1000+ 
images on gelbooru)
on danbooru or gelbooru,
it crash all the time after a few images downloaded!! (about 300~400)

Original comment by ser...@hotmail.com on 7 Dec 2012 at 2:08

[GoogleCodeExporter]

Oh my oh my, I just noticed that too while trying a big download. Seems like 
the dev version I sent you is broken :S
At least I can reproduce this one so it should be easy to fix (even if the 
crash reporter doesn't produce any dump).

Original comment by bio.nus@hotmail.fr on 7 Dec 2012 at 10:04

[GoogleCodeExporter]

Seems to be fixed in this version (managed to download ~12,000+ images from a 
single batch download). Tell me if you still have issues.
Well, at least that allowed me to fix some old memory leaks. ^^'

Original comment by bio.nus@hotmail.fr on 7 Dec 2012 at 1:09

Attachments:

• Grabber.exe

[GoogleCodeExporter]

Doing some 4877 images download tag search ... no crash!!!! (fixed!)

Original comment by ser...@hotmail.com on 8 Dec 2012 at 12:03