Image limit detection for g.e/exhentai

[GoogleCodeExporter]

I wonder if it's possible to have NG2 detect how close the user is to the image 
limit and slow down or stop image downloading until it detects that it's safe 
to continue.

Currently if you go over the image limit you download an "image limit reached" 
placeholder image and NG2 changes the original image link to the new "image 
cap" link. This means if you reset the limit and go back to download again you 
will re-download the "image cap" image again.
The only way to fix this (that I know of) is to create a new list and populate 
it with the album's image links all over again.

As always, thanks for your work on the project.

Original issue reported on code.google.com by tehvesu...@gmail.com on 8 Jan 2014 at 7:11

[GoogleCodeExporter]

Yeah, I thinking about this. I think I'll just check for placeholder, if you 
getting placeholder then application automatically will stop downloading. I 
don't really know how to "detect" if it safe to continue. Maybe resume after ~ 
2min is enough for it. I also think maybe you can use ~15 second delay between 
pictures.

Original comment by catgirlfighter on 9 Jan 2014 at 8:48

[GoogleCodeExporter]

That sounds excellent.

Original comment by tehvesu...@gmail.com on 9 Jan 2014 at 7:40

[GoogleCodeExporter]

Буржуй дело говорит, я уже писал ранее что 
ошибка 504 идет при скачке больших галерей 
примерно после 100-150 картинки, было бы 
хорошо добавить в настройки пункт, 
временной интервал настраиваемый и после 
какой картинки его делать, ну или как вы уже 
писали ранее выявление ошибки и после 
пауза на минуты две (но я думаю второй 
вариант программно довольно сложно 
реализовать).

З.Ы.
Присоеденяюсь к буржую, Огромная 
благодарность вам за ваш труд и этот проект.

Original comment by server26...@gmail.com on 10 Jan 2014 at 7:19

[GoogleCodeExporter]

That feature, or something like it, would surely be helpful.

Also, I've noticed that if you use $fname$, it doesn't actually use the 
filename. Instead, it uses the last part of the Web Address of the post showing 
the image.

Could this be fixed as well? (Or does not using $fname$ make it work in this 
case?

Original comment by JarieSui...@gmail.com on 12 Jan 2014 at 9:17

[GoogleCodeExporter]

You should use %original_name% key to get "real file's name", $fname$ designed 
to be unique and represent only one possible for this name image on the same 
resource.

Original comment by catgirlfighter on 13 Jan 2014 at 8:21

[GoogleCodeExporter]

I added 509 check in e/exhentai script, and when it will be encountered, script 
will stop downloading (for all resources, not onlt e/exhentai). And I added an 
option in Threads Settings (Stop Signal Timer) where you can set an interval in 
seconds to restart downloading.

Original comment by catgirlfighter on 2 Apr 2014 at 2:04

• Changed state: Fixed

[GoogleCodeExporter]

Doesn't appear to be working. I enabled stop signal timer with a 121 second 
delay and it refuses to start downloading the first image in the entire list, 
even though I'm well under image limit cap.

Good to see you return, I thought you'd gone on holiday or taken a break :P

Original comment by tehvesu...@gmail.com on 3 Apr 2014 at 2:49

[GoogleCodeExporter]

In addition to my last post, I can't seem to download at all now, even after 
changing stop signal timer back to 0.

Original comment by tehvesu...@gmail.com on 3 Apr 2014 at 3:15

[GoogleCodeExporter]

Nah, it's was just a deadline on my job, just got no time for grabber.
Can you provide keywords and name format to reproduce it?

Original comment by catgirlfighter on 3 Apr 2014 at 8:14

• Changed state: Accepted

[GoogleCodeExporter]

I just got a problem with reproduction, I did download around 1500 pic in one 
session yesterday and didn't get 509 message even once, so I just made changes 
by intuition :)

Original comment by catgirlfighter on 3 Apr 2014 at 8:19

[GoogleCodeExporter]

Found it. Should work now.

Original comment by catgirlfighter on 3 Apr 2014 at 8:39

[GoogleCodeExporter]

Sorry to say, no it doesn't work, at least not yet (for me.)
Grabber is quite happy to keep trying to download pictures in regular or full 
size well over 200 images past my limit. The only way I can trigger the image 
limit detection is to try re-downloading the images after they fail.

Original comment by tehvesu...@gmail.com on 3 Apr 2014 at 3:50

[GoogleCodeExporter]

Maybe they use not only one placeholder for 509 error. When you will get an 
error next time, please leave urls for that pictures (ploceholder urls, not 
actual pictures).
As I told, I failed to reproduce a bandwidth exceed (it's just not happens to 
me for some reason), so I should ask people, who actually encounter it.

Original comment by catgirlfighter on 4 Apr 2014 at 8:36

[GoogleCodeExporter]

Hmm this is odd. Keep in mind I'm downloading full images by default.
When I go over the limit Grabber reports the exhentai.org/fullimg.php etc image 
url with ": can not get file extension" on the end.
I can go to the "url" link in my browser and I get the picture attached to this 
post, so there's no 509 image to download it's just text.
I can open the "post_url" link and the picture loads fine, no 509.
I can't load "alt_url", that DOES come up with 509.

So I guess image limit only seems to increment when Grabber requests either:
1. A normal sized image that isn't downloaded through the hentai@home network 
and needs to be pulled from the server (default URL or ALT URL.)
2. A full sized image.

You're right that there isn't only one placeholder for 509 errors, it's just 
that the 509 error for a full size image isn't being detected by Grabber since 
it's a text page (again, see attached picture.)

Original comment by tehvesu...@gmail.com on 4 Apr 2014 at 3:14

Attachments:

• exhentaioverlimit.png

[GoogleCodeExporter]

I see now, I'll fix it (but now only next week).

Original comment by catgirlfighter on 4 Apr 2014 at 3:21

[GoogleCodeExporter]

Is going nowhere :/ I need a source of page with an error, to know, where I 
should look for a message. Trying to get a limit again. 440 pics done for now.

Original comment by catgirlfighter on 10 Apr 2014 at 11:15

[GoogleCodeExporter]

How many pictures you download in one session, or it's just me? almost 3k 
pictures without limits.

I need you to upload a source of page with an error message (file with 
html-text in it) when you encounter it next time.

Original comment by catgirlfighter on 10 Apr 2014 at 2:31

Attachments:

• screen1.png

[GoogleCodeExporter]

Sent you an email.

Original comment by tehvesu...@gmail.com on 15 Apr 2014 at 4:09

[GoogleCodeExporter]

Wow, this is weird. Didn't see it in my mail client, checked it through gmail 
web-interface.

Original comment by catgirlfighter on 16 Apr 2014 at 8:19

[GoogleCodeExporter]

I see now, maybe is because you're downloading from exhentai, and I from 
g.e-hentai. Testing now.

Original comment by catgirlfighter on 16 Apr 2014 at 8:38

[GoogleCodeExporter]

No, not because of it, is because of "Full image" flag in settings. There is 
limit for downloading full images.

>Image Limits

>You are currently at 204 towards a limit of 180. This regenerates at a rate of 
1 per minute.
>Note that image views only count towards the limit if they cannot be loaded 
through H@H.
>Also note that the limit is not constant - it varies with server load and time 
of day.

So you actually can download only 1 pic per minute to be out of limit range. Or 
you can reset limit for 1GP per limit point you exceeded. And you should reset 
it every 180 pictures. Annoying feature.

Original comment by catgirlfighter on 16 Apr 2014 at 9:07

[GoogleCodeExporter]

Bingo :)
I don't mind paying the GP to reset the images, but if I leave Grabber running 
unattended and don't pay attention to the image limit it can reach around 
1000/200 because Grabber isn't detecting a 509 error for full sized images.
That means paying an extra 800 credits because for some reason their system 
keeps counting image requests to your total even if you don't get an image.

Original comment by tehvesu...@gmail.com on 16 Apr 2014 at 10:57

[GoogleCodeExporter]

Ah, why you didn't say it to me from the start?
Currenlty working on it. Will add it with some another changes.

Original comment by catgirlfighter on 16 Apr 2014 at 11:36

[GoogleCodeExporter]

Updated.

Later I maybe will add an option to reset limit automatically. Not sure, how to 
do it right for now.

Original comment by catgirlfighter on 16 Apr 2014 at 11:47

[GoogleCodeExporter]

It's working.
Thank you very much.

Original comment by tehvesu...@gmail.com on 18 Apr 2014 at 3:14

[GoogleCodeExporter]

Added an option to reset limits automatically as well.

Original comment by catgirlfighter on 25 Apr 2014 at 3:43

• Changed state: Fixed

[GoogleCodeExporter]

12:44:30 Stop signal sent with a reason: Can not reset Full Size Limit
12:44:30 exhentai.org: Can not reset Full Size Limit

Limit reset doesn't work and it stops grabber completely instead of using the 
stop signal.

Original comment by tehvesu...@gmail.com on 26 Apr 2014 at 6:07

[GoogleCodeExporter]

Hm. Didn't test it on exhetai, thought it should be the same as g.e-hentai.

>Can not reset Full Size Limit
means script didn't get confirmation about success. It means an error accured, 
and I'm not sure if progress should be able continue.

Original comment by catgirlfighter on 28 Apr 2014 at 8:44

• Changed state: Accepted

[GoogleCodeExporter]

Found it out. To reset exhentai's limits you should use g.ehentai's home page.

Original comment by catgirlfighter on 28 Apr 2014 at 9:10

[GoogleCodeExporter]

Updated.

Original comment by catgirlfighter on 28 Apr 2014 at 9:17

• Changed state: Fixed

[GoogleCodeExporter]

Aces.

Original comment by tehvesu...@gmail.com on 30 Apr 2014 at 5:53