search

croot / nekopaw

Automatically exported from code.google.com/p/nekopaw
1 stars 0 forks source link

Add g.e-hentai.org? #107

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
Idk if you would be able to add g.e-hentai.org since it's different platform 
than most in the downloader but I thought I'd suggest it since it would be a 
nice suggestion and all other e-hentai downloaders I found were no good lol 
Thanks! :D

Original issue reported on code.google.com by animeult...@gmail.com on 15 Apr 2013 at 7:37

GoogleCodeExporter commented 9 years ago 
This resource is really annoying and hard to add. Planned to add, but when it 
will be I dont know.

Original comment by catgirlfighter on 15 Apr 2013 at 10:37

GoogleCodeExporter commented 9 years ago 
That's cool. I know they have a pretty strict security. I got banned for an
hour cuz I was viewing images too quickly. I was sure if you could even do
it but like I said just thought I'd suggest it :)

Original comment by animeult...@gmail.com on 15 Apr 2013 at 8:33

GoogleCodeExporter commented 9 years ago 
>cuz I was viewing images too quickly
Lol, how quickly were you doing that in order to trigger an autoban? I never 
had such isuue.

Original comment by tit65rty...@inbox.ru on 27 Apr 2013 at 12:58

GoogleCodeExporter commented 9 years ago 
haha well I was kinda scanning through a gallery for download and viewing the 
sources and copying image urls on each page so it was probly about 3 pages in a 
couple min and then when I clicked for the 4th page it told me to slow down or 
I'll be banned so I waited a couple min and clicked it again and it said I was 
banned for an hour lol

Original comment by animeult...@gmail.com on 27 Apr 2013 at 6:12

GoogleCodeExporter commented 9 years ago 
Pretty common issue, the best delay between queries I got is around ~10 
seconds. Any faster can cause minibans, later or sooner (once one user even got 
24h ban, haha).

Original comment by catgirlfighter on 27 Apr 2013 at 8:25

GoogleCodeExporter commented 9 years ago 
both g.e-hentai and exhentai uses the same system. The problem is extracting 
the links of the images of the gallery you want to download. I use a link 
grabber script and DownThemAll to download them. However there are problems:

Linkgrabber script takes its time since it individually visits the gallery 
pages in the background and saves it. Considering the "You are opening too many 
pages too fast" error, there is a delay of 1.5 scs of opening a page. So you 
can get a 300 images gallery in about 450 scs considering there was no problem 
in the process.

However there are problems:

- the link of the image may change after a while i.e:

http://94.242.220.96:34113/h/ee26ba8c50ebc8e3d4cde2bccf3a5afa40bcd2d8-435329-128
0-1700-jpg/keystamp=1367629049-b06827762c/KOS_MOS03_3.jpg

This is one of the image links from exhentai, the keystamp here changes after a 
while or in a completely random way. I m not so sure of the process so i dont 
know the conditions of this change but it s certain that after some time it 
will change. You need to visit the image page again and grab the new image url 
to download it. 

- image not properly loading:

Some of the images may not load properly so you need to click on the link at 
the bottom of the image to open it again. I guess this is due to server issues. 
However, its a pain in the ass for automatic downloading.

- original images are hidden:

The images you browse may be the lower scaled low quality images if the gallery 
is high quality. You need to get the link of the original image at the bottom 
of the page by clicking the download button. This was not a problem for me 
since the script i use checks for original images.

- Takes too much damn time:

Like i said, the delay of 1.5 scs or you get an error is a huge pain in the 
ass. I dont have any idea how you are going to grab the image link file without 
bypassing this issue. And if u insist on still opening pages after receiving 
this error you get an ip ban of 1 hour.

- Bandwidth restrictions:

After browsing a bit you may get an image instead of the gallery image saying 
"bandwidth liit exceeded". Either you need to get a new ip adress or spend some 
credits to unlock this limit.

-------------------------------------------------------------------------------

The best way to get rid of some of these restrictions is to get the links of 
the image without browsing the gallery pages. I dont have a damn idea of how to 
do that though. 

After going through these issues, i concur that the best way to download is 
using the torrent. However, there are too many galleries without torrents ot 
torrents with no seeders and you are stuck with .

Also, i notices that your grabber downloads files based on tags. This is not a 
problem since the web-sites this program uses are tag-based sites. In contrast 
g.e and exhentai uses galleries and tags. Just wondering how you gonna 
implement that functionality considering you got past all the annoying 
restrictions.

All in all, these are my first hand experiences of using these sites. Hope they 
will be helpfull in case you decide to give it a try. Though personally i dont 
have much hope that these sites will be added. If you have some questions, you 
can ask them.

Original comment by laxarus on 4 May 2013 at 1:39

GoogleCodeExporter commented 9 years ago 
Personally I could really care less whether it's added or not lol I kinda
like getting the galleries from the actually site considering I like to
keep galleries separate from my image pools....besides with all the
restrictions they have on there sites I think it would be much more
productive focusing on and improving the downloader than trying to figure
out the ultimate puzzle of adding these sites lol They've obviously
considered every possibility when it comes to auto downloaders and blocked
them out

Original comment by animeult...@gmail.com on 5 May 2013 at 9:23

GoogleCodeExporter commented 9 years ago 
> #6 laxarus
Actually, I know about all this tricks, and tryed to avoid it in graber1 
(previous version). But because I know about it, I'm realy feel the pain when 
thinking about release it in the second one.

There is no way to get a pic url without parsing pages. Not only keystamp 
changes, even domain name can change. It's "volunteer" programm, used on 
g/exhentai. It means user's PCs used as sources of pictures. It's why URLs 
always changes and can lag and freeze.

Original comment by catgirlfighter on 6 May 2013 at 8:55

GoogleCodeExporter commented 9 years ago 
well if you know all about them, that is good. Just trying to help.

It would be good if u can make it work but like i said, it's a huge pain in the 
ass to get past all of those and you still have to wait to parse pages. 

Just an idea, make it possible for grabber to queue the galleries you want to 
download and. I mean like Jdownloader. You enter gallery urls to the grabber 
and go out have fun while it parses the pages and downloads. The key points are:

1- You dont need to be actively on pc to succesfully download a gallery, you 
just queue and go

2- Either make it possible to remotely restart the modem (again like in 
jdownloader) to get a new ip or automatically click the "reset limit" button in 
"my home" (Careful this spends gallery points) to get past the bandwidth limit.

In short, i m more than okay to download slowly as long as i dont need to 
manually check every single thing to succesfully download a gallery. What are 
your thoughts on this method? Just curious.

Original comment by laxarus on 6 May 2013 at 10:22

GoogleCodeExporter commented 9 years ago 
In nekopaw grabber (2.0) you can start automatically download pictures just 
after list finished. If you want to start download a list of galleries at the 
same time, same resources will be in the same queue of each other. In theory, 
you can just start list of queries with "start download after finish" flag, and 
leave it. But in practic, I don't know how good it will work because there is 
some hard to catch troubles with stability of huge tasks (I mean ALOT of 
galleries and pictures at the same time).

I can try to release "restart modem" feature, but it works only for dial-up 
modems, another connections don't change IP like this. Maybe proxy switch, but 
for it you need a list of stable proxies. "Reset limit" is most reliable option.

Original comment by catgirlfighter on 7 May 2013 at 8:39

GoogleCodeExporter commented 9 years ago 
Hmm, so queuing huge amount of galleries generates problems but enforcing a 
limit to how many you can queue can fix it at least to some degree. And i don't 
get why the "restart modem" feature only works in dial-up modems. It can also 
work in adsl modems or any modem that has a web interface. There are some 3rd 
party applications to restart your modem with 1 click. You can place a shortcut 
to your desktop to restart your modem easily. Also i believe you don't need a 
3rd party app to do that, you can use a command console to do it too. There are 
plenty of info on the net about it.

Original comment by laxarus on 7 May 2013 at 10:51

GoogleCodeExporter commented 9 years ago 
Is not exactly what I meant. Yes, for ADSL it will work too, if provider not 
changed something in their rules.

I mean how much people use modems nowadays? Even me changed 1.5y ago my 
satellite net to PPPoE "direct" cable connection. If you restart your 
connection or router you're will not get new IP, because it "remembered" by 
DNS-server and will be changed only after awhile.

And there is trouble with realisation. For release it I need a way to test it, 
but obviously I can't. :)

Original comment by catgirlfighter on 7 May 2013 at 11:48

GoogleCodeExporter commented 9 years ago 
Well, many ppl still use them. More like everyone around me uses them as an 
example. I have not met any single person that doesn't use a modem in real 
life. (Excluding ppl connecting from campus)

And you are right testing is a problem if you dont have any means to do it. But 
you can implement it as experimental and get user feedback with logs to sort it 
out. Though it will be a pain. Or you can make a command for the grabber to 
launch a 3rd party app(which is tested beforehand) to make it work. These are 
all i can think of right now.  

But let me tell you if you don't have a private proxy, using a public proxy to 
change the IP is a lot more work since you need to find a working one and keep 
updating. Also download speeds of public proxies are pathetic. So proxies are 
out of the question.

Original comment by laxarus on 7 May 2013 at 4:50

GoogleCodeExporter commented 9 years ago 
BTW, I personally like the option of launching a 3rd party app since you will 
not have to go through extensive coding to implement that and have a sure way 
to achieve the desired result.

Original comment by laxarus on 7 May 2013 at 5:00

GoogleCodeExporter commented 9 years ago 
>But you can implement it as experimental and get user feedback with logs to 
sort it out.
I think I can try this.

But anyway, I don't know, when muse will come to start releasing g/exhentai :)

Original comment by catgirlfighter on 8 May 2013 at 9:00

GoogleCodeExporter commented 9 years ago 
Hahaha, i know it's shitload of work to get it work so dont blame you if you 
cant get it done but Good luck anyway man.I am waiting for it. Hope it comes.

Original comment by laxarus on 8 May 2013 at 12:27

GoogleCodeExporter commented 9 years ago 
Added experimental access to g.e-hentai. Looking for issues like "403 
pictures", "too fast downloading" and stuff to debug.

Original comment by catgirlfighter on 22 Aug 2013 at 3:35

GoogleCodeExporter commented 9 years ago

Original comment by catgirlfighter on 10 Sep 2013 at 11:27