modrzew / pokeminer

Pokemon location scraper
MIT License
220 stars 62 forks source link

Proxies questions. #252

Open paul30001 opened 7 years ago

paul30001 commented 7 years ago

I know there have been other issues about this, but they didn't have the answers to my questions.

I'm tired of having my Dynamic IP blocked, and then having to wait 2 days for it to change.

Can someone please make a quick step-by-step on how to setup proxies? Please post a sample proxy config. And what is the ssh -D [port] all about?

I did everything the person in #246 did, but I keep getting EXCEPTION. Worker.log says could not parse the proxy ip. Heeelp please

gunawanputra commented 7 years ago

Use {'http': 'http://IP:PORT', 'https': 'http://IP:PORT'} for http/https proxy. I'm not sure about address in https key, maybe should use https://. So far when I try this I'm able to send request but unfortunately proxy I use for trial is being banned.

For shock5 proxy you need to run pip install requests[shocks], then use this format: PROXIES = {'http': 'socks5://IP:PORT', 'https': 'socks5://IP:PORT'}

Check this link about requests advance usage [http://docs.python-requests.org/en/master/user/advanced/]

runmorereadmore commented 7 years ago

I'm using this:

PROXIES = { 'http': 'socks5://212.47.239.185:9009', 'https': 'socks5://212.47.239.185:9009'}

which is some random proxy I found online that doesn't work.

I keep getting Login Failed, even though I know the accounts work fine without a proxy. Any thoughts on why I would get Login Failed?

gunawanputra commented 7 years ago

@runmorereadmore Since a lot of pokemongo botter also use shocks proxy, your proxy could be also in use to access the server by others. I'm not sure, but I think more access on the same proxy mean more faster ip ban. Check your log, did your PTC or google account login successfully? If yes then the proxy is banned.

gunawanputra commented 7 years ago

'LOGIN FAIL' in status bar or 'Login failed!' in log don't always mean your PTC or google login also failed . Pokeminer still use login function in pgoapi.py. When worker call this function, pgoapi will call set_authentication function and send request to server. If authentication failed, 'LOGIN FAIL'. If authentication success but don't receive any response from server, 'LOGIN FAIL'. 'Login failed!' mean pgoapi don't receive any response.

gunawanputra commented 7 years ago

If you use windows machine, there will be and exception, AttributeError: 'module' object has no attribute 'inet_pton'. This happen because in module Pysocks, inet_pton is only available on unix. One workaround to resolve this issue by is by run pip install win_inet_pton, and add import win_inet_pton in pgoapi.py. After I try this workaround, my workers could run on shocks5 proxy.

runmorereadmore commented 7 years ago

@gunawanputra I didn't think about the fact that botters could be using those proxies too. Are you using a private proxy or just scrounging around for some?

The PTC account seems to be working fine, when I remove the proxy miner works fine, for a little while till they all get banned.

I'm gonna try increasing the delay and cutting my workers from 18 to 12, maybe even 9. My area is really only like 2 square miles, but I try to use this as a live feed to catch stuff when I'm home. Freaking suburbs.

gunawanputra commented 7 years ago

Almost all bot application use the same pgoapi. When pgoapi add proxy access, all bot will have proxy access too. Botters will use free shocks proxy available online, of course after trial and error to find a good one. Some will share it on a forum, other will keep it for themselves.

paul30001 commented 7 years ago

@gunawanputra @runmorereadmore How do you find a good proxy? Once you start using it, does it keep working until ban, or can someone else use the IP and then you won't be able to? The ones I have tried all say LOGIN FAIL. How can I get a good one? Can I know for sure that it's a problem with the Proxy IP and not with my own Proxy/pgoapi/Python Installation? @gunawanputra Why do you keep saying "shocks?" Just a joke? lol

Also, is there an easy way to change Dynamic IP address when it gets banned? I tried doing it via cloning Router's MAC address, via releasing my IP, via unscrewing the COAX cable from my modem. Nothing works! It only changes every few days/weeks by itself. How can I speed it up for sure, guys? @modrzew

crhbetz commented 7 years ago

Oh boy :-)

How do you find a good proxy?

Look. A proxy is, basically, just another computer on the internet that will forward your requests to their target on your behalf. The target will think that the request came from the proxy, so it will hand the answer back to the proxy, which then, in turn, will hand it back to you. The target does not know this.

Now if your home IP is banned and you have a proxy that has an IP that is not banned, this will work around your ban, because again, the target will not know that the proxy is asking on your behalf.

If however this proxy works for a lot of people it will at some point get banned for too many requests, just like your home IP.

Since everyone wants everything for free, those free proxies will have to handle a lot of load, send a lot of requests to the servers, and will get banned too. So you'll need a proxy that isn't used by everyone. How do you get that? Simple - by paying for exclusive service, either for a proxy or for some kind of server to run your own private proxy server on. You still have to be careful not to get that one banned too, though, especially because servers like that usually have static IPs.

Can I know for sure that it's a problem with the Proxy IP and not with my own Proxy/pgoapi/Python Installation?

You need to have an IP that you know is not banned to find out(so wait for a new dynamic IP or get a own static one).

Also, is there an easy way to change Dynamic IP address when it gets banned?

That entirely depends on your provider. My own private IP on a cable provider changes like once every full moon regardless of what I do, my Mum's access has a forced 24-hour reconnect and the IP will change every time it reconnects (so I put a raspberry pi into her cupboard and ran a private proxy on it, for science).

gunawanputra commented 7 years ago

@paul30001 I'm sorry for that.. :D I was so sleepy when try to find an answer for you. A good explanation from @Innenseide. I just use random free proxy I found on the internet

runmorereadmore commented 7 years ago

@Innenseide

Is there a good solution to the problem of IP bans? Even if I paid for a private proxy it likely has the same probability of getting IP banned that my home computer does and then I'm just stuck in the same situation.

gunawanputra commented 7 years ago

Set a proxy--scan--get banned--stop scan process--repeat. Do that manually or create a proxy switcher function to set a new proxy from proxy dictionary list, then scan process don't need to stop

paul30001 commented 7 years ago

@Innenseide well, I knew what proxy does, but otherwise, excellent explanation! Thank you! I appreciate the time you put into it :) Sometimes there are ways to force a Dynamic IP change, that's why I was asking.

And about my installation: I know that everything is installed properly because pokeminer is working with my own IP. I meant that maybe I messed up with the socks installation or the syntax of Proxies in config.py. That could give errors, even though the proxies I'm using may be good. No way to know until I make it work haha.

@Innenseide also, what do you think the limit of workers is where Niantic start detecting it? Because before, as long as I ran under 70 workers, I was fine for days. But for the past few days even under 50 workers get my IP banned in half a day. Can they detect it better now? That probably means running any amount of workers can get detected...