No Pokemon returned due to soft ban

[Matsu16]

No pokemon. 2 minutes ago it was still working. whats going on. no errors ..scans are just fine

[dikkedeur]

it seems there is a problem with having a lot of accounts. i did a test on a small area on my city, with 2 accounts it all looks okay, so i expand the radius to 2,7KM covering my whole city with 12 accounts. i see then a lot less pokemons then there should be, some area's are blank.

so it seems that it then goes to fast, a full scan take roughly 4 minutes.

[Matsu16]

It starts up perfectly but it stops showing after like 5 min. Got 240 accounts on 4km

How do i post javascript console?

[dikkedeur]

did another run on a total different location outside my city, no pokemns show up anymore.. it keeps on scanning but nothing pops up. no error in javascript console ( or i might be looking wrong )

is it possible that the database is corrupt and how to fix it? i dont mind doing a rescan it only takes 5 minutes max.. can i just delete the database file and will it be remade??

[dikkedeur]

no that is not the problem i run 1 instance just port 5000, and i did a new fresh db pull, and that doesnt work either. it seems that nia is blocking too much requests, but then again it would trow out an error i assume. and that is not the case.

17.94% of scan. 2016-08-09 14:17:54,012 [ scan] [ INFO] Completed 18.00% of scan. 2016-08-09 14:17:54,575 [ scan] [ INFO] Completed 18.05% of scan. 2016-08-09 14:17:59,757 [ scan] [ INFO] Completed 18.11% of scan. 2016-08-09 14:17:59,832 [ scan] [ INFO] Completed 18.17% of scan. 2016-08-09 14:18:00,285 [ scan] [ INFO] Completed 18.22% of scan. 2016-08-09 14:18:00,718 [ scan] [ INFO] Completed 18.28% of scan. 2016-08-09 14:18:01,174 [ scan] [ INFO] Completed 18.33% of scan. 2016-08-09 14:18:01,547 [ scan] [ INFO] Completed 18.39% of scan. 2016-08-09 14:18:02,146 [ scan] [ INFO] Completed 18.45% of scan. 2016-08-09 14:18:02,279 [ scan] [ INFO] Completed 18.50% of scan. 2016-08-09 14:18:03,011 [ scan] [ INFO] Completed 18.56% of scan. 2016-08-09 14:18:03,204 [ scan] [ INFO] Completed 18.61% of scan. 2016-08-09 14:18:04,175 [ scan] [ INFO] Completed 18.67% of scan. 2016-08-09 14:18:04,743 [ scan] [ INFO] Completed 18.73% of scan. 2016-08-09 14:18:09,924 [ scan] [ INFO] Completed 18.78% of scan. 2016-08-09 14:18:10,200 [ scan] [ INFO] Completed 18.84% of scan. 2016-08-09 14:18:10,499 [ scan] [ INFO] Completed 18.89% of scan. 2016-08-09 14:18:10,884 [ scan] [ INFO] Completed 18.95% of scan. 2016-08-09 14:18:11,339 [ scan] [ INFO] Completed 19.01% of scan.

it just keeps on going.. so either they are sending just null, that can be it? but how to determine if that is the case?

[dikkedeur]

well what do you know... suddenly something pops up, i dont know for sure if everything is returning right now.. we must wait and let it scan for some time to see if it works again and that it is not a lucky coincidence.

edit: it goes way slower then before... dont know why but i see now pokemons popping up on the map.. its still scanning my area, at a slower rate.

[nborrmann]

I have the assumption that this is in fact be a rate-limit.

In cbea6b37e4ea187b1f9da4671e5c1e440e8d81ba I pushed a change, which will print a warning if that's the case. Please report back whether or not you get that warning.

I am sometimes getting the warning as well, but it doesn't seem to follow any logic when it occurs. Maybe you guys find something?

[djv2]

i'm 99% sure you got an IP ban. i have the same issue @nborrmann do you think you can make a notice when you get an IP ban? niatnic returns 0 of everything when it happens

[dikkedeur]

got it indeed..

2016-08-09 14:39:37,030 [ models] [WARNING] Received valid response but without any data. Possibly rate-limited? 2016-08-09 14:39:37,030 [ scan] [ INFO] Completed 0.22% of scan. 2016-08-09 14:39:37,030 [ pgoapi] [ INFO] Attempting login: stinkpoep 2016-08-09 14:39:37,056 [ models] [WARNING] Received valid response but without any data. Possibly rate-limited? 2016-08-09 14:39:37,056 [ scan] [ INFO] Completed 0.28% of scan. 2016-08-09 14:39:37,056 [ pgoapi] [ INFO] Attempting login: eenhooppoep ^X2016-08-09 14:39:37,121 [ models] [WARNING] Received valid response but without any data. Possibly rate-limited? 2016-08-09 14:39:37,122 [ scan] [ INFO] Completed 0.34% of scan. 2016-08-09 14:39:37,122 [ pgoapi] [ INFO] Attempting login: flapperdeflap

[nborrmann]

For me, those errors just disappear after some time.

[dikkedeur]

and now i got less warnings.. so there is when you have too much accounts a possibility that it will sent nothing to you..

[Matsu16]

And if i make the area bigger. Would that solve a rate limit?

[dikkedeur]

not a change... its due more accounts im afraid.. the scanning goes also slower it seems, so it might be that nia is doing their magic and limiting requests, since its slower then before and if you go to fast ( read more accounts from same IP ) then they just limit you more ( assumption im not an expert on this case ^^)

[PhoenixICE]

this would mean that public wifi areas would have max accounts that can connect to them, interesting...

[nborrmann]

I am only getting these errors very occasionally:

[Matsu16]

I just think were pushing to many workers. But als not an expert. Wonder how many accounts will be stable for my area then..

[djv2]

@nborrmann i am getting it a lot with 100 accounts would it be possible, to in the code, add something that says if 1 account is returning 0 results after like 5 retries, skip that account

[Matsu16]

Djv2, got the same screen u got with 240 accounts on 4km.

[dikkedeur]

get them now also more

d valid response but without any data. Possibly rate-limited? 2016-08-09 15:01:24,645 [ scan] [ INFO] Completed 79.65% of scan. 2016-08-09 15:01:25,146 [ models] [WARNING] Received valid response but without any data. Possibly rate-limited? 2016-08-09 15:01:25,146 [ scan] [ INFO] Completed 79.71% of scan. 2016-08-09 15:01:25,551 [ scan] [ INFO] Completed 79.77% of scan. 2016-08-09 15:01:25,705 [ scan] [ INFO] Completed 79.82% of scan. 2016-08-09 15:01:26,344 [ scan] [ INFO] Completed 79.88% of scan. 2016-08-09 15:01:29,115 [ scan] [ INFO] Completed 79.93% of scan. 2016-08-09 15:01:29,702 [ scan] [ INFO] Completed 79.99% of scan. 2016-08-09 15:01:29,835 [ pgoapi] [ INFO] Server seems to be busy or offline! 2016-08-09 15:01:30,675 [ scan] [ INFO] Completed 80.04% of scan. 2016-08-09 15:01:32,655 [ scan] [ INFO] Completed 80.10% of scan. 2016-08-09 15:01:33,470 [ scan] [ INFO] Completed 80.16% of scan. 2016-08-09 15:01:34,556 [ scan] [ INFO] Completed 80.21% of scan. 2016-08-09 15:01:34,812 [ scan] [ INFO] Completed 80.27% of scan. 2016-08-09 15:01:35,319 [ scan] [ INFO] Completed 80.32% of scan. 2016-08-09 15:01:35,712 [ models] [WARNING] Received valid response but without any data. Possibly rate-limited? 2016-08-09 15:01:35,712 [ scan] [ INFO] Completed 80.38% of scan. 2016-08-09 15:01:35,889 [ models] [WARNING] Received valid response bu

and also

2016-08-09 15:01:29,835 [ pgoapi] [ INFO] Server seems to be busy or offline!

[nborrmann]

This is definitely not an IP-based ban. It seems to be per account.

So adding more accounts should actually resolve this, because the time between requests on the same account will be larger

[johved]

Is there a system in place in choosing which accounts scans what location?

Because Niantec uses soft-bans if an account has travelled extremly fast i.e. jumps from one location to the next instantly. I think soft-bans return zero pokemons and last for something between 0.5-3 hours. I thought it was account specific but perhaps it also is IP specific?. Anyway I guess people should not have locations very far apart if the server doesnt keep the same accounts at separate locations.

[djv2]

@nborrmann it definitely IS an IP-based ban. if i change my IP (via VPN), no more errors during scan

[Matsu16]

Are u saying we need te make more😂

[Matsu16]

Good question @johved

[nborrmann]

If this were IP-based all requests would fail, instead of just a few.

The errors are soft account bans, as @johved describes.

[Matsu16]

Then there should be a system like @johved is talking about

[Matsu16]

Almost every account got a softban now. Just waiting for a solution.

[dikkedeur]

for me its still working, but its better not to scan large area's with alot of accounts form same ip just to be sure

[dikkedeur]

the best would be ( just a theory dunno if its possible) make it that a few accounts are attached to a small area ( so not scan large area in 1 blow ) then multiple instances running in batches ( dont know how to explain it better )

in short, you have if you have 12 accounts 4 small area;s with 3 accounts attached to 1 area, they scan it... after 100% wait 20 seconds before doing rescan.. that could limit it... maybe.

[djv2]

@nborrmann all i have to say is if this is really softban issues, i never knew changing my IP address would fix and get rid of softbans lol

[Baklap4]

I've been getting the message as well: "Received valid response but without any data. Possibly rate-limited?"

What i used to do was create 1 account and place it somewhere where it'll "walk" for 7 steps (a radius of 1.5km). This worked like a charm. 1 Account for each location and showing them all together on one map :)

If there's a thing to place 1 - 4 characters at one spot to increase searching i'm happy and will be glad to make more and more accounts to get a big area covered. It'd be even faster if those 4 characters on 1 spot are divided over the spot so everyone has 1500/4 radius to walk.

[Matsu16]

50% of my accounts are giving the error atm. Its not working voor me at all.

[YonderGod]

When you're soft banned you can still see pokemon, they just run away after your first pokeball.

[stlmikey80]

wow.. new update i get alot of this..

016-08-09 10:51:42,164 [ scan] [ INFO] Completed 26.91% of scan. 2016-08-09 10:51:42,269 [ models] [WARNING] Received valid response but without any data. Possibly rate-limited? 2016-08-09 10:51:42,273 [ scan] [ INFO] Completed 26.98% of scan. 2016-08-09 10:51:42,282 [ models] [WARNING] Received valid response but without any data. Possibly rate-limited? 2016-08-09 10:51:42,282 [ models] [WARNING] Received valid response but without any data. Possibly rate-limited? 2016-08-09 10:51:42,286 [ scan] [ INFO] Completed 27.05% of scan. 2016-08-09 10:51:42,286 [ scan] [ INFO] Completed 27.11% of scan. 2016-08-09 10:51:42,288 [ scan] [ INFO] Completed 27.18% of scan. 2016-08-09 10:51:42,398 [ scan] [ INFO] Completed 27.25% of scan. 2016-08-09 10:51:42,765 [ scan] [ INFO] Completed 27.32% of scan. 2016-08-09 10:51:42,832 [ scan] [ INFO] Completed 27.38% of scan. 2016-08-09 10:51:44,368 [ scan] [ INFO] Completed 27.45% of scan. 2016-08-09 10:51:44,494 [ models] [WARNING] Received valid response but without any data. Possibly rate-limited? 2016-08-09 10:51:44,497 [ scan] [ INFO] Completed 27.52% of scan. 2016-08-09 10:51:44,822 [ models] [WARNING] Received valid response but without any data. Possibly rate-limited? 2016-08-09 10:51:44,825 [ scan] [ INFO] Completed 27.59% of scan. 2016-08-09 10:51:44,831 [ scan] [ INFO] Completed 27.65% of scan. 2016-08-09 10:51:46,391 [ scan] [ INFO] Completed 27.72% of scan. 2016-08-09 10:51:46,506 [ scan] [ INFO] Completed 27.79% of scan. 2016-08-09 10:51:46,690 [ models] [WARNING] Received valid response but without any data. Possibly rate-limited? 2016-08-09 10:51:46,694 [ scan] [ INFO] Completed 27.86% of scan. 2016-08-09 10:51:46,801 [ models] [WARNING] Received valid response but without any data. Possibly rate-limited? 2016-08-09 10:51:46,803 [ scan] [ INFO] Completed 27.92% of scan. 2016-08-09 10:51:48,384 [ scan] [ INFO] Completed 27.99% of scan. 2016-08-09 10:51:48,673 [ models] [WARNING] Received valid response but without any data. Possibly rate-limited? 2016-08-09 10:51:48,674 [ scan] [ INFO] Completed 28.06% of scan. 2016-08-09 10:51:48,770 [ models] [WARNING] Received valid response but without any data. Possibly rate-limited? 2016-08-09 10:51:48,773 [ scan] [ INFO] Completed 28.13% of scan. 2016-08-09 10:51:48,908 [ scan] [ INFO] Completed 28.19% of scan.

[fabiomfpinto]

I loaded 240 accounts with radius of 10000 (10Km) and no errors. More or less when finish the first scan and restarted for the second scan I didn't see any pokemon and I got that error message. I dunno if softban don't allow us to see pokemon. But in my opinion is something like hide objects (pokemons) per request per IP. It makes sense thought. If it is true we are done! Anyway nice tool!

[djv2]

@YonderGod is right, softban you can still see pokemon just can't catch. guess @nborrmann is wrong?

[nborrmann]

@djv2 @YonderGod Ok then, I guess we'll have to find another name for this.

[fabiomfpinto]

Guys did you tried to cover up same area but with more than one center?

[djv2]

@nborrmann @YonderGod the name is IP ban, i've been saying that for many hours now. if you immediately change your IP and rescan it works fine

[fabiomfpinto]

@djv2 maybe a temporary IP Ban or something like that. Because Now I reduced number of workers and the radius and it is working good. Just sometimes in some acc's get the error.

[nborrmann]

It's not IP based. It's per account. The error occurs when you make too many requests too quickly from one account. I can have one account banned and another working all from the same IP.

Then why does it work when you change the IP? Probably you're also pausing for a few seconds and that resets the ban. Or just logging in reset the ban.

[Matsu16]

How long does the ban take?

[fabiomfpinto]

@nborrmann so it's better try to increase the scan delay just to test if this acc's continue getting ban?

[dikkedeur]

indeed a longer delay would propably be the thing to do, prefer by config so that you can set it yourself due your own liking.

[nborrmann]

Maybe it is related to the new "driving/passenger" feature. Maybe the server detects "driving" and wants us to accept the popup before letting us continue.

[fabiomfpinto]

I tested another map tool (slower) and it detects well the pokemons. Maybe something related with this tool. Maybe this feature that you are talking about. Anyway this map is blazing fast!

[dikkedeur]

and to clarify things up.. its NOT an ip ban, since otherwise you would not get ANY data, and NO response from the server at all ( set by the time the ban would last ) What Nia does is just dropping your request with an empty packet back to you.. This means that the scan will go on but the data packet is null, meaning you dont get any information on your request. thats why you dont see pokemons appear on the map, you simply miss data back from your request.

Now how to avoid that..

1) add a longer delay on your request per account. 2) this will resolve in longer scan time.. 3) can be fixed by adding more accounts..

This is all theory and based on how in the past on wow emulation we fight against large attacks on packets sent towards your server.. you can either drop them or turn rubish or empty crap back to the requested client ^^

[dikkedeur]

@nborrmann that could very well be the case as well, that we are not even having a ban but just the problem with the you are driving and accepting that you are a passenger... not a bad theory.. didnt saw that comming :P

[djv2]

@dikkedeur your #3 add more accounts you say ;) muahahahah going for that 1mil account list lol jkjk

[dikkedeur]

@djv2 good luck with making them... i dont have the tools to do that 👍

[nborrmann]

@Lionir pogom now also automatically accepts ToS