No map objects returned

[yoeriboven]

When requesting the map objects no objects are returned. response.response returns statusCode: 3 requestId: 8145806132888207460 authTicket { start: <275b1cef dc027e2d 7f171746 5c275b05 3c87cf01 9825e57a 3a246f7a 57642ab4 2dd20b37 2137d9e9 77b241bb 69b73740 f272f91e 6ae6a8ff 4df33e07 b779904d> expireTimestampMs: 1469291523873 end: <72acd742 7e0e5654 2af248e1 5b8cb4f7> } returns[0]: <> and response.subresponses returns an empty array

[lsapan]

Right, the call needs more than just latitude and longitude. It also needs a list of s2 geometry cell IDs, which I mentioned needs to be ported to swift. I've started doing it, but I haven't gotten very far yet due to lack of time. If you check out the sphere.py file I mentioned in Contributing, you'll find the CellID class. We basically need that and it's dependencies.

[lsapan]

Feel free to check out S2Geometry.swift. It really shouldn't be too hard to finish porting it, there's just a bunch of mathematical functions that need to be ported. It's straight forward, just a little time consuming.

[yoeriboven]

In which file are these calculations? I don't know Python but porting the mathematical functions should be doable.

[lsapan]

Check out: https://github.com/qedus/sphere/blob/master/sphere.py

You'll find class CellId on line 809. We only need the from_point initializer on line 853, which would be great if it didn't have a bunch of other calls. 😄 But basically, just start with from_point and work your way through the other methods that it needs. It shouldn't be too many, and then we can use the id and send that.

[yoeriboven]

Tried it, but unfortunately goes over my head. 😞 Really looking forward to the fix for this though. Thanks for your work for the community.

[spinach]

Thank you Luke for posting this API! Any ETA on the map objects? I see you've assigned it to yourself and lately commited "Finishing touches on S2Geometry".

[lsapan]

The geometry library is fully ported, I just haven't had time to fully test loading map objects. It should be very doable now though.

[spinach]

So I quickly tried to test it. messagebuilder.cellId is an Array. It seems we're lacking one function to get all cells from the location coordinates? (I am very new to the S2Geometry thingy, so it might be something straightforward that I am not seeing..)

I saw how they do it in Obj-C here: https://github.com/ruffnecktsk/pokemap_live_ios

    NSArray *cells = [s2geometry cellsForCoordinate:self.coordinate];

    for (NSString *cell in cells) {
        unsigned long long result = 0;

        NSScanner *scanner = [NSScanner scannerWithString:cell];
        [scanner scanHexLongLong:&result];

        [mapMessageBuilder addCellId:result];
        [mapMessageBuilder addSinceTimestampMs:0];
    }

Can you point in right direction? Maybe I can help and send a PR.

[TheDarkCode]

I added the libc++ and s2geometry.a dependencies to a demo that just quickly lets you test GetMapObjects without the final implementation in Swift being used - I was out of coffee so didn't go much further. Since it doesn't have libcrypto.a as a dependency for x86_64 architecture support, only armv7, you need to run it on a device and not the simulator to avoid build errors. There are ~22 errors associated with the x86_64 platform support which can get resolved with the extra lib dependency being added manually.

https://github.com/thedarkcode/pgoapi-swift-demo

When I quickly tested it, it returned the mapCells array with s2CellId and currentTimestampMs. Essentially a direct lift off the pokemap_live_ios implementation, but will let you play around this morning.

[lsapan]

@guydaher @TheDarkCode sorry I didn't reply in time to keep you from using the C++ library! I went down that route originally and opted to just port the relevant portions to Swift because it'd be much cleaner. In any case, it's actually been done, I just hadn't updated the usage in the API. You can change the getMapObjects function in PGoApiRequest to this and it'll calculate the proper cell id:

func getMapObjects(latitude: Double, longitude: Double) {
    let messageBuilder = Pogoprotos.Networking.Requests.Messages.GetMapObjectsMessage.Builder()
    messageBuilder.latitude = latitude
    messageBuilder.longitude = longitude

    // These two lines are new, they use the S2Geometry.swift file I created
    let cell = S2CellId(p: S2LatLon(latDegrees: latitude, lonDegrees: longitude).toPoint())
    messageBuilder.cellId = [cell.id]

    // TODO: figure out what it wants
    // messageBuilder.sinceTimestampMs =

    methodList.append(ApiMethod(id: .GetMapObjects, message: try! messageBuilder.build(), parser: { data in
        return try! Pogoprotos.Networking.Responses.GetMapObjectsResponse.parseFromData(data)
    }))
}

I haven't had time to check out your port @TheDarkCode yet, but I'm guessing you know what it wants for sinceTimestampMs! Can you try this out and see if you can get it working? 😄

By the way, once your delegate gets the response you should be able to test with:

let r = response.subresponses[0] as! Pogoprotos.Networking.Responses.GetMapObjectsResponse
let cell = r.mapCells[0]
print(cell.nearbyPokemons)
print(cell.wildPokemons)
print(cell.catchablePokemons)

[TheDarkCode]

@Isapan You're only sending 1 cell in your request when you do that. You need to send 21 cells. Also, the code will just return [] when you call cell.nearbyPokemon, etc. Unless it gives you back a response with more than the s2CellId and currentTimestampMs fields - which is dependent on the lat long input and what the spawnings are.

In terms of sinceTimestampMs, since it is an array, you need to do append. In the other repo pokemap, they are using an add function to just add 0's. I was playing with both adding the current time as an Int and just passing 0. Both would return the mapCells array with the correct timestamp in response regardless of what time you put in the request.

[lsapan]

@TheDarkCode how did they determine which 21 cells to send? I'm assuming it's the lat/lon +- some distance for each right? Or is it something that s2geometry came up with? If so, I can add that as well.

Update: Actually I just remembered that CellId has next and prev functions. I didn't port those, but I remember thinking it'd be easy to. Do we just need to call those each 10 times to get the 21?

[TheDarkCode]

@lsapan It is from s2 geometry.

[lsapan]

Alright, I'll get those functions ported and see if it works with sinceTimestampMs = [0, 0..].

[TheDarkCode]

I started adding in other API methods like Fort requests, I'll post the updates later. Lots of boring code to do to complete coverage.

[lsapan]

Sure, thanks @TheDarkCode. Definitely submit a PR when you're ready. I have some other things to take care of atm, but I should have getMapObjects working for tonight.

[spinach]

Not to put pressure of any sort here, but any ETA here? Just planning my next few days. Lots of cool stuff can be built in Swift using this api once the getMapObjects works :D

[lsapan]

@guydaher @TheDarkCode Alright, I've ported the other methods and updated getMapObjects to send along 21 cells and 21 timestamps of 0. It still doesn't seem to be working though. I can't investigate too much yet, maybe one of you can play around with it?

[spinach]

I tried to investigate but no luck for me.. especially that I'm not very familiar with how the Api works. The crash that is happening can be solved from an answer in SO (see Issue#4)

[tyler1314371]

@lsapan @TheDarkCode I changed it so that I am now getting a response of

[ mapCells[0] { s2CellId: 9812014442764828672 currentTimestampMs: 1469795368212 } status: .Success ]

but then again, there are no pokemons. I tried setting the radius to even larger scans but still the same.

Plus it seems like the s2CellId returned is not even part of the cells that I inputed and I have no idea why

Update: OK got a bit of progress, now I am getting the right # of mapCells with # of input cells but the id is still not matching...and not even a single pokemon appeared in any cells which is not correct haha.

Update: Ok...I checked the python version of pogoapi...it seems like it is doing a range of latitude and longitudes and then create 21 CellIds for each. Then pass that into the api. However I did try that but it didn't go very far as I believe there is something wrong with the S2Cell as the two CellID for the same latitude and longitude didn't match.

Also, you are using .toPoint, the python version is using .parent(15). Any idea on why? I don't know too much about S2 mapping but just noticed that

[PokemonGoSucks]

The S2Geometry isn't done yet I'm guessing, the int64s are not correct, but even when you pass 20 correct cell ids, there isn't any fort data for whatever reason. :/

I'm going to try writing a new S2 port.

[tyler1314371]

@PokemonGoSucks

Nice, I tried hardcoding a few correct sets of CellIDs as well but got no data in the return cells too. Also I believe we do need to do a range of lat and lon and feeds into the S2 to create different sets of CellIDs. I am currently investigating to see if that's the case and will pass in multiple sets of CellIDs

btw the correct return status should be 1, if it's not 1 then it's not even close

@TheDarkCode @lsapan Update and Conclusion: -S2 needs to be fixed as the CellIDs generated from lat and lon is definitely not correct -need to add functions to generate a range of lat and long call covers, and generate multiple sets of CellIDs -Currently investigating why even with the correct CellIDs there is nothing as well.At least the return mapCells are correct as long as you pass in the correct set of CellIDs

[TheDarkCode]

@lsapan @tyler1314371

I'm going to spend some time tonight / tomorrow to figure out what is going on. I'm doing a related project here: https://www.github.com/TheDarkCode/PokeGodar where I'm experimenting with an ASP.NET 4.6.x Web API implementation.

I'm trying to make a conversational bot that you can run in the cloud and process interactions or reporting of activity from the game for each active player being run by the client. When run locally on iOS 10, it would use Siri Kit. In the demo, I was going to run some sample accounts indefinitely to just walk across the map and scan. However, the focus is for people to have either their own hosted WebAPI to run bots on a private web server / Azure Apps, or run the automation in the background of an iOS app. If we have a standard endpoint for reporting of map scans from the clients directly, then we can also populate a global search index from all the map object scans.

Microsoft and my startup @dryverless are going to be providing uncapped hosting for the public search index and demo.

---

Update: Passing cellIds that are correct, as well as lat / long values in UInt64 format - that work when passed with other APIs, did not return any map objects either. I'll dig more later.

[tyler1314371]

@TheDarkCode

nice, let me know how it goes. I am pretty tired today so I will continue tomorrow. I want to be able to fix the lat, lon -> CellID function first so then it matches the python version.

[lsapan]

I'd be interested to know why the CellIDs are not the same, I've verified there are with sphere.py. Perhaps that is implemented wrong as well? Hopefully it's a simple change either way.

[PokemonGoSucks]

You did a good job porting the python, but it might be the wrong level or something? (I'm not familiar with s2 geometry.)

[PokemonGoSucks]

I tried a few things... 1) I included the auth-token in the request envelope. 2) I included the location in request envelope. 3) I overwrote the bytes for 106/2/1 with the bytes from the python code.

Overwriting the bytes for 106/2/1 got me map objects... Not sure why the request message is different from the python one.

[lsapan]

@PokemonGoSucks Great discovery, thank you for sharing that. It's worth noting that (as of the last time I checked) the python map is using an older or at least custom protos implementation. It'd be a low hanging fruit to simply see if the relevant message declarations in POGOProtos and the python map agree with each other.

[TheDarkCode]

@PokemonGoSucks Awesome. I figured it was the ByteString difference that was the only thing else that could be changed.

I was noticing earlier that the large trackers - i.e. pokevision - got taken down by the recent update, but testing the API has shown that you can still get the local forts, pokemon, etc. The alternative providers are all stumbling. Might need to put in some extra time today to get the public map up and running.

[beeb]

Thanks for the great work! Pokevision was shut down by their creators, it was working perfectly until yesterday so I'm guessing that any stumbling would be due to PTC servers instabilities. Also, it seems there is now a 5s throttling limit for map requests coming from a single account, with no map objects returned if the request is sent during those 5 seconds after a successful request.

[spinach]

@PokemonGoSucks Awesome! Any PR coming in the way for the fix?

[PokemonGoSucks]

@guydaher Yes, I'll PR once I figure out s2. But here's a rough idea on how you could get map objects:

First I edited globals.swift to include a location parameter & to store the auth token.

struct Location {
    static var lat:Double = 0
    static var long:Double = 0
    static var alt:Double = 0
}

struct Api {
    static var endpoint = Endpoint.Rpc
    static let id: UInt64 = 8145806132888207460
    static let SettingsHash = "05daf51635c82611d1aac95c0b051d3ec088a930"
    static var receivedToken = false
    static var authToken: Pogoprotos.Networking.Envelopes.AuthTicket = Pogoprotos.Networking.Envelopes.AuthTicket()
}

Then in rpcapi.swift, I replaced buildmainrequest. I changed unknown12 and unknown 2 to match the app's requests, but I don't really know if it makes any difference.

private func buildMainRequest() -> Pogoprotos.Networking.Envelopes.RequestEnvelope {
        print("Generating main request...")
        let requestBuilder = Pogoprotos.Networking.Envelopes.RequestEnvelope.Builder()
        requestBuilder.statusCode = 2
        requestBuilder.requestId = Api.id
        requestBuilder.unknown12 = 1431

        requestBuilder.latitude = Location.lat
        requestBuilder.longitude = Location.long
        requestBuilder.altitude = Location.alt

        if (!Api.receivedToken) {
            let authInfoBuilder = requestBuilder.getAuthInfoBuilder()
            authInfoBuilder.provider = "ptc"
            let authInfoTokenBuilder = authInfoBuilder.getTokenBuilder()
            authInfoTokenBuilder.contents = Auth.sharedInstance.accessToken!
            authInfoTokenBuilder.unknown2 = 10800
        } else {
            requestBuilder.authTicket = Api.authToken
        }

        print("Generating subrequests...")
        for subrequest in subrequests {
            print("Processing \(subrequest)...")
            let subrequestBuilder = Pogoprotos.Networking.Requests.Request.Builder()
            subrequestBuilder.requestType = subrequest.id
            subrequestBuilder.requestMessage = subrequest.message.data()
            requestBuilder.requests += [try! subrequestBuilder.build()]
        }

        print("Building request...")
        return try! requestBuilder.build()
    }

Then I set the location and cued getPlayer() let request = PGoApiRequest() Location.alt = 0 Location.lat = 43 Location.long = -79 request.getPlayer() request.makeRequest(.getPlayer, delegate: self)

In s2.swift I increased the step size, trying 15/30 etc:

func prev() -> S2CellId{
        return S2CellId(id: id - (lsb() << 30))
    }

    func next() -> S2CellId {
        return S2CellId(id: id + (lsb() << 30))
    }

Then for the request response:

Api.endpoint = "https://\((response.response as! Pogoprotos.Networking.Envelopes.ResponseEnvelope).apiUrl)/rpc"
            Api.authToken = (response.response as! Pogoprotos.Networking.Envelopes.ResponseEnvelope).authTicket
            Api.receivedToken = true

Then attempt to get map objects..

request.getMapObjects(43, longitude: -79)
            dispatch_after(5000, dispatch_get_main_queue()) {
                request.makeRequest(.getMapObjects, delegate: self)
            }

[victorlandeira]

@PokemonGoSucks After this update we need native crowdsourced maps (servers can't request) and you guys are the closest one on swift. give us hope. can i help in any way?

[spinach]

@PokemonGoSucks Great job, it actually works 👍

[TheDarkCode]

@PokemonGoSucks Since most people are lazy, I updated the demo repo here so all they have to do is plug in a lat, long, and login in Globals.swift: https://github.com/TheDarkCode/pgoapi-swift-demo

Update:

I've started playing with doing automated-data mining by grabbing map objects at initial location, going to closest fort, grabbing map objects again, ad infinitum with a parsing of the pokemon and so forth. Doing human walking and finishing the other API methods will allow us to put out an app to make everyone an automatic contributor to a global search index - PokeGodar. Who needs a centralized scanning service on AWS like Pokevision.

I had considered running web jobs with hooks for each "scanner" instance running on the cloud - would let you run different bot instances as a sort of Pokemon Go SaaS platform. However, that concept didn't last long once l saw how Niantic had banned the existing cloud services and the basis of the decision. Some people are considering switching to Google's App service. Others are considering somewhat unknown VPS providers, VPN services, etc. Those are all questionable long-term options - at some point you're likely modifying something. The most versatile choice is to request data from IPs as if it were a normal user - on a cellular, public, university, or home network IP address. Which also means usual delay intervals between requests, and utilizing background threads.

Thus, the client running the app is the direct interface to the Pokemon Go servers - an unblockable entity. No way they block AT&T, Sprint, or Verizon's IP blocks for New York City or San Francisco. The client mobile users would then pass the data - validated first by the app, to a Web API that further validates before archiving the report on a DocumentDB server. Any documents added to DocumentDB would be imported into the index on Azure Search in real-time. Every so often batch operations would clear any stale reports (despawned pokemon) to keep the index lean.

[lsapan]

This shouldn't be an issue anymore thanks to some amazing contributions! Now that it's fixed and I've refactored the project structure a bit, the next step would be getting an example app fleshed out to help people get going faster. Considering you already have started one @TheDarkCode, would you mind tying it into the project with a PR?

[BigChase]

Please share sets of interesting sample cell_id's which contain pokestops, forts, and known spawn points for testing purposes? Thanks.