search

PokemonGoers / PokeData

In this project you will scrape as much data as you can get about the *actual* sightings of Pokemons. As it turns out, players all around the world started reporting sightings of Pokemons and are logging them into a central repository (i.e. a database). We want to get this data so we can train our machine learning models. You will of course need to come up with other data sources not only for sightings but also for other relevant details that can be used later on as features for our machine learning algorithm (see Project B). Additional features could be air temperature during the given timestamp of sighting, location close to water, buildings or parks. Consult with Pokemon Go expert if you have such around you and come up with as many features as possible that describe a place, time and name of a sighted Pokemon. Another feature that you will implement is a twitter listener: You will use the twitter streaming API (https://dev.twitter.com/streaming/public) to listen on a specific topic (for example, the #foundPokemon hashtag). When a new tweet with that hashtag is written, an event will be fired in your application checking the details of the tweet, e.g. location, user, time stamp. Additionally, you will try to parse formatted text from the tweets to construct a new “seen” record that consequently will be added to the database. Some of the attributes of the record will be the Pokemon's name, location and the time stamp. Additional data sources (here is one: https://pkmngowiki.com/wiki/Pok%C3%A9mon) will also need to be integrated to give us more information about Pokemons e.g. what they are, what’s their relationship, what they can transform into, which attacks they can perform etc.
Apache License 2.0
9 stars 6 forks source link

Data from Pokesniper #129

Closed vivek-sethia closed 8 years ago

vivek-sethia commented 8 years ago

@gyachdav @sacdallago @jonas-he The appearedOn collected from pokesniper data source contains the time when the pokemon was last seen as opposed to other sources like pokeradar, pokecrew where it has the time when the pokemon actually spawned. As far as I know from some articles that the pokemon stays for 15 minutes at a place after spawning. So should I subtract 15 minutes from the time for the data from pokesniper or is there anything which I am missing?

Articles read : 1, 2, 3

sacdallago commented 8 years ago

To me that sounds reasonable. I am still wondering how it is possible that they have an absolute init or terminal time

jonas-he commented 8 years ago

@vivek-sethia i am also subtracting 15 minutes for my sources if the specify something like expire_At

jonas-he commented 8 years ago

@sacdallago dont quite get your question. At one moment in time there are lets say thousands of pikachus all over the globe. And if they wouldnt expire then after some days everyone would know that theres a pikachu which would make it kind of boring.

gyachdav commented 8 years ago

@jonas-he that sounds reasonable in principal although time of sighting does not mean spawn+15, it's more like spawn + x | 0 ≤ x ≤ 15. not quite sure how we account for that. ideas?

jonas-he commented 8 years ago

@gyachdav most of the sites where we get the data from run scanners or base their predictions on scanners. Scanners are using reverse engineered API to Niantic servers and thus they deliver the exact data from the game, meaning either the exact spawn or despawn time.

gyachdav commented 8 years ago

Except for pokesniper which gives you sighting time? Can you cross reference between pokesniper and other sources?

Btw, Using Niantic data may become a problem down the road.

jonas-he commented 8 years ago

@gyachdav No pokesnipers gives you the exact expire time. This is why we subtract 15 minutes from that to get the appearedOn time. I checked this by looking at one entry from pokesnipers, then i run a scanner at this location (either through fastpokemap.se or on my own pc, see https://github.com/PokemonGoMap/PokemonGo-Map) and check if the time from pokesnipers is equal to the expire time given by the "illegal" source.

Well you can never be sure how these sources aquire their data, but for instance fastpokemap.se claims that it runs scanners, see https://fastpokemap.se/faq.html. Speculations: https://www.reddit.com/r/pokemongodev/comments/4wgm1j/how_does_skiplagged_work/ https://www.reddit.com/r/pokemongodev/comments/51avwv/how_does_pokeradar_predict_the_spawns/

vivek-sethia commented 8 years ago

Done #132