search

PokemonGoers / PokeData

In this project you will scrape as much data as you can get about the *actual* sightings of Pokemons. As it turns out, players all around the world started reporting sightings of Pokemons and are logging them into a central repository (i.e. a database). We want to get this data so we can train our machine learning models. You will of course need to come up with other data sources not only for sightings but also for other relevant details that can be used later on as features for our machine learning algorithm (see Project B). Additional features could be air temperature during the given timestamp of sighting, location close to water, buildings or parks. Consult with Pokemon Go expert if you have such around you and come up with as many features as possible that describe a place, time and name of a sighted Pokemon. Another feature that you will implement is a twitter listener: You will use the twitter streaming API (https://dev.twitter.com/streaming/public) to listen on a specific topic (for example, the #foundPokemon hashtag). When a new tweet with that hashtag is written, an event will be fired in your application checking the details of the tweet, e.g. location, user, time stamp. Additionally, you will try to parse formatted text from the tweets to construct a new “seen” record that consequently will be added to the database. Some of the attributes of the record will be the Pokemon's name, location and the time stamp. Additional data sources (here is one: https://pkmngowiki.com/wiki/Pok%C3%A9mon) will also need to be integrated to give us more information about Pokemons e.g. what they are, what’s their relationship, what they can transform into, which attacks they can perform etc.
Apache License 2.0
9 stars 6 forks source link

API sightings limit #192

Closed bensLine closed 8 years ago

bensLine commented 8 years ago

Hey guys,

is there a limit for the sightings request? I always receive only 1000 entries for a request like: http://pokedata.c4e3f8c7.svc.dockerapp.io:65014/api/pokemon/sighting/ts/2016-09-14T08:00:00.000Z/range/1d But we actually need all entries of the day.

jonas-he commented 8 years ago

Yes there is. This is due to a request from the maps team that the map wont get overloaded with Pokémon. In the response there is a flag "limited" which tells you if it had been cut off or not. If yes you should lower the time range. Do this until it is not limited and then query the neighboring time interval with the same method

bensLine commented 8 years ago

hmm, this seems a bit tricky. finding the right timespan by several requests is time-consuming and the wrong approach for that, in my opinion. Can you not provide a parameter limited which is by default true and we can set it to false, so the map team won't be affected.

We would en up with something like this: http://pokedata.c4e3f8c7.svc.dockerapp.io:65014/api/pokemon/sighting/ts/2016-09-14T08:00:00.000Z/range/1d?limited=false

jonas-he commented 8 years ago

well this would allow anyone to make arbitrary big request to our server. My idea would be to communicate a secret api key and with that key you will be able to receive responses unlimited in size. Would implement if @sacdallago approves this.

sacdallago commented 8 years ago

@bensLine , @jonas-he is right, that would open up a pretty dangerous behavior. Of course, if someone wants to get all the data all they need to do is send multiple requests with different timestamps, but that requires more work and DDosing the server, which will block them.

If you can implement a GET parameter (or better, header) containing a token which will validate the requester, that is a good enough security measure for now :)

bensLine commented 8 years ago

oh I see, thanks for the update!

jonas-he commented 8 years ago

@bensLine @sacdallago will implement tomorrow

jonas-he commented 8 years ago

@bensLine @sacdallago it's done, see #193. Once this is merged the API at http://pokedata.c4e3f8c7.svc.dockerapp.io:65014/ will be "unlimited" provided you know the token which is publicly available in this github repo. But i guess @sacdallago will be shutting this down soon anyways. So @bensLine should switch over to the API at predictemall.online

sacdallago commented 8 years ago

Thanks @jonas-he . Have a small change request in your PR

jonas-he commented 8 years ago

@sacdallago done

sacdallago commented 8 years ago

@jonas-he 🙈

bensLine commented 8 years ago

@jonas-he thanks for the implementation! :) I just wanted to test it, but I get a 502 for https://predictemall.online/api/pokemon/sighting/ts/2016-11-01T08:00:00.000Z/range/1h ad the old API is down as you mentioned. Do you know if there is a running API currently?

jonas-he commented 8 years ago

Nope you have to ask @sacdallago about the status of the API

bensLine commented 8 years ago

ok, thanks!

sacdallago commented 8 years ago

Not really, the latest deployment broke the container:

[pokedatadevelop-1]2016-11-07T07:07:41.726450999Z npm info it worked if it ends with ok
[pokedatadevelop-1]2016-11-07T07:07:41.727811739Z npm info using npm@2.14.2
[pokedatadevelop-1]2016-11-07T07:07:41.728444895Z npm info using node@v4.0.0
[pokedatadevelop-1]2016-11-07T07:07:42.904858049Z npm info prestart pokemongo-api@0.0.1
[pokedatadevelop-1]2016-11-07T07:07:42.941744903Z npm info start pokemongo-api@0.0.1
[pokedatadevelop-1]2016-11-07T07:07:42.946630669Z 
[pokedatadevelop-1]2016-11-07T07:07:42.946667642Z > pokemongo-api@0.0.1 start /usr/src/app
[pokedatadevelop-1]2016-11-07T07:07:42.946674266Z > NODE_ENV=production node app.js
[pokedatadevelop-1]2016-11-07T07:07:42.946689918Z 
[pokedatadevelop-1]2016-11-07T07:07:50.020635738Z 2016-11-07T07:07:49.987Z started prediction script, init DC
[pokedatadevelop-1]2016-11-07T07:07:50.026570802Z fs.js:549
[pokedatadevelop-1]2016-11-07T07:07:50.026605857Z   return binding.open(pathModule._makeLong(path), stringToFlags(flags), mode);
[pokedatadevelop-1]2016-11-07T07:07:50.026618937Z                                                   ^
[pokedatadevelop-1]2016-11-07T07:07:50.026624811Z 
[pokedatadevelop-1]2016-11-07T07:07:50.026629680Z Error: ENOENT: no such file or directory, open '/usr/src/app/node_modules/predict-pokemonprediction_feature_config.json'
[pokedatadevelop-1]2016-11-07T07:07:50.026650768Z     at Error (native)
[pokedatadevelop-1]2016-11-07T07:07:50.026656315Z     at Object.fs.openSync (fs.js:549:18)
[pokedatadevelop-1]2016-11-07T07:07:50.026661239Z     at Object.fs.readFileSync (fs.js:397:15)
[pokedatadevelop-1]2016-11-07T07:07:50.026666295Z     at fileToJson (/usr/src/app/node_modules/predict-pokemon/dataSet_creator.js:364:23)
[pokedatadevelop-1]2016-11-07T07:07:50.026671683Z     at Object.DC.init (/usr/src/app/node_modules/predict-pokemon/dataSet_creator.js:30:22)
[pokedatadevelop-1]2016-11-07T07:07:50.026690748Z     at /usr/src/app/node_modules/predict-pokemon/prediction.js:41:8
[pokedatadevelop-1]2016-11-07T07:07:50.026697357Z     at Object.<anonymous> (/usr/src/app/node_modules/predict-pokemon/prediction.js:263:3)
[pokedatadevelop-1]2016-11-07T07:07:50.026702793Z     at Module._compile (module.js:434:26)
[pokedatadevelop-1]2016-11-07T07:07:50.026707226Z     at Object.Module._extensions..js (module.js:452:10)
[pokedatadevelop-1]2016-11-07T07:07:50.026736200Z     at Module.load (module.js:355:32)
[pokedatadevelop-1]2016-11-07T07:07:50.026743785Z     at Function.Module._load (module.js:310:12)
[pokedatadevelop-1]2016-11-07T07:07:50.026748186Z     at Module.require (module.js:365:17)
[pokedatadevelop-1]2016-11-07T07:07:50.026752309Z     at require (module.js:384:17)
[pokedatadevelop-1]2016-11-07T07:07:50.026756842Z     at Object.<anonymous> (/usr/src/app/app/api/sighting.js:4:20)
[pokedatadevelop-1]2016-11-07T07:07:50.026761735Z     at Module._compile (module.js:434:26)
[pokedatadevelop-1]2016-11-07T07:07:50.026766083Z     at Object.Module._extensions..js (module.js:452:10)
[pokedatadevelop-1]2016-11-07T07:07:51.074654282Z 
[pokedatadevelop-1]2016-11-07T07:07:51.098053066Z npm info pokemongo-api@0.0.1 Failed to exec start script
[pokedatadevelop-1]2016-11-07T07:07:51.098337075Z npm ERR! Linux 4.4.0-42-generic
[pokedatadevelop-1]2016-11-07T07:07:51.098826806Z npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "start"
[pokedatadevelop-1]2016-11-07T07:07:51.099217175Z npm ERR! node v4.0.0
[pokedatadevelop-1]2016-11-07T07:07:51.099637946Z npm ERR! npm  v2.14.2
[pokedatadevelop-1]2016-11-07T07:07:51.100102290Z npm ERR! code ELIFECYCLE
[pokedatadevelop-1]2016-11-07T07:07:51.100521810Z npm ERR! pokemongo-api@0.0.1 start: `NODE_ENV=production node app.js`
[pokedatadevelop-1]2016-11-07T07:07:51.100791150Z npm ERR! Exit status 1
[pokedatadevelop-1]2016-11-07T07:07:51.101071172Z npm ERR! 
[pokedatadevelop-1]2016-11-07T07:07:51.101421899Z npm ERR! Failed at the pokemongo-api@0.0.1 start script 'NODE_ENV=production node app.js'.
[pokedatadevelop-1]2016-11-07T07:07:51.101790140Z npm ERR! This is most likely a problem with the pokemongo-api package,
[pokedatadevelop-1]2016-11-07T07:07:51.102139520Z npm ERR! not with npm itself.
[pokedatadevelop-1]2016-11-07T07:07:51.103145004Z npm ERR! Tell the author that this fails on your system:
[pokedatadevelop-1]2016-11-07T07:07:51.104269882Z npm ERR!     NODE_ENV=production node app.js
[pokedatadevelop-1]2016-11-07T07:07:51.104963878Z npm ERR! You can get their info via:
[pokedatadevelop-1]2016-11-07T07:07:51.113246038Z npm ERR!     npm owner ls pokemongo-api
[pokedatadevelop-1]2016-11-07T07:07:51.113678360Z npm ERR! There is likely additional logging output above.
[pokedatadevelop-1]2016-11-07T07:07:51.124612824Z 
[pokedatadevelop-1]2016-11-07T07:07:51.130326765Z npm ERR! Please include the following file with any support request:
[pokedatadevelop-1]2016-11-07T07:07:51.180667374Z npm ERR!     /usr/src/app/npm-debug.log

still missing files errors. This is due to the PR being merged before the problem was fixed.

bensLine commented 8 years ago

are sure it's the latest version? since the error refers to this line DC.init(__dirname + '/prediction_feature_config.json', true); which is fixed in the latest package (yesterday evening).

sacdallago commented 8 years ago

I'll trigger the build on docker hub again. If the service restarts, problem solved.

swathi-ssunder commented 8 years ago

@sacdallago - The package version was updated and merged just now. So the next docker build should reflect the latest changes.