search

PokemonGoers / PokeData

In this project you will scrape as much data as you can get about the *actual* sightings of Pokemons. As it turns out, players all around the world started reporting sightings of Pokemons and are logging them into a central repository (i.e. a database). We want to get this data so we can train our machine learning models. You will of course need to come up with other data sources not only for sightings but also for other relevant details that can be used later on as features for our machine learning algorithm (see Project B). Additional features could be air temperature during the given timestamp of sighting, location close to water, buildings or parks. Consult with Pokemon Go expert if you have such around you and come up with as many features as possible that describe a place, time and name of a sighted Pokemon. Another feature that you will implement is a twitter listener: You will use the twitter streaming API (https://dev.twitter.com/streaming/public) to listen on a specific topic (for example, the #foundPokemon hashtag). When a new tweet with that hashtag is written, an event will be fired in your application checking the details of the tweet, e.g. location, user, time stamp. Additionally, you will try to parse formatted text from the tweets to construct a new “seen” record that consequently will be added to the database. Some of the attributes of the record will be the Pokemon's name, location and the time stamp. Additional data sources (here is one: https://pkmngowiki.com/wiki/Pok%C3%A9mon) will also need to be integrated to give us more information about Pokemons e.g. what they are, what’s their relationship, what they can transform into, which attacks they can perform etc.
Apache License 2.0
9 stars 6 forks source link

endpoint for predictions #183

Open gyachdav opened 7 years ago

gyachdav commented 7 years ago

Hello there pokedata, it has been a while 😆

I didn't see any spec in your API that will allow the map team to retrieve predictions. Are predictions included in the sighting data? is there a parameter that distinguish predictions from sightings?

Same questions apply to the model that holds prediction data.

Please respond ASAP.

bensLine commented 7 years ago

Oh, good :) The file was to big for git, if there is no limitation for npm packages we should be able to do that.

sacdallago commented 7 years ago

@bensLine for the reference: you can use unzip to unzip a zip file in your package before the packe is being installed, by defining this directive in the preinstall script of the package.json. This is fancy AF. Or you can do the unzipping in prepublish, but that way you would have the unzipped files in your repo, although you could .gitignore them. There are PLENTY way to handle this :) but it should be done on your side!

bensLine commented 7 years ago

@sacdallago thanks for the info, I never used JS and npm before the seminar, so every pointer is very welcome :) someone of my team @PokemonGoers/predictpokemon-2 will look into it.

sacdallago commented 7 years ago

Since https://github.com/PokemonGoers/PredictPokemon-2/pull/74#event-856132523, @swathi-ssunder version bump, docker hub trigger: we see if it works? @bensLine

swathi-ssunder commented 7 years ago

Updated to latest version #196. Hopefully everything works this time :pray:

jonas-he commented 7 years ago

@sacdallago its merged and docker build finished successfully. However DNS is not working right now. Maybe because there is still an error (->container shuts down) like we had in the past?

sacdallago commented 7 years ago

Merci! @jonas-he will have a look at the docker cloud log in the mornin'

sacdallago commented 7 years ago

Not over yet! 🗡 

[pokedatadevelop-1]2016-11-13T01:16:24.075715817Z npm info it worked if it ends with ok
[pokedatadevelop-1]2016-11-13T01:16:24.077662800Z npm info using npm@2.14.2
[pokedatadevelop-1]2016-11-13T01:16:24.080686356Z npm info using node@v4.0.0
[pokedatadevelop-1]2016-11-13T01:16:25.659310042Z npm info prestart pokemongo-api@0.0.1
[pokedatadevelop-1]2016-11-13T01:16:25.708214646Z npm info start pokemongo-api@0.0.1
[pokedatadevelop-1]2016-11-13T01:16:25.716176367Z 
[pokedatadevelop-1]2016-11-13T01:16:25.716226527Z > pokemongo-api@0.0.1 start /usr/src/app
[pokedatadevelop-1]2016-11-13T01:16:25.716235727Z > NODE_ENV=production node app.js
[pokedatadevelop-1]2016-11-13T01:16:25.716241644Z 
[pokedatadevelop-1]2016-11-13T01:16:35.636328011Z 2016-11-13T01:16:35.633Z started prediction script, init DC
[pokedatadevelop-1]2016-11-13T01:16:51.489398555Z fs.js:549
[pokedatadevelop-1]2016-11-13T01:16:51.489440515Z   return binding.open(pathModule._makeLong(path), stringToFlags(flags), mode);
[pokedatadevelop-1]2016-11-13T01:16:51.489452597Z                                                   ^
[pokedatadevelop-1]2016-11-13T01:16:51.489458631Z 
[pokedatadevelop-1]2016-11-13T01:16:51.489463581Z Error: ENOENT: no such file or directory, open '/usr/src/app/node_modules/predict-pokemon/feature_sources/../json/pokestop_groups.json'
[pokedatadevelop-1]2016-11-13T01:16:51.489468642Z     at Error (native)
[pokedatadevelop-1]2016-11-13T01:16:51.489473087Z     at Object.fs.openSync (fs.js:549:18)
[pokedatadevelop-1]2016-11-13T01:16:51.489477925Z     at Object.fs.readFileSync (fs.js:397:15)
[pokedatadevelop-1]2016-11-13T01:16:51.489482905Z     at fileToJson (/usr/src/app/node_modules/predict-pokemon/feature_sources/place_features.js:139:23)
[pokedatadevelop-1]2016-11-13T01:16:51.489488102Z     at /usr/src/app/node_modules/predict-pokemon/feature_sources/place_features.js:7:26
[pokedatadevelop-1]2016-11-13T01:16:51.489541341Z     at Object.<anonymous> (/usr/src/app/node_modules/predict-pokemon/feature_sources/place_features.js:142:3)
[pokedatadevelop-1]2016-11-13T01:16:51.489554023Z     at Module._compile (module.js:434:26)
[pokedatadevelop-1]2016-11-13T01:16:51.489559155Z     at Object.Module._extensions..js (module.js:452:10)
[pokedatadevelop-1]2016-11-13T01:16:51.489563271Z     at Module.load (module.js:355:32)
[pokedatadevelop-1]2016-11-13T01:16:51.489567232Z     at Function.Module._load (module.js:310:12)
[pokedatadevelop-1]2016-11-13T01:16:51.489571127Z     at Module.require (module.js:365:17)
[pokedatadevelop-1]2016-11-13T01:16:51.489575012Z     at require (module.js:384:17)
[pokedatadevelop-1]2016-11-13T01:16:51.489579314Z     at /usr/src/app/node_modules/predict-pokemon/dataSet_creator.js:52:30
[pokedatadevelop-1]2016-11-13T01:16:51.489583692Z     at Array.forEach (native)
[pokedatadevelop-1]2016-11-13T01:16:51.489588267Z     at Object.DC.init (/usr/src/app/node_modules/predict-pokemon/dataSet_creator.js:38:32)
[pokedatadevelop-1]2016-11-13T01:16:51.489593029Z     at /usr/src/app/node_modules/predict-pokemon/prediction.js:48:8
[pokedatadevelop-1]2016-11-13T01:16:51.641364127Z 
[pokedatadevelop-1]2016-11-13T01:16:51.650631189Z npm info pokemongo-api@0.0.1 Failed to exec start script
[pokedatadevelop-1]2016-11-13T01:16:51.651150535Z npm ERR! Linux 4.4.0-42-generic
[pokedatadevelop-1]2016-11-13T01:16:51.654334520Z npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "start"
[pokedatadevelop-1]2016-11-13T01:16:51.655635753Z npm ERR! node v4.0.0
[pokedatadevelop-1]2016-11-13T01:16:51.657848977Z npm ERR! npm  v2.14.2
[pokedatadevelop-1]2016-11-13T01:16:51.659788031Z npm ERR! code ELIFECYCLE
[pokedatadevelop-1]2016-11-13T01:16:51.661028434Z npm ERR! pokemongo-api@0.0.1 start: `NODE_ENV=production node app.js`
[pokedatadevelop-1]2016-11-13T01:16:51.662301128Z npm ERR! Exit status 1
[pokedatadevelop-1]2016-11-13T01:16:51.663278508Z npm ERR! 
[pokedatadevelop-1]2016-11-13T01:16:51.664938311Z npm ERR! Failed at the pokemongo-api@0.0.1 start script 'NODE_ENV=production node app.js'.
[pokedatadevelop-1]2016-11-13T01:16:51.666558686Z npm ERR! This is most likely a problem with the pokemongo-api package,
[pokedatadevelop-1]2016-11-13T01:16:51.667410798Z npm ERR! not with npm itself.
[pokedatadevelop-1]2016-11-13T01:16:51.669910098Z npm ERR! Tell the author that this fails on your system:
[pokedatadevelop-1]2016-11-13T01:16:51.671229682Z npm ERR!     NODE_ENV=production node app.js
[pokedatadevelop-1]2016-11-13T01:16:51.674070397Z npm ERR! You can get their info via:
[pokedatadevelop-1]2016-11-13T01:16:51.675422797Z npm ERR!     npm owner ls pokemongo-api
[pokedatadevelop-1]2016-11-13T01:16:51.677335732Z npm ERR! There is likely additional logging output above.
[pokedatadevelop-1]2016-11-13T01:16:51.690191312Z 
[pokedatadevelop-1]2016-11-13T01:16:51.694651110Z npm ERR! Please include the following file with any support request:
[pokedatadevelop-1]2016-11-13T01:16:51.704430089Z npm ERR!     /usr/src/app/npm-debug.log
sacdallago commented 7 years ago

@bensLine @Aurel-Roci

Aurel-Roci commented 7 years ago

@sacdallago did you call unzipFiles() first? Like here https://www.npmjs.com/package/predict-pokemon. If it still works after that, I can do something with callbacks, but I would like your feedback on it.

swathi-ssunder commented 7 years ago

@sacdallago , @Aurel-Roci - Even after unzipFiles(), the status is same as in https://github.com/PokemonGoers/PokeData/issues/183#issuecomment-260123638

bensLine commented 7 years ago

@Aurel-Roci I added a post install script to unzip the files. can you please release the package again to test it. https://github.com/PokemonGoers/PredictPokemon-2/pull/75

Aurel-Roci commented 7 years ago

@swathi-ssunder the new version is up

jonas-he commented 7 years ago

see #198

swathi-ssunder commented 7 years ago

@Aurel-Roci , @bensLine 

Error: Cannot find module 'fstream'
    at Function.Module._resolveFilename (module.js:336:15)
    at Function.Module._load (module.js:286:25)
    at Module.require (module.js:365:17)
    at require (module.js:384:17)
    at Object.<anonymous> (/usr/src/app/node_modules/predict-pokemon/scripts/postinstall.js:3:15)
    at Module._compile (module.js:434:26)
    at Object.Module._extensions..js (module.js:452:10)
    at Module.load (module.js:355:32)
    at Function.Module._load (module.js:310:12)
    at Function.Module.runMain (module.js:475:10)
npm info predict-pokemon@0.2.3 Failed to exec postinstall script
npm ERR! Linux 3.19.0-66-generic
npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "install"
npm ERR! node v4.0.0
npm ERR! npm  v2.14.2
npm ERR! code ELIFECYCLE

npm ERR! predict-pokemon@0.2.3 postinstall: `node scripts/postinstall.js`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the predict-pokemon@0.2.3 postinstall script 'node scripts/postinstall.js'.
npm ERR! This is most likely a problem with the predict-pokemon package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node scripts/postinstall.js
npm ERR! You can get their info via:
npm ERR!     npm owner ls predict-pokemon
npm ERR! There is likely additional logging output above.
npm info preuninstall predict-pokemon@0.2.3
npm info uninstall predict-pokemon@0.2.3
npm info postuninstall predict-pokemon@0.2.3
npm ERR! Please include the following file with any support request:
npm ERR!     /usr/src/app/npm-debug.log
The command '/bin/sh -c npm install' returned a non-zero code: 1
The command "docker build -t pokemongoers/pokedata ." failed and exited with 1 during .
Aurel-Roci commented 7 years ago

@swathi-ssunder package dependency was missing, added now. Sorry for all this

swathi-ssunder commented 7 years ago

@sacdallago - The docker build was successful this time, the DNS still does not work though!

jonas-he commented 7 years ago

so i looked into this and it turns out that after executing npm start this is the output:

npm info it worked if it ends with ok
npm info using npm@2.14.2
npm info using node@v4.0.0
npm info prestart pokemongo-api@0.0.1
npm info start pokemongo-api@0.0.1

> pokemongo-api@0.0.1 start /usr/src/app
> NODE_ENV=production node app.js

2016-11-16T16:39:27.282Z started prediction script, init DC
4837ms to construct shortcut table
Average timezones per 1° lat x 1° lng: 4.414128352490422
2016-11-16T16:39:35.486Z created training arff file
2016-11-16T16:39:35.487Z requesting http://predictemall.online:65014/api/pokemon/sighting/ts/2016-09-14T08:00:00.000Z/range/1d?token=I0TPIIpCLH8lR8iDrCMV
npm info poststart pokemongo-api@0.0.1
npm info ok

after this the process exits. Dont know exactly whats causing this. might be that it exits because the request to the predictemall site fails (i get a bad gateway when trying it in browser).

jonas-he commented 7 years ago

sorry it turns out that my docker container exits because it couldnt connect to the database. It was still configured with mlab database, but i guess it was taken down, so authentication error -> process exits. To know what the real problem is well have to wait on @sacdallago

sacdallago commented 7 years ago

It seems to be up and running to me :) Can you confirm that the predictions are visible?

jonas-he commented 7 years ago

@sacdallago are you talking about the dockerhub thing or predictemall.online ? The first is down for me the latter is up. However I see no possibility to request predictions

sacdallago commented 7 years ago

@jonas-he 😮 how didn't I reply to that. Sorry :(

Yes, yes. Everything is up where it should be, and down where it should be down :) Why don't you see a possibility to request predictions?

jonas-he commented 7 years ago

@sacdallago i can turn prediction toggle on/off and change the time range but nothing happens on the map as well as in terms of network requests (i see requests for sightings but not for predictions).

Another thing is that there is still some dummy running, which is generating tons of pokemobs in munich.

samitsv commented 7 years ago

updated the api for prediction, https://github.com/PokemonGoers/PokeData/pull/199, but one of the build fails https://travis-ci.org/PokemonGoers/PokeData/jobs/184915843