search

PokemonGoers / PokeData

In this project you will scrape as much data as you can get about the *actual* sightings of Pokemons. As it turns out, players all around the world started reporting sightings of Pokemons and are logging them into a central repository (i.e. a database). We want to get this data so we can train our machine learning models. You will of course need to come up with other data sources not only for sightings but also for other relevant details that can be used later on as features for our machine learning algorithm (see Project B). Additional features could be air temperature during the given timestamp of sighting, location close to water, buildings or parks. Consult with Pokemon Go expert if you have such around you and come up with as many features as possible that describe a place, time and name of a sighted Pokemon. Another feature that you will implement is a twitter listener: You will use the twitter streaming API (https://dev.twitter.com/streaming/public) to listen on a specific topic (for example, the #foundPokemon hashtag). When a new tweet with that hashtag is written, an event will be fired in your application checking the details of the tweet, e.g. location, user, time stamp. Additionally, you will try to parse formatted text from the tweets to construct a new “seen” record that consequently will be added to the database. Some of the attributes of the record will be the Pokemon's name, location and the time stamp. Additional data sources (here is one: https://pkmngowiki.com/wiki/Pok%C3%A9mon) will also need to be integrated to give us more information about Pokemons e.g. what they are, what’s their relationship, what they can transform into, which attacks they can perform etc.
Apache License 2.0
9 stars 6 forks source link

endpoint for predictions #183

Open gyachdav opened 7 years ago

gyachdav commented 7 years ago

Hello there pokedata, it has been a while 😆

I didn't see any spec in your API that will allow the map team to retrieve predictions. Are predictions included in the sighting data? is there a parameter that distinguish predictions from sightings?

Same questions apply to the model that holds prediction data.

Please respond ASAP.

swathi-ssunder commented 7 years ago

@gyachdav - Hello, the pokemon sightings from the API(that are fetched from the db) do not include the predictions.

sacdallago commented 7 years ago

Yes, because they should be generated on the fly :)

I guess once you can run the predictions, aka call the right function, and get an output, it will be pretty straightforward to implement the route.. Although: you could already implement the route, and imagine to have some magic predictShit(lat, lng, timestamp) function that returns the predictions and you later expose this to a GET request :)

bensLine commented 7 years ago

Btw. predictions for predict(lat, lng, timestamp) will be available as an array of the following objects: {"pokemonId":"16","confidence":"0.242","latitude":11.6088567,"longitude":48.1679286}

vivek-sethia commented 7 years ago

Refer #187 . Changes will need to be made after the predictions are made available later.

gyachdav commented 7 years ago

referencing https://github.com/PokemonGoers/PredictPokemon-2/issues/67 again since I did not get any answer so far.

According to @Aurel-Roci the integration of predictions into this project is complete. On the demo app I do not see any prediction.

What is the status then? Is there a demo URL that will show that this integration is really done?

@PokemonGoers/pokedata @PokemonGoers/predictpokemon-2

gyachdav commented 7 years ago

@sacdallago please make sure this is handled in the next 24 hours thanks

sacdallago commented 7 years ago

Guys, can you give us an endpoint? :)

vivek-sethia commented 7 years ago

Referring to this, the route is already implemented.

As mentioned in the above comments, we only need to know which function needs to be invoked and were waiting on @PokemonGoers/predictpokemon-2 for this.

Correct me if I am missing something here.

Aurel-Roci commented 7 years ago

The function is mentioned in the comments above: function: predict(lat, lng, timestamp) response : {"pokemonId":"16","confidence":"0.242","latitude":11.6088567,"longitude":48.1679286}

vivek-sethia commented 7 years ago

@Aurel-Roci - And how are we supposed to include your project? Is it available as an npm package?

Aurel-Roci commented 7 years ago

@vivek-sethia Yes it is available as a package, I mentioned that in the issue with adding the java in the docker container. I thought this was already known by you guys since @jonas-he mentioned the package.

vivek-sethia commented 7 years ago

@Aurel-Roci - seems like some miscommunication. Will integrate the package and commit in a while.

vivek-sethia commented 7 years ago

@Aurel-Roci - I am trying to integrate and get the following error about the file prediction_feature_config.json. Is there something that needs to be done additionally?

2016-10-28T18:05:13.742Z started prediction script, init DC
fs.js:634
  return binding.open(pathModule._makeLong(path), stringToFlags(flags), mode);
                 ^

Error: ENOENT: no such file or directory, open 'prediction_feature_config.json'
    at Error (native)
    at Object.fs.openSync (fs.js:634:18)
    at Object.fs.readFileSync (fs.js:502:33)
    at fileToJson (PokeData/node_modules/predict-pokemon/dataSet_creator.js:364:23)
    at Object.DC.init (PokeData/node_modules/predict-pokemon/dataSet_creator.js:30:22)
    at PokeData/node_modules/predict-pokemon/prediction.js:31:8
    at Object.<anonymous> (PokeData/node_modules/predict-pokemon/prediction.js:241:3)
    at Module._compile (module.js:541:32)
    at Object.Module._extensions..js (module.js:550:10)
    at Module.load (module.js:456:32)
    at tryModuleLoad (module.js:415:12)
    at Function.Module._load (module.js:407:3)
Aurel-Roci commented 7 years ago

@vivek-sethia Yeah, sorry forgot to tell you that you have to unzip json/pokestop_groups.7z file in the prediction npm package location. It will take some time to initialize the module. Let me know if it works

vivek-sethia commented 7 years ago

@Aurel-Roci - I already extracted it after referring to https://github.com/PokemonGoers/PredictPokemon-2/issues/67#issuecomment-254126234.

And still got this error, so wanted to know if there is anything else?

Aurel-Roci commented 7 years ago

@vivek-sethia I cant seem to reproduce that error. The only thing that comes to my mind is to add / before the file name /prediction_feature_config.json, like so.

vivek-sethia commented 7 years ago

@Aurel-Roci - Where exactly am I supposed to make this change ?

Aurel-Roci commented 7 years ago

@vivek-sethia in node_modules/predict-pokemon/prediction.js. Line 31

vivek-sethia commented 7 years ago

@Aurel-Roci - Doesn't work. Well, do I have to copy the extracted pokestop_groups.json to some other location?

Aurel-Roci commented 7 years ago

@vivek-sethia from what I see from the stacktrace the problem is with prediction_feature_config.json. For some reason it cannot find it, which I dont understand

vivek-sethia commented 7 years ago

@Aurel-Roci - Exactly, I already verified that the prediction_feature_config.json file exists at the expected location. I have committed the change. Maybe you could check once if there is something missing.

Aurel-Roci commented 7 years ago

@vivek-sethia the require('predict-pokemon'), should be require('predict-pokemon').predictor.

vivek-sethia commented 7 years ago

@Aurel-Roci - Already tried that too. Same error as above.

vivek-sethia commented 7 years ago

@Aurel-Roci - Maybe you could try the code yourself and see it it works.

semioniy commented 7 years ago

@vivek-sethia I guess we found an error, stay tuned, we try to solve it.

Aurel-Roci commented 7 years ago

@vivek-sethia Thanks to @semioniy, this is what we found, first what has to be done is move the content of the predict-pokemon package from Project/node_modules/predict-pokemon directly into the project directory. For some reason the paths do not seem to work correctly in the package. Also change the tzwhere like described here https://www.npmjs.com/package/predict-pokemon

vivek-sethia commented 7 years ago

@Aurel-Roci Can you please make this change and test it yourself? Seems to be much easier that way. You can amend my commit if needed.

Aurel-Roci commented 7 years ago

@vivek-sethia @semioniy already tested it and it works

vivek-sethia commented 7 years ago

@Aurel-Roci @semioniy Can I then merge my commit ?

Aurel-Roci commented 7 years ago

@vivek-sethia did you move the contents of the package directly to the project?

vivek-sethia commented 7 years ago

@Aurel-Roci - You mean in the commit? No, because we don't push the node_modules.

Aurel-Roci commented 7 years ago

@sacdallago can you help us with this problem. When the prediction package is published the paths seem not work properly. What is the cause of that?

vivek-sethia commented 7 years ago

@Aurel-Roci - Also, have you tested by including your npm package as a dependency in another project?

semioniy commented 7 years ago

@vivek-sethia that's what I did to test. It works after moving files to parent-parent directory, unzipping zip-files and changing tzwhere dependence. From a box it doesn't work.

vivek-sethia commented 7 years ago

@sacdallago - What do you suggest? Should we commit the entire predict-pokemon package in this repo too?

vivek-sethia commented 7 years ago

@sacdallago , @gyachdav - Any inputs on this?

sacdallago commented 7 years ago

Sorry guys I'm currently on a conference. I'll be able to give you valuable input tomorrow!

jonas-he commented 7 years ago

@sacdallago just reminding you that you wanted to give input

sacdallago commented 7 years ago

Thanks @jonas-he for reminding, sorry about that. Looks like a relative path problem, to me.

@semioniy @Aurel-Roci I've noticed a whole bunch of no-no's in your code:

  1. https://github.com/PokemonGoers/PredictPokemon-2/blob/develop/prediction.js#L8 NEVER use constant URIs. That link is gonna be down in a couple days, what then? You should give a way of changing this via init() options.
  2. https://github.com/PokemonGoers/PredictPokemon-2/blob/develop/prediction.js#L23 this is relative to the discussion here: NEVER, EVER load resources without specifying the relative file path! Always chain paths with the __dirname through path. More at: http://stackoverflow.com/questions/8131344/what-is-the-difference-between-dirname-and-in-node-js
  3. Point 2 will solve also the current issue, if you path chain https://github.com/PokemonGoers/PredictPokemon-2/blob/develop/prediction.js#L31
vivek-sethia commented 7 years ago

@Aurel-Roci , @semioniy , @sacdallago - Once the issues are fixed, we could merge #194 and then close this issue.

Aurel-Roci commented 7 years ago

@vivek-sethia The new package is released, you can install it and try it out. And let me now if it works

vivek-sethia commented 7 years ago

@sacdallago @semioniy @Aurel-Roci - Lots of errors after installing the new package and extracting the json/pokestop_groups.7z file. Some of the initial errors are below.

  1. In prediction.js, File prediction_feature_config.json could not be found on line 41. This is because of a missing / before the file name.
  2. After fixing the above error, similar error in file dataSet_creator.js on line 31.
  3. After fixing the second error, file not found in terrain_features.js on line 6. This time, the file actually does not exist at the specified path but at a totally different location in resources.

Since the integration is already done in our project, I would suggest, that you fix all these issues and just change the package version in package.json of our project, test it and push it.

sacdallago commented 7 years ago

@Aurel-Roci @semioniy please fix this ASAP

Aurel-Roci commented 7 years ago

@vivek-sethia Now the package is working. Needs to be checked again when the API is up, but it should work then too

sacdallago commented 7 years ago

Mh @jonas-he can you check that this really is the case? If so, I will trigger a new docker deployment which will contain the update package

swathi-ssunder commented 7 years ago

@Aurel-Roci - It now refers to the latest version 0.1.6

sacdallago commented 7 years ago

Still not working:

[pokedatadevelop-1]2016-11-07T10:16:17.644914864Z npm info it worked if it ends with ok
[pokedatadevelop-1]2016-11-07T10:16:17.645389310Z npm info using npm@2.14.2
[pokedatadevelop-1]2016-11-07T10:16:17.645692252Z npm info using node@v4.0.0
[pokedatadevelop-1]2016-11-07T10:16:19.096545617Z npm info prestart pokemongo-api@0.0.1
[pokedatadevelop-1]2016-11-07T10:16:19.162998917Z npm info start pokemongo-api@0.0.1
[pokedatadevelop-1]2016-11-07T10:16:19.170461598Z 
[pokedatadevelop-1]2016-11-07T10:16:19.170491476Z > pokemongo-api@0.0.1 start /usr/src/app
[pokedatadevelop-1]2016-11-07T10:16:19.170497735Z > NODE_ENV=production node app.js
[pokedatadevelop-1]2016-11-07T10:16:19.170502501Z 
[pokedatadevelop-1]2016-11-07T10:16:29.305042849Z 2016-11-07T10:16:29.271Z started prediction script, init DC
[pokedatadevelop-1]2016-11-07T10:16:51.054250201Z fs.js:549
[pokedatadevelop-1]2016-11-07T10:16:51.054310117Z   return binding.open(pathModule._makeLong(path), stringToFlags(flags), mode);
[pokedatadevelop-1]2016-11-07T10:16:51.054321872Z                                                   ^
[pokedatadevelop-1]2016-11-07T10:16:51.054327505Z 
[pokedatadevelop-1]2016-11-07T10:16:51.054332621Z Error: ENOENT: no such file or directory, open '/usr/src/app/node_modules/predict-pokemon/feature_sources/../json/pokestop_groups.json'
[pokedatadevelop-1]2016-11-07T10:16:51.054338077Z     at Error (native)
[pokedatadevelop-1]2016-11-07T10:16:51.054343085Z     at Object.fs.openSync (fs.js:549:18)
[pokedatadevelop-1]2016-11-07T10:16:51.054347806Z     at Object.fs.readFileSync (fs.js:397:15)
[pokedatadevelop-1]2016-11-07T10:16:51.054352990Z     at fileToJson (/usr/src/app/node_modules/predict-pokemon/feature_sources/place_features.js:139:23)
[pokedatadevelop-1]2016-11-07T10:16:51.054358426Z     at /usr/src/app/node_modules/predict-pokemon/feature_sources/place_features.js:7:26
[pokedatadevelop-1]2016-11-07T10:16:51.054363620Z     at Object.<anonymous> (/usr/src/app/node_modules/predict-pokemon/feature_sources/place_features.js:142:3)
[pokedatadevelop-1]2016-11-07T10:16:51.054369031Z     at Module._compile (module.js:434:26)
[pokedatadevelop-1]2016-11-07T10:16:51.054373923Z     at Object.Module._extensions..js (module.js:452:10)
[pokedatadevelop-1]2016-11-07T10:16:51.054380581Z     at Module.load (module.js:355:32)
[pokedatadevelop-1]2016-11-07T10:16:51.054385643Z     at Function.Module._load (module.js:310:12)
[pokedatadevelop-1]2016-11-07T10:16:51.054390497Z     at Module.require (module.js:365:17)
[pokedatadevelop-1]2016-11-07T10:16:51.054395194Z     at require (module.js:384:17)
[pokedatadevelop-1]2016-11-07T10:16:51.054399834Z     at /usr/src/app/node_modules/predict-pokemon/dataSet_creator.js:52:30
[pokedatadevelop-1]2016-11-07T10:16:51.054465777Z     at Array.forEach (native)
[pokedatadevelop-1]2016-11-07T10:16:51.054475841Z     at Object.DC.init (/usr/src/app/node_modules/predict-pokemon/dataSet_creator.js:38:32)
[pokedatadevelop-1]2016-11-07T10:16:51.054481277Z     at /usr/src/app/node_modules/predict-pokemon/prediction.js:42:8
[pokedatadevelop-1]2016-11-07T10:16:51.339961005Z 
[pokedatadevelop-1]2016-11-07T10:16:51.358394394Z npm info pokemongo-api@0.0.1 Failed to exec start script
[pokedatadevelop-1]2016-11-07T10:16:51.359595861Z npm ERR! Linux 4.4.0-42-generic
[pokedatadevelop-1]2016-11-07T10:16:51.360807991Z npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "start"
[pokedatadevelop-1]2016-11-07T10:16:51.361674069Z npm ERR! node v4.0.0
[pokedatadevelop-1]2016-11-07T10:16:51.369809860Z npm ERR! npm  v2.14.2
[pokedatadevelop-1]2016-11-07T10:16:51.371324803Z npm ERR! code ELIFECYCLE
[pokedatadevelop-1]2016-11-07T10:16:51.372038133Z npm ERR! pokemongo-api@0.0.1 start: `NODE_ENV=production node app.js`
[pokedatadevelop-1]2016-11-07T10:16:51.372839100Z npm ERR! Exit status 1
[pokedatadevelop-1]2016-11-07T10:16:51.374822582Z npm ERR! 
[pokedatadevelop-1]2016-11-07T10:16:51.375998085Z npm ERR! Failed at the pokemongo-api@0.0.1 start script 'NODE_ENV=production node app.js'.
[pokedatadevelop-1]2016-11-07T10:16:51.383628766Z npm ERR! This is most likely a problem with the pokemongo-api package,
[pokedatadevelop-1]2016-11-07T10:16:51.387430954Z npm ERR! not with npm itself.
[pokedatadevelop-1]2016-11-07T10:16:51.391594875Z npm ERR! Tell the author that this fails on your system:
[pokedatadevelop-1]2016-11-07T10:16:51.396719920Z npm ERR!     NODE_ENV=production node app.js
[pokedatadevelop-1]2016-11-07T10:16:51.406632817Z npm ERR! You can get their info via:
[pokedatadevelop-1]2016-11-07T10:16:51.410209885Z npm ERR!     npm owner ls pokemongo-api
[pokedatadevelop-1]2016-11-07T10:16:51.412017196Z npm ERR! There is likely additional logging output above.
[pokedatadevelop-1]2016-11-07T10:16:51.423986162Z 
[pokedatadevelop-1]2016-11-07T10:16:51.435966064Z npm ERR! Please include the following file with any support request:
[pokedatadevelop-1]2016-11-07T10:16:51.447907497Z npm ERR!     /usr/src/app/npm-debug.log
swathi-ssunder commented 7 years ago

@sacdallago - The above file not found error seems to be because the json/pokestop_groups.7z file is not unzipped. @Aurel-Roci - Is it possible for you to directly include the pokestop_groups.json file?

bensLine commented 7 years ago

Great. What would life be without this constant predict pokemon struggle x)

Did you by chance not unzip the zips in the data and json directory? /usr/src/app/node_modules/predict-pokemon/feature_sources/../json/pokestop_groups.json Is at least zipped if you just checked the package out.

swathi-ssunder commented 7 years ago

@bensLine - node_modules folder is not pushed to the repository. The packages are installed using the commands in the Dockerfile. (npm install).