Preferred way of importing?

[phdowling]

Hey guys,

I make use of the Twitter stream you open in your project. Currently I import it by cloning your repo to a certain location and doing an import of the file the stream is opened in:

var PokemonTwitter = require("../PokeData/app/controllers/filler/twitter");

However, for production, this of course needs to change - is your project on npm already? How should I expect to import the module I need? Also pinging @PokemonGoers/catch-em-all for this.

[samitsv]

we do not have a plan to ship a npm package, and if you want get the data generated by twitter, then you can do so using http://pokedata.c4e3f8c7.svc.dockerapp.io:65014/doc/#api-PokemonSighting-GetSightingBySource or http://pokedata.c4e3f8c7.svc.dockerapp.io:65014/api/pokemon/sighting/search?source=twitter

[phdowling]

The REST API won't do in this case, we need live tweets, i.e. the raw feed. I guess we can just start a seperate feed in our own module and be fairly independent from yours there. Another way to go would be for us to PR the code we wrote into your repo, then we could access all data sources freely - not sure what the best way to go is here. @gyachdav or @sacdallago , any suggestions here?

[gyachdav]

I recommend you stick with analyzing the Streaming API on your own, separated from project A.

[sacdallago]

@phdowling yup, I would suggest you create a npm package that the guys from A can use on the tweets to perform the sentiment analysis. They are listening to the tweets anyway, I imagine it to be something like adding a function (from your package) which calculates the score, deffer a write to a dedicated collection ({tweetId: xyz, sentiment: +1.2} and that's it.

Or, eventually, the guys from A can implement a npm runner to perform the score analysis for all the tweets.

@samitsv are you storing the RAW tweets somewhere? I can't remember

[samitsv]

@sacdallago raw tweets are not being stored

[sacdallago]

@samitsv MH. It might make sense that they are :) @gyachdav we did this last semester, but I'm not entirely sure it makes sense.

Taking the idea from https://github.com/PokemonGoers/HashPokemonGo/issues/12#issuecomment-246899760 maybe extend that object ({twitterId: xyz, sentiment: 1.23}) with:

• the original text of the tweet
• lat/lng info if available
• timestamp of the tweet

and save that?

Also @samitsv , checking out the data ccoming from twitter: why store null lat/lng values? Aka:

{"_id":"57c936554e3bd9e1024717fc","source":"TWITTER","appearedOn":"2016-09-02T08:20:37.452Z","__v":0,"pokemonId":7,"location":null}

It makes sense in the collection mentioned above, but no sense in the sightings collection... just knowing that they spawned somewhere in the globe seems like the least informative feature ever to me :laughing: @goldbergtatyana do you agree?

[samitsv]

@sacdallago about null lat/lng, maybe important if someone wants to see the number of appearances of pokemons or want to know the appearance time of pokemon, like one pokemon appears mostly on day and not during night time?

[sacdallago]

hmm.. @goldbergtatyana @juanmirocks @gyachdav opinions?

[goldbergtatyana]

valid points from both of you. null values of long/lat

• are not useful for prediction, since we can neither infer exact location, nor weather or topological features (e.g. proximity to water)
• could be useful for general statistics on the reportings of pokemons

If we have an issue with storage space, then I would recommend to not store sightings with empty locations. If there is no issue, then yes store them :smile:

[swathi-ssunder]

@goldbergtatyana @sacdallago To add further to the discussion, we had to store null values for location even when there is no data for it(without skipping the location key altogether) since we have indexed data based on location data field for geospatial queries. So if we rather prefer not to have these entries, then we could skip/ignore the record altogether.

[goldbergtatyana]

thanks @swathi-ssunder ! Again, for predictions these records are useless. For statistics on sightings they are nice. However, for doing the statistics these data will be most likely used just once. Therefore, I would suggest to get rid of entries with empty location altogether.

[samitsv]

@phdowling @sacdallago i think i forgot about the part with twitter texts being stored. So do we store it or not? I see some sentiment api is already implemented, @phdowling could you let me know how I could access it?

[sacdallago]

@samitsv I would store it. Better to have this data than not.. @gyachdav @goldbergtatyana @juanmirocks opinions?

[samitsv]

being worked on https://github.com/PokemonGoers/PokeData/pull/162