PokemonGoers / PokeData

In this project you will scrape as much data as you can get about the *actual* sightings of Pokemons. As it turns out, players all around the world started reporting sightings of Pokemons and are logging them into a central repository (i.e. a database). We want to get this data so we can train our machine learning models. You will of course need to come up with other data sources not only for sightings but also for other relevant details that can be used later on as features for our machine learning algorithm (see Project B). Additional features could be air temperature during the given timestamp of sighting, location close to water, buildings or parks. Consult with Pokemon Go expert if you have such around you and come up with as many features as possible that describe a place, time and name of a sighted Pokemon. Another feature that you will implement is a twitter listener: You will use the twitter streaming API (https://dev.twitter.com/streaming/public) to listen on a specific topic (for example, the #foundPokemon hashtag). When a new tweet with that hashtag is written, an event will be fired in your application checking the details of the tweet, e.g. location, user, time stamp. Additionally, you will try to parse formatted text from the tweets to construct a new “seen” record that consequently will be added to the database. Some of the attributes of the record will be the Pokemon's name, location and the time stamp. Additional data sources (here is one: https://pkmngowiki.com/wiki/Pok%C3%A9mon) will also need to be integrated to give us more information about Pokemons e.g. what they are, what’s their relationship, what they can transform into, which attacks they can perform etc.
Apache License 2.0
9 stars 6 forks source link

Database oh database #191

Closed sacdallago closed 8 years ago

sacdallago commented 8 years ago

Dear all,

I am setting up the final docker machines and such. I'm having problems with the database initialization. I created a docker container that executes the build line:

npm run build --collection=pokemon

The logs after the execution are:

[pokedataBASHdevelop-1]2016-10-23T02:41:22.780492421Z npm info it worked if it ends with ok 
[pokedataBASHdevelop-1]2016-10-23T02:41:22.780735592Z npm info using npm@2.14.2 
[pokedataBASHdevelop-1]2016-10-23T02:41:22.780917768Z npm info using node@v4.0.0 
[pokedataBASHdevelop-1]2016-10-23T02:41:22.962375385Z npm info prebuild pokemongo-api@0.0.1 
[pokedataBASHdevelop-1]2016-10-23T02:41:22.966944876Z npm info build pokemongo-api@0.0.1 
[pokedataBASHdevelop-1]2016-10-23T02:41:22.968729799Z  
[pokedataBASHdevelop-1]2016-10-23T02:41:22.968743226Z > pokemongo-api@0.0.1 build /usr/src/app 
[pokedataBASHdevelop-1]2016-10-23T02:41:22.968747122Z > NODE_ENV=production node scripts/build.js 
[pokedataBASHdevelop-1]2016-10-23T02:41:22.968750306Z  
[pokedataBASHdevelop-1]2016-10-23T02:41:24.912365411Z npm info postbuild pokemongo-api@0.0.1 
[pokedataBASHdevelop-1]2016-10-23T02:41:24.916616405Z npm info ok

But as you can see, the result is pretty different:

http://pokedata.c4e3f8c7.svc.dockerapp.io:65014/api/pokemon/ vs https://api.predictemall.online/api/pokemon/

Any idea? The stack file in case someone wonders:

catchtemalldevelop:
  autoredeploy: true
  environment:
    - API_ENDPOINT=api.predictemall.online
    - WEBSOCKET_ENDPOINT=api.predictemall.online:65024
  image: 'pokemongoers/catch-em-all:develop'
  ports:
    - '6010:8080'
mongo:
  image: 'mongo:latest'
pokedataINITdevelop:
  autoredeploy: true
  command: npm run build --collection=pokemon
  environment:
    - ACCESS_TOKEN=########
    - ACCESS_TOKEN_KEY=########
    - CONSUMER_KEY=########
    - CONSUMER_SECRET=########
    - MLAB_COLLECTION=pokemongo
    - MLAB_URI=mongo
  image: 'pokemongoers/pokedata:develop'
  links:
    - mongo
pokedatadevelop:
  autoredeploy: true
  environment:
    - ACCESS_TOKEN=########
    - ACCESS_TOKEN_KEY=########
    - CONSUMER_KEY=########
    - CONSUMER_SECRET=########
    - MLAB_COLLECTION=pokemongo
    - MLAB_URI=mongo
  image: 'pokemongoers/pokedata:develop'
  links:
    - mongo
  ports:
    - '4050:8080'
    - '3322:65024'
pokedatatwitterdevelop:
  autoredeploy: true
  command: npm run listen --collection=twitter
  environment:
    - ACCESS_TOKEN=########
    - ACCESS_TOKEN_KEY=########
    - CONSUMER_KEY=########
    - CONSUMER_SECRET=########
    - MLAB_COLLECTION=pokemongo
    - MLAB_URI=mongo
  image: 'pokemongoers/pokedata:develop'
  links:
    - mongo
jonas-he commented 8 years ago

@sacdallago to me those two responses look the same

sacdallago commented 8 years ago

WTF. I swear to god this was not working this morning!

sacdallago commented 8 years ago

Closing for now. Mysteries of life? My guess is that it had to do with caching, and since the first answer got back empty, until the cache refreshed, the subsequent requests also got back empty.

jonas-he commented 8 years ago

yeah thats probably the cause