search

PokemonGoers / PokeData

In this project you will scrape as much data as you can get about the *actual* sightings of Pokemons. As it turns out, players all around the world started reporting sightings of Pokemons and are logging them into a central repository (i.e. a database). We want to get this data so we can train our machine learning models. You will of course need to come up with other data sources not only for sightings but also for other relevant details that can be used later on as features for our machine learning algorithm (see Project B). Additional features could be air temperature during the given timestamp of sighting, location close to water, buildings or parks. Consult with Pokemon Go expert if you have such around you and come up with as many features as possible that describe a place, time and name of a sighted Pokemon. Another feature that you will implement is a twitter listener: You will use the twitter streaming API (https://dev.twitter.com/streaming/public) to listen on a specific topic (for example, the #foundPokemon hashtag). When a new tweet with that hashtag is written, an event will be fired in your application checking the details of the tweet, e.g. location, user, time stamp. Additionally, you will try to parse formatted text from the tweets to construct a new “seen” record that consequently will be added to the database. Some of the attributes of the record will be the Pokemon's name, location and the time stamp. Additional data sources (here is one: https://pkmngowiki.com/wiki/Pok%C3%A9mon) will also need to be integrated to give us more information about Pokemons e.g. what they are, what’s their relationship, what they can transform into, which attacks they can perform etc.
Apache License 2.0
9 stars 6 forks source link

Expose HashPokemonGo websocket port #168

Closed johartl closed 7 years ago

johartl commented 8 years ago

Hi everyone, after having integrated the work from the @PokemonGoers/hashpokemongo team in PR #162 you still need to make their websocket port avavilable in your Docker container.

sacdallago commented 8 years ago

Exposed and mapped to 65024 --> http://pokedata.c4e3f8c7.svc.dockerapp.io:65024/

johartl commented 8 years ago

Is the websocket port actually opened? Shouldn't I get the socket.io library when requesting http://pokedata.c4e3f8c7.svc.dockerapp.io:65024/socket.io/socket.io.js ? Or am I missing something here?

sacdallago commented 8 years ago

I haven't tested it, actually. But I was also wondering, don't know if it's behaving as should, but from docker cloud:

image image

So: I opened the port + redeployed the machine, and docker cloud is saying it's alright. Server is saying:

# docker ps
CONTAINER ID        IMAGE                                   COMMAND                  CREATED             STATUS              PORTS                                              NAMES
8a3787d4f05d        pokemongoers/pokedata:develop           "npm start"              2 hours ago         Up 2 hours          0.0.0.0:65024->3322/tcp, 0.0.0.0:65014->8080/tcp   pokedata-1.baab41d0
johartl commented 8 years ago

Yes the configuration looks good. My assumption is that the port 3322 is not opened by the node server at all. The important piece of code seems to be

// app/controllers/filler/hashPokemonGo.js
const io = require('socket.io')(config.hashPokemonGoWebSocketPort);

Can someone verify if this code is even executed? I can't find any other module require'ing this file.

johartl commented 8 years ago

Can someone check this please? @PokemonGoers/pokedata @PokemonGoers/hashpokemongo Could we also reopen this issue so it doesn't get lost.

jonas-he commented 8 years ago

@johartl sorry just came across this issue. From my understanding it seems like "npm start" does not start the hashPokemonGo thing, you need to run "npm run listen --collection=hashPokemonGo" @sacdallago maybe you did not include that.

sacdallago commented 7 years ago

Nope, I did not include that :) So it's entirely a different process than the original api... mmhh, does this make sense? @jonas-he

johartl commented 7 years ago

It should be possible to run both express and socket.io in the same process. Having two different processes running in the same Docker container seems really awkward.

phdowling commented 7 years ago

Definitely possible, we should use a pattern like this:

server = http.createServer(app)
io = require('socket.io').listen(server)
johartl commented 7 years ago

Would this also mean that they are both using the same port?

jonas-he commented 7 years ago

Im not an expert on docker and wether or not you should run several processes in one container or use separate containers... What we could do is: add a call to the hashpokemongo "module" inside the startup script that gets called when executing "npm start". since node is asynchronous it should start both the hashpokemongo and the API...

johartl commented 7 years ago

@jonas-he Sure that should work as well. Can you take care of this?

sacdallago commented 7 years ago

In principle docker is supposed to wrap one service, but in my head the API and the websocket are one service, just using two protocols -but, again: conceptually they serve the same purpose-

I originally thought that websocket and traditional REST calls wouldn't be allowed to exist on the same port, but it seems they can: http://stackoverflow.com/questions/29497725/can-websockets-and-http-server-both-run-on-the-same-port-number and http://stackoverflow.com/questions/13791050/is-it-possible-to-enable-tcp-http-and-websocket-all-using-the-same-port

HOWEVER it is pretty inexpensive to open two ports for the same service. This is the case with many services, anyway ;) So, if making the two protocols exist on the same port is too expensive (time, headaches,..), just code the two protocols into the npm start script using the ports previously found in the Dockerfile (as they are already being exposed anyway).

johartl commented 7 years ago

Just make this work somehow please :D

jonas-he commented 7 years ago

see #176 @johartl @sacdallago

sacdallago commented 7 years ago

If #176 get's merged, we can see if the autoredeployed image on docker cloud collaborates :)

vivek-sethia commented 7 years ago

@sacdallago its merged

johartl commented 7 years ago

Again, shouldn't the socket.io client library be served at http://pokedata.c4e3f8c7.svc.dockerapp.io:65024/socket.io/socket.io.js now? Or am I missing something here?

johartl commented 7 years ago

Oh nevermind, your API seems to be down entirely :/ http://pokedata.c4e3f8c7.svc.dockerapp.io:65014/doc/

sacdallago commented 7 years ago

Yup, I have the same feeling ahah

sacdallago commented 7 years ago

Docker cloud logs say:

2016-10-06T20:04:46.223292421Z npm info it worked if it ends with ok 
2016-10-06T20:04:46.223809507Z npm info using npm@2.14.2 
2016-10-06T20:04:46.224150180Z npm info using node@v4.0.0 
2016-10-06T20:04:46.490650676Z npm info prestart pokemongo-api@0.0.1 
2016-10-06T20:04:46.499430995Z npm info start pokemongo-api@0.0.1 
2016-10-06T20:04:46.501060874Z  
2016-10-06T20:04:46.501077556Z > pokemongo-api@0.0.1 start /usr/src/app 
2016-10-06T20:04:46.501085079Z > NODE_ENV=production node app.js 
2016-10-06T20:04:46.501091503Z  
2016-10-06T20:04:48.058532023Z /usr/src/app/node_modules/mongoose/node_modules/mongodb/lib/utils.js:98 
2016-10-06T20:04:48.058589576Z     process.nextTick(function() { throw err; }); 
2016-10-06T20:04:48.058601658Z                                   ^ 
2016-10-06T20:04:48.058612233Z  
2016-10-06T20:04:48.058638529Z Error: Cannot find module 'twitter' 
2016-10-06T20:04:48.058649364Z     at Function.Module._resolveFilename (module.js:336:15) 
2016-10-06T20:04:48.058659779Z     at Function.Module._load (module.js:286:25) 
2016-10-06T20:04:48.058670003Z     at Module.require (module.js:365:17) 
2016-10-06T20:04:48.058680195Z     at require (module.js:384:17) 
2016-10-06T20:04:48.058690567Z     at Object.<anonymous> (/usr/src/app/app/controllers/filler/hashPokemonGo.js:3:21) 
2016-10-06T20:04:48.058701960Z     at Module._compile (module.js:434:26) 
2016-10-06T20:04:48.058712359Z     at Object.Module._extensions..js (module.js:452:10) 
2016-10-06T20:04:48.058722479Z     at Module.load (module.js:355:32) 
2016-10-06T20:04:48.058732611Z     at Function.Module._load (module.js:310:12) 
2016-10-06T20:04:48.058747237Z     at Module.require (module.js:365:17) 
2016-10-06T20:04:48.058758145Z     at require (module.js:384:17) 
2016-10-06T20:04:48.058768467Z     at NativeConnection.<anonymous> (/usr/src/app/startUp.js:19:38) 
2016-10-06T20:04:48.058779227Z     at emitNone (events.js:67:13) 
2016-10-06T20:04:48.058790336Z     at NativeConnection.emit (events.js:166:7) 
2016-10-06T20:04:48.058796772Z     at open (/usr/src/app/node_modules/mongoose/lib/connection.js:518:11) 
2016-10-06T20:04:48.058803181Z     at /usr/src/app/node_modules/mongoose/lib/connection.js:524:7 
2016-10-06T20:04:48.071046541Z  
2016-10-06T20:04:48.074960086Z npm info pokemongo-api@0.0.1 Failed to exec start script 
2016-10-06T20:04:48.075181577Z npm ERR! Linux 3.13.0-85-generic 
2016-10-06T20:04:48.075464197Z npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "start" 
2016-10-06T20:04:48.075667558Z npm ERR! node v4.0.0 
2016-10-06T20:04:48.075869283Z npm ERR! npm  v2.14.2 
2016-10-06T20:04:48.076081478Z npm ERR! code ELIFECYCLE 
2016-10-06T20:04:48.076291456Z npm ERR! pokemongo-api@0.0.1 start: `NODE_ENV=production node app.js` 
2016-10-06T20:04:48.076454093Z npm ERR! Exit status 1 
2016-10-06T20:04:48.076626099Z npm ERR!  
2016-10-06T20:04:48.076971152Z npm ERR! Failed at the pokemongo-api@0.0.1 start script 'NODE_ENV=production node app.js'. 
2016-10-06T20:04:48.077184708Z npm ERR! This is most likely a problem with the pokemongo-api package, 
2016-10-06T20:04:48.077399704Z npm ERR! not with npm itself. 
2016-10-06T20:04:48.077816646Z npm ERR! Tell the author that this fails on your system: 
2016-10-06T20:04:48.078014002Z npm ERR!     NODE_ENV=production node app.js 
2016-10-06T20:04:48.078599673Z npm ERR! You can get their info via: 
2016-10-06T20:04:48.078972304Z npm ERR!     npm owner ls pokemongo-api 
2016-10-06T20:04:48.079294656Z npm ERR! There is likely additional logging output above. 
2016-10-06T20:04:48.083108305Z  
2016-10-06T20:04:48.083637199Z npm ERR! Please include the following file with any support request: 
2016-10-06T20:04:48.087125463Z npm ERR!     /usr/src/app/npm-debug.log
sacdallago commented 7 years ago

https://github.com/PokemonGoers/PokeData/blob/develop/app/controllers/filler/hashPokemonGo.js#L3

AlexanderLill commented 7 years ago

@sacdallago I think the package.json of the PokeData team might also need the "twitter": "^1.2.5" dependency? Although the HashPokemonGo team has it in their package.json.

samitsv commented 7 years ago

added module https://github.com/PokemonGoers/PokeData/pull/177

jonas-he commented 7 years ago

so the twitter dependency is merged now, however dockerapp DNS is not working. Theres nothing we can do about it but wait. We already had this issue in the past.

AlexanderLill commented 7 years ago

@jonas-he @sacdallago is there a solution for this issue? E.g. using the IP address instead of the docker.io DNS lookup? And can someone check if it really is a DNS issue now, and that the docker container started correctly? :)

jonas-he commented 7 years ago

@AlexanderLill build status is success: https://hub.docker.com/r/pokemongoers/pokedata/builds/ It is indeed the DNS service of dockerapp.io that does not give an answer to the question "what is the IP of pokedata.c4e3f8c7.svc.dockerapp.io ?" So their nameservers are up (e.g. ns1.dockerapp.io) and they also give an answer for e.g. your catch-em-all container but unfortunately not for ours.

Similiar issue from last month: https://forums.docker.com/t/dockerapp-io-dns-down-september/21652/5

A solution would be to finally switch to "real" servers but i dont know how far @sacdallago is in this process.

AlexanderLill commented 7 years ago

@jonas-he thanks for the explanation, I found a similiar thread yesterday: https://forums.docker.com/t/resolved-dockerapp-io-dns-down/18448 ^^ They said they switched to the IP which obviously solved the DNS issue, maybe this is an option for us, too (like I mentioned in the post before)

The last build which didnt work (see @sacdallago post with Error: Cannot find module 'twitter') didn't fail, too. Neither in Travis, nor in the dockercloud - that is why I am asking @sacdallago to check if the current container runs, or if it shows another error message in the log.

Furthermore we might really consider the IP address, assuming it doesnt change on re-deployment: docker inspect [container-id]

The API is still offline, it slowly gets really annoying ;) @PokemonGoers/catch-em-all

jonas-he commented 7 years ago

@AlexanderLill yes i know, but i cant find his exact error message in the build log, i think one needs to log in to docker cloud to see that, but i don't have credentials for that. I don't think you will find the IP address that easily since we don't have access to the machine where this docker container runs. I think we have to wait on @sacdallago to solve the mystery

sacdallago commented 7 years ago

Sorry for the delay. Nope, it's not 100% fixed, still won't start up:

[pokedata-1]2016-10-08T04:39:16.212157567Z npm info it worked if it ends with ok 
[pokedata-1]2016-10-08T04:39:16.212737464Z npm info using npm@2.14.2 
[pokedata-1]2016-10-08T04:39:16.213251771Z npm info using node@v4.0.0 
[pokedata-1]2016-10-08T04:39:16.477047873Z npm info prestart pokemongo-api@0.0.1 
[pokedata-1]2016-10-08T04:39:16.485457076Z npm info start pokemongo-api@0.0.1 
[pokedata-1]2016-10-08T04:39:16.488122410Z  
[pokedata-1]2016-10-08T04:39:16.488146792Z > pokemongo-api@0.0.1 start /usr/src/app 
[pokedata-1]2016-10-08T04:39:16.488161388Z > NODE_ENV=production node app.js 
[pokedata-1]2016-10-08T04:39:16.488173551Z  
[pokedata-1]2016-10-08T04:39:18.427964311Z /usr/src/app/node_modules/mongoose/node_modules/mongodb/lib/utils.js:98 
[pokedata-1]2016-10-08T04:39:18.428024069Z     process.nextTick(function() { throw err; }); 
[pokedata-1]2016-10-08T04:39:18.428036353Z                                   ^ 
[pokedata-1]2016-10-08T04:39:18.428046915Z  
[pokedata-1]2016-10-08T04:39:18.428056850Z Error: More than 1 database name in URL 
[pokedata-1]2016-10-08T04:39:18.428067468Z     at module.exports (/usr/src/app/node_modules/hashpokemongo/node_modules/mongodb/lib/url_parser.js:128:13) 
[pokedata-1]2016-10-08T04:39:18.428078139Z     at connect (/usr/src/app/node_modules/hashpokemongo/node_modules/mongodb/lib/mongo_client.js:290:16) 
[pokedata-1]2016-10-08T04:39:18.428088667Z     at Function.MongoClient.connect (/usr/src/app/node_modules/hashpokemongo/node_modules/mongodb/lib/mongo_client.js:114:3) 
[pokedata-1]2016-10-08T04:39:18.428133680Z     at Object.module.exports.start.MongoClient.connect.Rx.Observable.interval.map.counter.do.pokemon.flatMap.pokemon.flatMap.emptyArrayObservable.flatMap.tweets.filter.tweet.map [as start] (/usr/src/app/node_modules/hashpokemongo/sentimentTwitter/TwitterSentimentsMiner.js:25:17) 
[pokedata-1]2016-10-08T04:39:18.428147887Z     at Object.module.exports.insertToDb (/usr/src/app/app/controllers/filler/hashPokemonGo.js:33:40) 
[pokedata-1]2016-10-08T04:39:18.428158112Z     at NativeConnection.<anonymous> (/usr/src/app/startUp.js:20:30) 
[pokedata-1]2016-10-08T04:39:18.428168764Z     at emitNone (events.js:67:13) 
[pokedata-1]2016-10-08T04:39:18.428178434Z     at NativeConnection.emit (events.js:166:7) 
[pokedata-1]2016-10-08T04:39:18.428188060Z     at open (/usr/src/app/node_modules/mongoose/lib/connection.js:518:11) 
[pokedata-1]2016-10-08T04:39:18.428197880Z     at /usr/src/app/node_modules/mongoose/lib/connection.js:524:7 
[pokedata-1]2016-10-08T04:39:18.428207482Z     at /usr/src/app/node_modules/mongoose/node_modules/mongodb/lib/db.js:1550:5 
[pokedata-1]2016-10-08T04:39:18.428217180Z     at handleCallback (/usr/src/app/node_modules/mongoose/node_modules/mongodb/lib/utils.js:96:12) 
[pokedata-1]2016-10-08T04:39:18.428227104Z     at _callback (/usr/src/app/node_modules/mongoose/node_modules/mongodb/lib/db.js:1466:5) 
[pokedata-1]2016-10-08T04:39:18.428241156Z     at /usr/src/app/node_modules/mongoose/node_modules/mongodb/lib/db.js:1509:7 
[pokedata-1]2016-10-08T04:39:18.428251181Z     at /usr/src/app/node_modules/mongoose/node_modules/mongodb/node_modules/mongodb-core/lib/connection/pool.js:701:7 
[pokedata-1]2016-10-08T04:39:18.428261276Z     at /usr/src/app/node_modules/mongoose/node_modules/mongodb/node_modules/mongodb-core/lib/connection/pool.js:679:11 
[pokedata-1]2016-10-08T04:39:18.443994462Z  
[pokedata-1]2016-10-08T04:39:18.448315349Z npm info pokemongo-api@0.0.1 Failed to exec start script 
[pokedata-1]2016-10-08T04:39:18.448907301Z npm ERR! Linux 3.13.0-85-generic 
[pokedata-1]2016-10-08T04:39:18.449083524Z npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "start" 
[pokedata-1]2016-10-08T04:39:18.449644802Z npm ERR! node v4.0.0 
[pokedata-1]2016-10-08T04:39:18.450167223Z npm ERR! npm  v2.14.2 
[pokedata-1]2016-10-08T04:39:18.450756762Z npm ERR! code ELIFECYCLE 
[pokedata-1]2016-10-08T04:39:18.451180384Z npm ERR! pokemongo-api@0.0.1 start: `NODE_ENV=production node app.js` 
[pokedata-1]2016-10-08T04:39:18.451711147Z npm ERR! Exit status 1 
[pokedata-1]2016-10-08T04:39:18.452119399Z npm ERR!  
[pokedata-1]2016-10-08T04:39:18.452558405Z npm ERR! Failed at the pokemongo-api@0.0.1 start script 'NODE_ENV=production node app.js'. 
[pokedata-1]2016-10-08T04:39:18.452963631Z npm ERR! This is most likely a problem with the pokemongo-api package, 
[pokedata-1]2016-10-08T04:39:18.453460159Z npm ERR! not with npm itself. 
[pokedata-1]2016-10-08T04:39:18.454177071Z npm ERR! Tell the author that this fails on your system: 
[pokedata-1]2016-10-08T04:39:18.454200815Z npm ERR!     NODE_ENV=production node app.js 
[pokedata-1]2016-10-08T04:39:18.454445754Z npm ERR! You can get their info via: 
[pokedata-1]2016-10-08T04:39:18.454765900Z npm ERR!     npm owner ls pokemongo-api 
[pokedata-1]2016-10-08T04:39:18.455249444Z npm ERR! There is likely additional logging output above. 
[pokedata-1]2016-10-08T04:39:18.458829045Z  
[pokedata-1]2016-10-08T04:39:18.459151182Z npm ERR! Please include the following file with any support request: 
[pokedata-1]2016-10-08T04:39:18.462307563Z npm ERR!     /usr/src/app/npm-debug.log

I'm trying to redeploy refreshing the container's volumes, see if that fixes the issue.

sacdallago commented 7 years ago

Nope. Still won't fire up, error as above.

jonas-he commented 7 years ago

@sacdallago it seems like the DB URI is not created correctly. For me everything is working. Maybe we can find a confidential communication channel and exchange DB parametres. I invited you to private gitter room.

sacdallago commented 7 years ago

This is fixed and working now:

http://pokedata.c4e3f8c7.svc.dockerapp.io:65024/socket.io/socket.io.js