search

usc-isi-i2 / dig-etl-engine

Download DIG to run on your laptop or server.
http://usc-isi-i2.github.io/dig/
MIT License
101 stars 39 forks source link

k.m.a.KafkaManagerActor - Updating internal state #149

Open calvinleungyk opened 6 years ago

calvinleungyk commented 6 years ago

I am attempting to run MyDIG on Ubuntu 16.04 on AWS. After running docker-compose up, the terminal keeps printing:

kafka_manager_1 | [info] k.m.a.KafkaManagerActor - Updating internal state...

I was also unable to add new project, which I'm not sure if it's related to the same issue. Have you encountered this problem?

GreatYYX commented 6 years ago

It's not a problem, it's just Kafka manager's heart beat log.

Have you seen any error logs in terminal, could you please run docker logs dig_mydig_ws_1 and./diagnose.sh?

calvinleungyk commented 6 years ago

When I run docker-compose up on local (Mac OSX), the repeated message scenario did not happen.

When I run docker-compose up on Ubuntu 16.04, I couldn't create a new project, plus kafka_manager keeps printing "updating internal state".

This happens when I try to add a new project: kafka_manager_1 | [info] k.m.a.KafkaManagerActor - Updating internal state... kafka_manager_1 | [info] k.m.a.KafkaManagerActor - Updating internal state... mydig_ws_1 | 172.19.0.14 - - [21/Dec/2017 10:34:51] "GET / HTTP/1.0" 200 - mydig_ws_1 | 172.19.0.14 - - [21/Dec/2017 10:34:53] "GET /constants HTTP/1.0" 200 - kafka_manager_1 | [info] k.m.a.KafkaManagerActor - Updating internal state... kafka_manager_1 | [info] k.m.a.KafkaManagerActor - Updating internal state...

Attached logs from docker logs: /app/mydig-webservice/docker_run_mydig.sh: ./daemon.sh: /usr/bin/env: bad interpreter: Text file busy /app/mydig-webservice/docker_run_mydig.sh: ./backend.sh: /usr/bin/env: bad interpreter: Text file busy /app/mydig-webservice/docker_run_mydig.sh: ./frontend.sh: /usr/bin/env: bad interpreter: Text file busy serve: Running on port 9881 killing daemon process (if exists) starting daemon (ETK spaCy) done killing backend process (if exists) starting backend done killing frontend process (if exists) starting frontend done

GreatYYX commented 6 years ago

I haven't met this problem before, there's a possible solution from internet: https://www.cyberciti.biz/faq/binbash-bad-interpreter-text-file-busy/

Please try to kill other processes which are using this script and run docker-compose restart mydig_ws

calvinleungyk commented 6 years ago

You mean killing processes that use /app/mydig-webservice/docker_run_mydig.sh and then restart mydig_ws? There is only one process using it, I killed it and the same thing happens.

Does anyone who uses MyDIG host it on AWS Ubuntu 16.04?

calvinleungyk commented 6 years ago

Update: the GET HTTP 2.0 does not happen when I attempt to add a new project, but when I access the web interface. But I still couldn't add a new project. Spinning up another clean AWS Ubuntu 16.04 instance to see if the problem persists.

GreatYYX commented 6 years ago

Found a possible solution https://github.com/moby/moby/issues/9547, it could be a linux cache problem.

@calvinleungyk please help me test it by:

  1. docker exec -it dig_mydig_ws_1 bash
  2. cd /app/mydig-webservice
  3. vi docker_run_mydig.sh (source code is here: https://github.com/usc-isi-i2/mydig-webservice/blob/master/docker_run_mydig.sh) and add sync between chmod and ./xxx.sh
  4. ./docker_run_mydig.sh
calvinleungyk commented 6 years ago

It solves the problem on docker logs, yielding this:

killing daemon process (if exists) starting daemon (ETK spaCy) done killing backend process (if exists) starting backend done killing frontend process (if exists) starting frontend done

But kafka_manager still outputs Updating internal state.

Seems like the problem with not being able to add a project is not due to this.

GreatYYX commented 6 years ago

Thanks. I'll add sync into new release to avoid this cache problem.

Again, "kafka_manager still outputs Updating internal state" is not an error message, it's just the heart beat log.

From what you pasted above, it seems myDIG backend is not running, that's why I asked you to run diagnose.sh. Now, let's find out why backend is not running.

  1. docker exec -it dig_mydig_ws_1 bash
  2. cd /app/mydig-webservice/ws
  3. tail -n 200 nohup.out