search

monasca / monasca-docker

Docker files and setup for Monasca
Apache License 2.0
24 stars 47 forks source link

monasca_persister can not acccess kafka #483

Open zhangjianweibj opened 5 years ago

zhangjianweibj commented 5 years ago

hello,i run monasca-docker in one node.api and other components woks well.only monasca_persister not working. logs as follow：

image

i wonder all components run in one docker node.api and other components can access kafka,only persister can not access kafka.why?very thanks.

matrixik commented 5 years ago

Sorry, never had such problem. Look like it can't event connect to Kafka. That's error from Python code so at the starting of container it should be able to connect properly to Kafka (connection is checked by start.sh script) and later something happen. Do you have any errors/warnings printed by Kafka? Did you change environment variable KAFKA_URI for persister? Is this 172.19.0.3 proper IP?

zhangjianweibj commented 5 years ago

ok.very thanks.172.19.0.3 is a proper ip.other components wroks well.

zhangjianweibj commented 5 years ago

i just run "docker stack deploy -c docker-compose.yml",but monasca forward and other components can not accss each other. image

i want to run monasca not in one node.so i want use docker swarm.but find "docker stack deploy -c docker-compose.yml" encountered many errors.can your give me some advises use docker-compose deploy monaca components on different nodes.very thanks

matrixik commented 5 years ago

As I wrote in other bug I never used Docker Swarm so can't help you with this problems...

zhangjianweibj commented 5 years ago

very thank.did your ever use monasca-docker tool, deploy the monasca components on different nodes?

matrixik commented 5 years ago

Nope, my part is to only make it working on one node. Sorry.

matrixik commented 5 years ago

I read a little about docker swarm and looks like it's capable of running from docker-compose.yml file. That's nice. Did you make many changes to default docker compose files? Could you share them? Did you try running docker swarm with default docker-compose file (as is from this repository) to check if default configuration is working fine or also have this problems?

zhangjianweibj commented 5 years ago 
version: '3'
services:

  memcached:
    image: memcached:1.5.0-alpine
    environment:
      LOGSTASH_FIELDS: "service=memcached"

  influxdb:
    image: influxdb:1.3.3-alpine
    environment:
      LOGSTASH_FIELDS: "service=influxdb"
  influxdb-init:
    image: monasca/influxdb-init:1.0.1
    environment:
      LOGSTASH_FIELDS: "service=influxdb-init"
    depends_on:
      - influxdb

  cadvisor:
    image: google/cadvisor:v0.27.1
    environment:
      LOGSTASH_FIELDS: "service=cadvisor"
    volumes:
      - "/:/rootfs:ro"
      - "/var/run:/var/run:rw"
      - "/sys:/sys:ro"
      - "/var/lib/docker:/var/lib/docker:ro"

  agent-forwarder:
    image: monasca/agent-forwarder:master-20180824-114532
    environment:
      NON_LOCAL_TRAFFIC: "true"
      LOGSTASH_FIELDS: "service=monasca-agent-forwarder"

  agent-collector:
    image: monasca/agent-collector:master-20180914-130916
    restart: on-failure
    environment:
      AGENT_HOSTNAME: "docker-host"
      FORWARDER_URL: "http://agent-forwarder:17123"
      CADVISOR: "true"
      CADVISOR_URL: "http://cadvisor:8080/"
      LOGSTASH_FIELDS: "service=monasca-agent-collector"
      MONASCA_MONITORING: "true"
      MONASCA_LOG_MONITORING: "false"
    cap_add:
      - FOWNER
    volumes:
      - "/:/rootfs:ro"

  alarms:
    image: monasca/alarms:1.2.0
    environment:
      LOGSTASH_FIELDS: "service=monasca-alarms"
    depends_on:
      - keystone
      - monasca

  zookeeper:
    image: zookeeper:3.4
    environment:
      LOGSTASH_FIELDS: "service=zookeeper"
    restart: on-failure

  kafka:
    image: monasca/kafka:0.9.0.1-2.11-1.1.6
    environment:
      KAFKA_DELETE_TOPIC_ENABLE: "true"
      LOGSTASH_FIELDS: "service=kafka"
    restart: on-failure
    depends_on:
      - zookeeper
  kafka-watcher:
    image: monasca/kafka-watcher:0.0.3
    environment:
      BOOT_STRAP_SERVERS: "kafka"
      PROMETHEUS_ENDPOINT: "0.0.0.0:8080"
      LOGSTASH_FIELDS: "service=kafka-watcher"
    depends_on:
      - kafka
    ports:
      - "18080:8080"
  kafka-init:
    image: monasca/kafka-init:0.0.3
    environment:
      ZOOKEEPER_CONNECTION_STRING: "zookeeper:2181"
      KAFKA_TOPIC_CONFIG: segment.ms=900000 # 15m
      KAFKA_CREATE_TOPICS: "\
        metrics:64:1,\
        alarm-state-transitions:12:1,\
        alarm-notifications:12:1,\
        retry-notifications:3:1,\
        events:12:1,\
        kafka-health-check:1:1,\
        60-seconds-notifications:3:1"
      LOGSTASH_FIELDS: "service=kafka-init"
    depends_on:
      - zookeeper

  mysql:
    image: mysql:5.7
    environment:
      MYSQL_ROOT_PASSWORD: secretmysql
      LOGSTASH_FIELDS: "service=mysql"
  mysql-init:
    image: monasca/mysql-init:1.5.4
    environment:
      MYSQL_INIT_DISABLE_REMOTE_ROOT: "false"
      MYSQL_INIT_RANDOM_PASSWORD: "false"
      LOGSTASH_FIELDS: "service=mysql-init"

  keystone:
    image: monasca/keystone:1.1.3
    environment:
      KEYSTONE_HOST: keystone
      KEYSTONE_PASSWORD: secretadmin
      KEYSTONE_DATABASE_BACKEND: mysql
      KEYSTONE_MYSQL_HOST: mysql
      KEYSTONE_MYSQL_USER: keystone
      KEYSTONE_MYSQL_PASSWORD: keystone
      KEYSTONE_MYSQL_DATABASE: keystone
      LOGSTASH_FIELDS: "service=keystone"
    depends_on:
      - mysql
    ports:
      - "5000:5000"
      - "35357:35357"

  monasca-sidecar:
    image: timothyb89/monasca-sidecar:1.0.0
    environment:
      LOGSTASH_FIELDS: "service=monasca-sidecar"

  monasca:
    image: monasca/api:master-20180105-072513
    environment:
      SIDECAR_URL: http://monasca-sidecar:4888/v1/ingest
      LOGSTASH_FIELDS: "service=monasca-api"
    depends_on:
      - influxdb
      - keystone
      - mysql
      - zookeeper
      - kafka
      - monasca-sidecar
      - memcached
    ports:
      - "8070:8070"

  monasca-persister:
    image: monasca/persister:1.6
    environment:
      LOGSTASH_FIELDS: "service=monasca-persister"
    restart: on-failure
    depends_on:
      - monasca
      - influxdb
      - zookeeper
      - kafka

  storm-nimbus:
    image: monasca/storm:1.0.3
    command: storm nimbus
    environment:
      STORM_LOCAL_HOSTNAME: "storm-nimbus"
      WORKER_LOGS_TO_STDOUT: "true"
    depends_on:
      - zookeeper
      - kafka

  storm-supervisor:
    image: monasca/storm:1.0.3
    command: storm supervisor
    depends_on:
      - storm-nimbus
      - zookeeper
      - kafka

  thresh-init:
    image: monasca/thresh:master
    environment:
      STORM_WAIT_RETRIES: 50
    depends_on:
      - zookeeper
      - storm-nimbus
      - storm-supervisor

  monasca-notification:
    image: monasca/notification:master-20171024-085537
    environment:
      NF_PLUGINS: "webhook"
      LOGSTASH_FIELDS: "service=monasca-notification"
      STATSD_HOST: monasca-statsd
      STATSD_PORT: 8125
    depends_on:
      - monasca
      - zookeeper
      - kafka
      - mysql

  grafana:
    image: monasca/grafana:4.0.0-1.5.1
    environment:
      GF_AUTH_BASIC_ENABLED: "false"
      GF_USERS_ALLOW_SIGN_UP: "true"
      GF_USERS_ALLOW_ORG_CREATE: "true"
      GF_AUTH_KEYSTONE_ENABLED: "true"
      GF_AUTH_KEYSTONE_AUTH_URL: http://keystone:5000
      GF_AUTH_KEYSTONE_VERIFY_SSL_CERT: "false"
      GF_AUTH_KEYSTONE_DEFAULT_DOMAIN: "Default"
      LOGSTASH_FIELDS: "service=grafana"
    ports:
      - "3000:3000"
    depends_on:
      - keystone
      - monasca

  grafana-init:
    image: monasca/grafana-init:1.4.0
    environment:
      LOGSTASH_FIELDS: "service=grafana-init"
    depends_on:
      - grafana

  monasca-statsd:
    image: monasca/statsd:master-20180615-123838
    environment:
      FORWARDER_URL: http://agent-forwarder:17123
      LOG_LEVEL: WARN
    ports:
      - "8125/udp"
zhangjianweibj commented 5 years ago

thanks.i replace docker-compse.yml ${variable} with env value.

zhangjianweibj commented 5 years ago

when monasca-docker_default network created,or which component create this network? image

matrixik commented 5 years ago

I believe it's default network created by docker-compose. It's name is based on your folder name: monasca-docker_default You are running it from monasca-docker folder.

zhangjianweibj commented 5 years ago

yes,network created by docker default.so i create a network type overlay.then run docker-compose.yml but components can not access kafka.

image

image

zhangjianweibj commented 5 years ago

docker-compose.yml file,network use mynetwork image

zhangjianweibj commented 5 years ago

and all components in mynetwork image

zhangjianweibj commented 5 years ago

and agent-forward can not access monasca as well

image

shail8989 commented 3 years ago

i am finding the same error where monasca-persistor is keep falling in docker containers . i am running wallaby all in one .

container-persistor

and when i started to checking the logs it

ERROR monasca_persister.repositories.persister r = adapter.send(request, **kwargs) ERROR monasca_persister.repositories.persister File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 498, in send ERROR monasca_persister.repositories.persister raise ConnectionError(err, request=request) ERROR monasca_persister.repositories.persister requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) ERROR monasca_persister.repositories.persister INFO monasca_persister.persister [-] Received signal 17, beginning graceful shutdown.

my monasca container is not stable at all, any one faced the same issue