cadvisor import stats not working with stdout and elasticsearch

[raji90]

Hi I am running cadvisor container in my Ubuntu machine as following. I can see the live streaming in the browser but when it comes to importing stats to a location (elasticsearch) then it throws me error.

docker run \ --volume=/:/rootfs:ro --volume=/var/run:/var/run:rw --volume=/sys:/sys:ro --volume=/var/lib/docker/:/var/lib/docker:ro --publish=9091:8080 --detach=true --name=cadvisor1 docker.fmr.com/google/cadvisor:latest -storage_driver=elasticsearch -storage_driver_es_host="http://my_elk.fcorp.com:9200"

Error: F0912 07:50:13.460296 1 cadvisor.go:117] Failed to initialize storage driver: failed to create the elasticsearch client - no Elasticsearch node available

My elsticsearch is present in instance "http://my_elk.fcorp.com:9200". Its a Ubuntu machine with elasticsearch version 5.4.0. Does it not work for version 5 or above? And how can I use 'stdout' option (https://github.com/google/cadvisor/tree/master/docs/storage) ?

[raji90]

@vishh @tallclair Can you please help here?

[tallclair]

I'm not familiar with elastic search. Perhaps someone else in the community can chime in.

[dashpole]

Looks like you found the related issue already: #1517

[raji90]

Thanks a lot @tallclair and @dashpole for the response. so with #1517 issues it looks like cadvisor still does not support version 5. I thought the post was a year back post and cadvisor might have been updated to use latest version of elasticsearch.

I tried with influxdb also and that also threw me error. I have an issue opened for it. https://github.com/google/cadvisor/issues/1757

[raji90]

@tallclair I saw couple comments from you for influxdb. CAn you or someone from team help me understanding/fixing the issue? https://github.com/google/cadvisor/issues/1757

[raji90]

I just need a mechanism through which I can store the stats. I was initially trying stdout option but did not work. I don't see any doc for that option. Then I tried with elasticsearch and now with influxdb and these are also not working.

[HarenBroog]

@raji90 I have managed to collect metrics from cAdvisor. I used storage driver stdout alongside with logspout. I think elasticsearch driver is broken with ES v5.

Snippet from my docker-stack.yml:

version: '3'
services:
  cadvisor:
    image: google/cadvisor:v0.26.1
    deploy:
      replicas: 1
    volumes:
      - "/../:/rootfs:ro"
      - "/var/run:/var/run:rw"
      - "/sys:/sys:ro"
      - "/var/lib/docker/:/var/lib/docker:ro"
      - "/dev/disk/:/dev/disk:ro"
    command: -storage_driver="stdout" -storage_driver_es_host="http://elk:9200"
    ports:
      - 8080:8080
  logspout:
    image: local/logspout
    volumes:
       - /var/run/docker.sock:/var/run/docker.sock
    environment:
      ROUTE_URIS: "logstash://elk:5000?filter.name=my_actual_stack_to_monitor*,logstash://elk:5000?filter.name=elk_stack_cadvisor*"
      DOCKER_LABELS: "true"
  elk:
    image: sebp/elk:551
    volumes:
      - /home/user/apps/elk_stack/data:/var/lib/elasticsearch
      - /home/user/apps/elk_stack/deploy/logstash/01-json.conf:/etc/logstash/conf.d/01-json.conf
      - /home/user/apps/elk_stack/deploy/logstash/30-output.conf:/etc/logstash/conf.d/30-output.conf
    ports:
     - 5601:5601

Moreover, you probably want to differentiate index name for metrics (CPU, memory etc) and your app logs. I used dynamic logstash index for that, with docker stack name as a variable.

I hope that will solve your issues :)

[raji90]

@HarenBroog Thanks a lot for the comment. I will try similar thing and will check