How to add Envs to influxdb

[orientzc]

Hi, I am using Docker with Mesos / Marathon and currently the metrics sent to influxdb do not contain the marathon_app_id. I think I need to whitelist the Env-vars I want to be send to influxdb with --docker_env_metadata_whitelist. Like this, docker run --volume=/:/rootfs:ro --volume=/var/run:/var/run:rw --volume=/sys:/sys:ro --volume=/var/lib/docker/:/var/lib/docker:ro --publish=8092:8080 --detach=true --link influxsrv:influxsrv --name=cadvisor google/cadvisor --docker_env_metadata_whitelist=MARATHON_APP_ID,MARATHON_APP_DOCKER_IMAGE -storage_driver=influxdb -storage_driver_db=cadvisorenv -storage_driver_host=influxsrv:8086 but it doesn't work for me. I use SHOW TAG KEYS FROM "cpu_usage_user". Like this, And I use docker logs container_id, Like this, Can anybody help me?

[orientzc]

use cadvisor API, like this: api/v1.2/docker/, i can see the env by cadvisor, but it may not write to influxdb. what do i miss? ps: i try google/cadvisor:latest and goole/cadvisor:canary.

[bergerx]

I think I also did hit the same problem, it seems like the docker_env_metadata_whitelist parameter is not used by influxdb storage plugin.

Appending this to cadvisor cmd:

-docker_env_metadata_whitelist=MESOS_TASK_ID,MARATHON_APP_ID,MARATHON_APP_VERSION

Having the docker_env_metadata_whitelist parameter with marathon labels in place, I can see the labels in /metrics endpoint of cadvisor, so the docker_env_metadata_whitelist parameter is working:

container_cpu_system_seconds_total{container_env_marathon_app_id="/cadvisor",container_env_marathon_app_version="2016-12-30T13:04:59.869Z",container_env_mesos_task_id="cadvisor.91dc395f-ce90-11e6-b28b-70b3d5800003",id="/docker/d4a910001c30574b6bc1161ba0260f76bfcaf290206faf502215761b95d00e17",image="google/cadvisor:v0.24.1",name="mesos-ee28b263-ad31-451d-aeb7-9b7856eddd44-S95.fbb98d0b-79de-4287-a5a9-6e126b00ee70"} 1.16

But can't see these tags in InfluxDB database, assuming that docker_env_metadata_whitelist parameter is not used in influxdb storage:

bdogan@Bekirs-MacBook-Pro:~$ influx -host influx-db.marathon.l4lb.thisdcos.directory -port 8086
Visit https://enterprise.influxdata.com to register for updates, InfluxDB server management, and monitoring.
Connected to http://influx-db.marathon.l4lb.thisdcos.directory:8086 version 0.13.0
InfluxDB shell version: v1.1.1
> use cadvisor
Using database cadvisor
> SHOW TAG KEYS FROM "cpu_usage_total"
name: cpu_usage_total
tagKey
------
build-date
container_name
license
machine
name
vendor

> 

[bergerx]

Oh, there is already a merge request for this: #1427

[orientzc]

thanks very much. i think we did hit the same problem, and i also see the issue:#1427. so i try google/cadvisor:latest and google/cadvisor:canary. because docker_env_metadata_whitelist parameter is not used by influxdb storage plugin in the code of google/cadvisor:latest, and it is used in the code of google/cadvisor:canary.(#1427: https://github.com/erSitzt/cadvisor). see this, but it is not working. but i donnt known the reason, or what do i miss?

[erSitzt]

Hmm i just tested it with my cadvisor version.. npm_config_loglevel is an env var only present in our node containers.

And this is my option for cAdvisor: -docker_env_metadata_whitelist HOST,NPM_CONFIG_LOGLEVEL

[erSitzt]

I use a single dash for docker_env_metadata_whitelist btw

[erSitzt]

And here cAdvisor /metrics endpoint container_cpu_system_seconds_total{container_env_host="Mesos-Slave2.neo-test.domain.com",container_env_npm_config_loglevel="info",container_label_collectd_docker_app="neo/artikel/demo",id="/system.slice/docker-75a83ec849cf3512e31d5676d1de4d7ac00afaa11209223698c22825c8e58807.scope",image="srv-nexus-docker-registry.domain.com/neo/artikel-neo:demo-3262",name="mesos-b7626133-ecc3-415e-9d38-cdeabaed1c7c-S3.45de2ba5-8a64-4033-85a9-9b325e889256"} 72.35

[erSitzt]

I just saw my merge request was still open ?... so this issue could be closed i think

[erSitzt]

If you want you can build your own cAdvisor docker image like this:

Dockerfile for binary

FROM golang:latest

RUN apt-get install -y git dmsetup
RUN go get github.com/tools/godep
RUN git clone https://github.com/erSitzt/cadvisor.git /go/src/github.com/google/cadvisor
RUN cd /go/src/github.com/google/cadvisor && git checkout release-v0.23 && make

#upload binary to some location you can use
RUN curl -v -k --user 'raw:raw' --upload-file /go/src/github.com/google/cadvisor/cadvisor https://somefilehosting-url.domain.com/cadvisor

Dockerfile for container

FROM alpine:3.4
MAINTAINER dengnan@google.com vmarmol@google.com vishnuk@google.com jimmidyson@gmail.com

ENV GLIBC_VERSION "2.23-r1"

RUN apk --no-cache add ca-certificates wget curl device-mapper && \
    apk --no-cache add zfs --repository http://dl-3.alpinelinux.org/alpine/edge/main/ && \
    wget -q -O /etc/apk/keys/sgerrand.rsa.pub https://raw.githubusercontent.com/sgerrand/alpine-pkg-glibc/master/sgerrand.rsa.pub && \
    wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/${GLIBC_VERSION}/glibc-${GLIBC_VERSION}.apk && \
    wget https://github.com/andyshinn/alpine-pkg-glibc/releases/download/${GLIBC_VERSION}/glibc-bin-${GLIBC_VERSION}.apk && \
    apk add glibc-${GLIBC_VERSION}.apk glibc-bin-${GLIBC_VERSION}.apk && \
    /usr/glibc-compat/sbin/ldconfig /lib /usr/glibc-compat/lib && \
    echo 'hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4' >> /etc/nsswitch.conf && \
    rm -rf /var/cache/apk/*

# Grab cadvisor from the staging directory.
RUN curl -k -o /usr/bin/cadvisor  https://somefilehosting-url.domain.com/cadvisor

EXPOSE 8080
ENTRYPOINT ["/usr/bin/cadvisor", "-logtostderr"]

This is using my Github Repository, so use at your own risk.

[orientzc]

Thanks very very much. i am sorry to be late. and using your Github Repository, will i only see envs in cpu_usage_per_cpu? is this because that you only edit the code of cpu_usage_per_cpu? and i will focus on the issue: #1427. and i expect good news. If you send a message to me when the merge request handled, i will be lucky. good luck to you and me. Thanks very very much.

[erSitzt]

@orientzc Nope, not only for cpu_usage_per_cpu, for all metrics.