google/cadvisor:0.10.0 over marathon

[samek]

Hi, I wanted to start the 0.10.0 from marathon but it keeps restarting.

Looking at the messos logs:

I0225 23:25:19.067898 414 exec.cpp:132] Version: 0.21.1 I0225 23:25:19.070999 438 exec.cpp:206] Executor registered on slave 20150225-221350-3238002698-5050-20322-S2 E0225 22:25:19.087483 00001 manager.go:180] Failed to start OOM watcher, will not get OOM events: neither /var/log/messages nor /var/log/syslog exists from which to read kernel errors F0225 22:25:19.087891 00001 cadvisor.go:153] listen tcp :8080: bind: address already in use goroutine 1 [running]: github.com/golang/glog.stacks(0x4c2080a5500, 0x0, 0x0, 0x0) /usr/local/google/home/vmarmol/go/src/github.com/google/cadvisor/Godeps/_workspace/src/github.com/golang/glog/glog.go:726 +0xcd github.com/golang/glog.(_loggingT).output(0xf1a4c0, 0x4c200000003, 0x4c20800c900) /usr/local/google/home/vmarmol/go/src/github.com/google/cadvisor/Godeps/_workspace/src/github.com/golang/glog/glog.go:677 +0x24c github.com/golang/glog.(_loggingT).print(0xf1a4c0, 0x4c200000003, 0x4c208217f38, 0x1, 0x1) /usr/local/google/home/vmarmol/go/src/github.com/google/cadvisor/Godeps/_workspace/src/github.com/golang/glog/glog.go:626 +0x17c github.com/golang/glog.Fatal(0x4c208217f38, 0x1, 0x1) /usr/local/google/home/vmarmol/go/src/github.com/google/cadvisor/Godeps/_workspace/src/github.com/golang/glog/glog.go:1019 +0x50 main.main()

/usr/local/google/home/vmarmol/go/src/github.com/google/cadvisor/cadvisor.go:153 +0x17c6

So it want's to bind on 8080 which is already taken but, I've defined it to use marathon delegated host port..

my marathon json:

{ "container": { "type": "DOCKER", "docker": { "image": "google/cadvisor:0.10.0" }, "volumes": [ { "containerPath": "/rootfs", "hostPath": "/", "mode": "RO" }, { "containerPath": "/var/run", "hostPath": "/var/run", "mode": "RW" }, { "containerPath": "/sys", "hostPath": "/sys", "mode": "RO" }, { "containerPath": "/var/lib/docker", "hostPath": "/var/lib/docker", "mode": "RO" }, { "containerPath": "/cgroup", "hostPath": "/cgroup", "mode": "RO" } ], "network": "BRIDGE", "portMappings": [ { "containerPort": 8080, "hostPort": 0, "servicePort": 8888, "protocol": "tcp" } ] }, "id": "cadvisor-beta", "instances": 1, "cpus": 0.5, "mem": 512, "constraints": [ [ "hostname", "UNIQUE" ] ]

}

I have lot's of docker images for which i use portmappings this way (like multiple nginx on same server)..

am I doing something wrong ?

P.S. I could not use :latest since I've got a different error:

I0225 22:44:04.620594 25157 exec.cpp:132] Version: 0.21.1

F0225 21:44:04.595594 00001 cadvisor.go:76] Failed to create a Container Manager: open /sys/class/net/bonding_masters/address: not a directory

I'm using marathon Version 0.8.0 and mesos 0.21.1 with Docker version 1.5.0 on Ubuntu 12.04.4 LTS

[vmarmol]

I'm not super familiar with running things on Marathon so bear with me here :)

To specify a different port for cAdvisor you need to tell the cAdvisor to listen on a different port through a --port flag. I vaguely remember that Mesos ran Docker containers in --net=host which means that it can't do the port mapping (although you mentioned that it did it for other servers so maybe I'm confused).

Can you provide the other port as a flag to cAdvisor?

[samek]

Something is not right and I don't know what is it :( Ok with the --port flag it starts and it runs but i don't have any access to it. (no port mapping)

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 46d3ed240800 google/cadvisor:0.10.0 "/usr/bin/cadvisor" 37 seconds ago Up 36 seconds mesos-f6565aef-cd98-4839-b2e0-9cf98e08c125

current json

{ "container": { "type": "DOCKER", "docker": { "image": "google/cadvisor:0.10.0" }, "volumes": [ { "containerPath": "/rootfs", "hostPath": "/", "mode": "RO" }, { "containerPath": "/var/run", "hostPath": "/var/run", "mode": "RW" }, { "containerPath": "/sys", "hostPath": "/sys", "mode": "RO" }, { "containerPath": "/var/lib/docker", "hostPath": "/var/lib/docker", "mode": "RO" }, { "containerPath": "/cgroup", "hostPath": "/cgroup", "mode": "RO" } ], "network": "BRIDGE", "portMappings": [ { "containerPort": 8833, "hostPort": 0, "protocol": "tcp" } ] }, "id": "cadvisor-beta3", "instances": 1, "arg": ["--port=8833"], "cpus": 0.5, "mem": 512, "constraints": [ [ "hostname", "UNIQUE" ] ]

}

bellow is an simple nginx json for marathon. (if you define hostport 0 then marathon will pick one), and will work on hosts with 80 already binded.

---

{ "container": { "type": "DOCKER", "docker": { "image": "nginx", "network": "BRIDGE", "portMappings": [ { "containerPort": 80, "hostPort": 0, "servicePort": 80, "protocol": "tcp" } ] } }, "id": "nginx", "instances": 1, "cpus": 0.25, "mem": 256, "uris": []

}

and the output: CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f51d2957b812 nginx:latest "nginx -g 'daemon of 6 seconds ago Up 5 seconds 443/tcp, 0.0.0.0:31690->80/tcp mesos-b87176d4-4b07-4e0a-bf83-e65c4f05630f
d37272ab373b nginx:latest "nginx -g 'daemon of 12 seconds ago Up 11 seconds 443/tcp, 0.0.0.0:31000->80/tcp mesos-10cbfabd-1a43-4f91-a387-aa8eb8bf023c

Well it's 1am - If i manage do set it up tomorrow I'll let you know the solution.

[samek]

Works like a charm.

my json was wrong, network and portmappings was outside of docker .....

---

{ "container": { "type": "DOCKER", "docker": { "image": "google/cadvisor:0.10.0", "network": "BRIDGE", "portMappings": [ { "containerPort": 8080, "hostPort": 0,

      "protocol": "tcp"
    }
  ]
},
"volumes": [
  {
    "containerPath": "/rootfs",
    "hostPath": "/",
    "mode": "RO"
  },
  {
    "containerPath": "/var/run",
    "hostPath": "/var/run",
    "mode": "RW"
  },
  {
    "containerPath": "/sys",
    "hostPath": "/sys",
    "mode": "RO"
  },
  {
    "containerPath": "/var/lib/docker",
    "hostPath": "/var/lib/docker",
    "mode": "RO"
  },
  {
    "containerPath": "/cgroup",
    "hostPath": "/cgroup",
    "mode": "RO"
  }
]

}, "id": "cadvisor-beta", "instances": 1,

"cpus": 0.5, "mem": 512, "constraints": [ [ "hostname", "UNIQUE" ] ] }

[vmarmol]

Awesome! Thanks for getting back to us btw.

[fenglvming]

I wanted to start the lastest version of cAdvisor from marathon but it keeps restarting.

{ "type": "DOCKER", "volumes": [ { "containerPath": "/rootfs", "hostPath": "/", "mode": "RO" }, { "containerPath": "/var/run", "hostPath": "/var/run", "mode": "RW" }, { "containerPath": "/sys", "hostPath": "/sys", "mode": "RO" }, { "containerPath": "/var/lib/docker", "hostPath": "/var/lib/docker/", "mode": "RO" }, { "containerPath": "/cgroup", "hostPath": "/cgroup", "mode": "RO" } ], "docker": { "image": "google/cadvisor", "network": "BRIDGE", "portMappings": [ { "containerPort": 8080, "hostPort": 0, "servicePort": 10000, "protocol": "tcp" } ], "privileged": true, "parameters": [], "forcePullImage": false } }

[samek]

Which version of marathon?

Sent from my iPhone

On 24 Nov 2015, at 08:21, fenglvming notifications@github.com wrote:

I wanted to start the lastest version of cAdvisor from marathon but it keeps restarting.

{ "type": "DOCKER", "volumes": [ { "containerPath": "/rootfs", "hostPath": "/", "mode": "RO" }, { "containerPath": "/var/run", "hostPath": "/var/run", "mode": "RW" }, { "containerPath": "/sys", "hostPath": "/sys", "mode": "RO" }, { "containerPath": "/var/lib/docker", "hostPath": "/var/lib/docker/", "mode": "RO" }, { "containerPath": "/cgroup", "hostPath": "/cgroup", "mode": "RO" } ], "docker": { "image": "google/cadvisor", "network": "BRIDGE", "portMappings": [ { "containerPort": 8080, "hostPort": 0, "servicePort": 10000, "protocol": "tcp" } ], "privileged": true, "parameters": [], "forcePullImage": false } }

— Reply to this email directly or view it on GitHub.

[fenglvming]

marathon : 0.11.1 The output of mesos's stdout file is : No such file or directory

[samek]

Add /usr/bin/cadvisor to the cmd field and try again.

Sent from my iPhone

On 24 Nov 2015, at 08:26, fenglvming notifications@github.com wrote:

marathon : 0.11.1

— Reply to this email directly or view it on GitHub.

[fenglvming]

still : No such file or directory

[fenglvming]

here is my configure page:

[samek]

try like that. I just give it a go and it works. * I did not expose any ports ..

Anyway I told you the wrong cmd .. use /usr/bin/cadvisor -logtostderr

On Nov 24, 2015, at 8:35 AM, fenglvming notifications@github.com wrote:

here is my configure page: https://cloud.githubusercontent.com/assets/2157195/11360592/f0310378-92c0-11e5-8aae-7fb591a5ed66.png https://cloud.githubusercontent.com/assets/2157195/11360594/f1dcaa2e-92c0-11e5-8a95-12b6971eeb3b.png — Reply to this email directly or view it on GitHub https://github.com/google/cadvisor/issues/537#issuecomment-159183489.

[samek]

Does it work ?

On Nov 24, 2015, at 8:44 AM, samo gabrovec samek@me.com wrote:

try like that. I just give it a go and it works. * I did not expose any ports ..

Anyway I told you the wrong cmd .. use /usr/bin/cadvisor -logtostderr <Screen Shot 2015-11-24 at 8.42.11 AM.png>

On Nov 24, 2015, at 8:35 AM, fenglvming <notifications@github.com mailto:notifications@github.com> wrote:

here is my configure page: https://cloud.githubusercontent.com/assets/2157195/11360592/f0310378-92c0-11e5-8aae-7fb591a5ed66.png https://cloud.githubusercontent.com/assets/2157195/11360594/f1dcaa2e-92c0-11e5-8a95-12b6971eeb3b.png — Reply to this email directly or view it on GitHub https://github.com/google/cadvisor/issues/537#issuecomment-159183489.

[fenglvming]

it didn't work. https://github.com/google/cadvisor/issues/537#issuecomment-159191134

[samek]

What does the stderr say in mesos ui completed tasks -> cadvisor Sandbox ?

On Nov 24, 2015, at 9:27 AM, fenglvming notifications@github.com wrote:

it didn't work.

537 (comment) https://github.com/google/cadvisor/issues/537#issuecomment-159191134

— Reply to this email directly or view it on GitHub https://github.com/google/cadvisor/issues/537#issuecomment-159193468.

[fenglvming]

I think it's my mesos-slave configuration problem. I just use --containerizers=docker,mesos ,It works. Before that I use --containerizers=mesos,docker . It's werid.