System-docker version number is incompatible with cAdvisor

[netsandbox]

BurmillaOS Version: (ros os version) v1.9.6

Where are you running BurmillaOS? (docker-machine, AWS, GCE, baremetal, etc.) Hyper-V VM

Which processor architecture you are using? x86_64

System-docker version is currently 17.06-ros6.

$ sudo system-docker info --format "{{.ServerVersion}}"     
17.06-ros6

It looks like the system-docker version was never updated since the fork from RancherOS.

Also I recently discovered a problem when I was trying to use cAdvisor to monitor the system-docker container (because of #129). cAdvisor can't detect the running container, the error message is: I0120 17:43:43.835320 1 factory.go:202] Registration of the docker container factory failed: failed to validate Docker info: version string "17.06-ros6" doesn't match expected regular expression: "(\d+)\.(\d+)\.(\d+)"

[PrplHaz4]

I think this is potentially a dupe of https://github.com/burmilla/os/issues/28 with version scheme added on - maybe that warrants its own issue?

[olljanat]

Yea system-docker is heavily modified by Rancher. That why they didn't ever update it and we cannot do it easily.

However it should not be too hard to modify version number in way that cAdvisor can run as system container.

In additionally there is certain improvements in newer Docker versions which I would like to cherry-pick. Here is some draft but it didn't worked right yet https://github.com/burmilla/os-system-docker/releases/tag/17.06-ros6-burmilla1

[olljanat]

However it should not be too hard to modify version number in way that cAdvisor can run as system container.

It was actually harder than I expected. Not because of version change but because old build tooling didn't worked anymore and because I tried also update Go version which looks causing new issues.

However just released v2.0.0-beta6 contains system-docker with updated version number. @netsandbox please test it with cAdvisor and let us know if that works or not?

When if comes to longterm plans with system-docker I left one comment to https://github.com/burmilla/os/issues/28#issuecomment-1399477050

[netsandbox]

With v2.0.0-beta6 I can now use cAdvisor. Thanks for your help.

It would be great if you can backport the system-docker version change to v1, so we can also start monitoring the system container with cAdvisor.

[olljanat]

With v2.0.0-beta6 I can now use cAdvisor.

That is good news. Can you share how you deploy it?

It would be great if you can backport the system-docker version change to v1, so we can also start monitoring the system container with cAdvisor.

Will do that soon if no one finds issues from that new system-docker version.

[netsandbox]

I deployed cAdvisor as a system service, because it needs access to /var/lib/system-docker.

/var/lib/rancher/conf/cadvisor.yaml:

cadvisor:
  image: gcr.io/cadvisor/cadvisor:v0.47.0
  command:
    - '--docker=unix:///var/run/system-docker.sock'
  ports:
    - 80:8080
  privileged: true
  restart: always
  volumes:
    - /:/rootfs:ro
    - /sys:/sys:ro
    - /var/lib/system-docker:/var/lib/system-docker:ro
    - /var/run:/var/run:ro
  labels:
    io.rancher.os.scope: system

[olljanat]

I deployed cAdvisor as a system service, because it needs access to /var/lib/system-docker.

That is good idea and I'm considering making it also available as optional system service as part of BurmillaOS but need to figure out some good port for it because 80 and 8080 are too commonly used by other services.

It would be great if you can backport the system-docker version change to v1, so we can also start monitoring the system container with cAdvisor.

Published as part of v1.9.7-rc1 Please test that and provide your feedback.

[netsandbox]

I just tested the cAdvisor deployment with v1.9.7-rc1 and it works. Thanks!

While in the first place it sounds like a good idea to provide a cAdvisor service out of the box, I'm not sure, because there are so many different deployment types users maybe want to use (behind proxy with HTTPS, Basic Auth).

But it maybe help the users deploy cAdvisor for system container monitoring if there is some guide in the documentation.

[olljanat]

FYI, starting from v2.0.0-rc1 socket file is on path /var/run/system-engine.sock other why same config should still works.