Mesos Agents not routeable

[jsmodic]

I started up minimesos with the default minimesosFile and created an application in Marathon. That application then registers in Marathon with the Mesos agent IP and host port.

However, that IP / port aren't routeable, so it can't be accessed that way. Also, the Marathon health checks fail for the same reason. The actual container IP / port are accessible though.

I'm not sure if this is intended behavior / minimesos intends to solve this problem, though.

[frankscholten]

@jsmodic Thanks for your report. Could you paste the Marathon file you used?

[jsmodic]

I used basically the default file, tweaking the agents resources a bit:

minimesos { clusterName = "Mesos Cluster" exposePorts = false loggingLevel = "INFO" mapAgentSandboxVolume = false mesosVersion = "0.25" timeout = 60

agent {
    imageName = "containersol/mesos-agent"
    imageTag = "# derive from mesos version"
    loggingLevel = "# INHERIT FROM CLUSTER"
    portNumber = 5051

    resources {

        cpu {
            role = "*"
            value = 6
        }

        disk {
            role = "*"
            value = 512
        }

        mem {
            role = "*"
            value = 4098
        }

        ports {
            role = "*"
            value = "[31000-32000]"
        }
    }
}

agent {
    imageName = "containersol/mesos-agent"
    imageTag = "# derive from mesos version"
    loggingLevel = "# INHERIT FROM CLUSTER"
    portNumber = 5052

    resources {

        cpu {
            role = "*"
            value = 6
        }

        disk {
            role = "*"
            value = 512
        }

        mem {
            role = "*"
            value = 4098
        }

        ports {
            role = "*"
            value = "[32000-33000]"
        }
    }
}

marathon {
    imageName = "mesosphere/marathon"
    imageTag = "v0.15.3"

    app {
        marathonJson = "https://raw.githubusercontent.com/ContainerSolutions/minimesos/27744df7071fb2d4ab15562bf2654ec37f4d539d/apps/weave-scope.json"
    }

}

master {
    imageName = "containersol/mesos-master"
    imageTag = "# derive from mesos version"
    loggingLevel = "# INHERIT FROM CLUSTER"
}

zookeeper {
    imageName = "jplock/zookeeper"
    imageTag = "3.4.6"
}

}

And my Docker version (I tried 1.9.1 as well):

docker -v Docker version 1.11.0, build 4dc5990

uname -a Linux Thinkpad540 3.19.0-43-lowlatency #49~14.04.1-Ubuntu SMP PREEMPT Thu Dec 31 16:45:05 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

containersol/mesos-agent 0.25.0-0.2.70.ubuntu1404 49727d9a7b1f 4 weeks ago 468.1 MB containersol/mesos-master 0.25.0-0.2.70.ubuntu1404 5df810059d87 4 weeks ago 468.1 MB containersol/consul-server 0.6 420f36a72aa9 5 weeks ago 37.75 MB containersol/mesos-master 0.27.1-2.0.226.ubuntu1404 287a8682c5f1 5 weeks ago 478.4 MB containersol/mesos-agent 0.27.1-2.0.226.ubuntu1404 cb0b7279f2d4 5 weeks ago 478.4 MB

[frankscholten]

Ok so you used the default minimesosFile. Could you also post the Marathon JSON file?

[jsmodic]

Sure, it was just a simple app that looked like (except with the real id / image):

{
  "id": "myid",
  "container": {
    "docker": {
      "image": "myimage",
      "network": "BRIDGE",
      "portMappings": [
        { "containerPort": 8080, "hostPort": 0, "protocol": "tcp" }
      ]
    },
    "type": "DOCKER"
  },
  "env":  {
    "CLOUD_ENVIRONMENT": "docker"
  },
  "cpus": 0.03,
  "mem": 600.0,
  "instances": 1,
  "upgradeStrategy": {
    "minimumHealthCapacity": 0.8,
    "maximumOverCapacity": 0.1
  }
}

[sadovnikov]

hi @jsmodic,

Containers with Mesos Agents do not have complete installation of docker. Their binaries use docker.sock file and talk to docker daemon of the host. Therefore your myimage container does not run inside Mesos Agent, but next to it, and it gets its own IP address.

Yes, this setup is different from real Mesos environment, but it gives you access to docker images on the host. Without that Mesos Agent would try to download your myimage image from Docker Hub.

Is there a chance you can adjust reporting of IP so it would work in both situations: inside the agent and next to agent?

Cheers, Viktor

[jsmodic]

Ahh, well that makes sense why the IP would not work as it is then, as the containers aren't actually running in the agent.

My intention is to use minimesos to test my containers using Marathon / Mesos out of the box, not actually write my own framework. So I'm not in control of the IP reporting. But I can work around it and ignore all the values in Marathon entirely and disable the healthchecks.

How do others use minimesos currently to test their frameworks if Mesos is reporting back a value that can't be reached by the framework? Like, if I was writing a Marathon equivalent, it seems I couldn't actually test out the healthcheck functionality unless I had special code to hash it out with Docker directly when running in minimesos and to trust the value when in real mesos.

[frankscholten]

@jsmodic Hi John, this issue can be solved by introducing IP-per-task, see http://mesos.apache.org/documentation/latest/networking-for-mesos-managed-containers and https://mesosphere.github.io/marathon/docs/ip-per-task.html

I have created an issue for it: https://github.com/ContainerSolutions/minimesos/issues/420

in case of Mesos Elasticsearch we use Elasticsearch' internal API to find the endpoints. See https://github.com/mesos/elasticsearch/blob/master/system-test/src/systemTest/java/org/apache/mesos/elasticsearch/systemtest/DiscoverySystemTest.java

[ozdanborne]

I wonder if this would be solved by simply passing the ipAddress: {} field in the application's Marathon spec. The presence of this field would tell Marathon the application is accessible by its IP address, and should result in the UI displaying the docker bridge ip. Kind of hacky approach, since it doesn't translate to a real cluster well (docker bridge IPs are not typically cluster-accessible). But Here's a close example of what that would look like

[frankscholten]

@djosborne Can you add the ipAddress: {} field without using net modules plugin like Project Calico?

[frankscholten]

Closing as work is tracked in https://github.com/ContainerSolutions/minimesos/issues/420