Create job using port forwarding (`ports` in JSON), worked in 0.6.0 - not in 0.7.5

[andypetrella]

Goal

What we would like to do is to forward the port on the slave created using $PORT0 in the command by Chronos (the app to deploy).

Statement

Before we were using marathon v0.6.0, and it used to work by posting the below JSON -- which specifies a ports field containing the local port (marathon machine) that will forward to Chronos.

So that we can use http://:8888 to access Chronos.

{
  "id": "core-service-chronos",
  "instances": 1,
  "cpus": 0.1,
  "mem": 200.0,
  "ports": [
    8888
  ],
  "requirePorts": false,
  "cmd": "java -cp 'target/*' com.airbnb.scheduler.Main --master zk://zk0:2181,zk1:2181,zk2:2181/mesos -z zk0:2181,zk1:2181,zk2:2181 --http_port $PORT0",
  "uris": [
    "hdfs://<hdfs-nnode>:8020/marathon/jobs/core-service-chronos/chronos.tar.gz"
  ]
}

Problem

Since upgrading to 0.7.5, using the same JSON is well deploying Chronos on Mesos, however the port is not opened on the Marathon node.

We dug here and there (we might have left some holes opened, sorry about that... we'll clean :-D). So we found that a parameter requirePorts might be needed, which is per default set to false. Setting it to true will indeed require Mesos to offer port ranges. Our Mesos slaves (0.20.0) aren't configured to do so.

And actually, it should be okay, because we don't want the slave to allow the port 8888 to be opened by the Marathon host to forward request on its 8888 port to the Slave's port $PORT0.

We also found that a containerizer for Mesos could help us by providing the portMappings but we didn't find any examples of such use case. ATM, we only found Docker examples, which we aren't using (yet).

Questions

The port, requirePorts and portMappings should be the right combination, AFAIU. The port and requirePorts to enable HAProxy. And portMappings to allow access from a static host, the Marathon one.

Hence, how the JSON should look like to enable this?

And will that require our Mesos installation to be adapted?

Our case

For the note, we don't require HAProxy now, but only the port forwarding.

[ConnorDoyle]

Hi there.

If I understand correctly, in the past you were able to hit the Marathon host at the application's service port and were load-balanced among the running instances. Are you sure there was nothing else set up on that host to do this for you? Mesos / Marathon does not set up any port forwarding for you automatically. The "service" port (a non-zero element of ports) is just an affordance for configuring a load-balancer. Neither Mesos nor Marathon will open a socket on the service ports. We distribute an example script that can be used to configure HAProxy by reading values from the Marathon REST API. Here is a tutorial describing its use.

Your description of the other options is all correct. The portMappings field can be used with the Docker containerizer, but only in bridged networking mode. The host port must be found in the resource offer, and the containerizer asks dockerd to configure the bridge to route traffic from the host port to the appropriate port on the container.

Let us know if any of that doesn't make sense. We'd like to improve the docs dealing with networking, so any suggestions in that direction are also welcome.

[andypetrella]

Mmmh that's a good point. You're right I was thinking that Marathon was kind-of acting as a proxy.

If it's not the case, hence I'll have to check intenaly what's going on, I guess now that something spooky has been hidden in this area :-D.

Thanks for the information, it was worth asking (and on our side, looking at the internals).

Regarding the doc, it's rather okay I have to say. What can be interesting to add is some words on requirePorts and the mappings I guess. Something that is confusing in the documentation sometimes is the use of host, it's not always clear (a priori) to understand what node is pointed. In this case, host is used in the ports documentation, hence we believed that it'd be the marathon host and thus we thought it was proxying the random port on the slave.

Thank you a lot for the reactivity! Much appreciated (as always)

[ConnorDoyle]

Closing for now, feel free to re-open if you run into more questions.