A "*" Constraint for unique should exist

[florianleibert]

This constraint would ensure that a command is run on all slaves in the cluster.

[sabraham]

I am very interested in this kind of deployment (and more generally, running 1 and only 1 instance on each slave with a given attribute). In this case, I think the interface should disallow scaling the number of instances entirely, as the instance count should always be #(slaves with given attribute).

[normanjoyner]

:+1: This is potentially a very useful constraint.

[aseppala]

+1, this is definitely a common usecase. I would use it combined with UNIQUE hostname constraint to run exactly one instance on each host.

[villind]

+1

[ConnorDoyle]

Makes sense, lots of support, let's do it.

[ConnorDoyle]

The only strange part in the API is the instances key, which will have no bearing on the number of instances we actually launch. Any suggestions of how you'd like to see this handled? One option is add a onePerHost (mod naming) boolean option to the app definition and ignore the instances field if that is not supplied.

[aseppala]

How about

"instances": "*",
"constraints": [["hostname", "UNIQUE"]]

[tnn1t1s]

exactly. and this functionality becomes critical as the size of the cluster becomes non-deterministic.

On Mon, Dec 1, 2014 at 8:38 PM, Florian Leibert notifications@github.com wrote:

This constraint would ensure that a command is run on all slaves in the cluster.

— Reply to this email directly or view it on GitHub https://github.com/mesosphere/marathon/issues/846.

[BrianHicks]

:+1: for "instances": "*" - would it just turn the constraint into a selector in general, though?

[defender]

+1

[andrewortman]

:+1:

[sttts]

+1

[prantaaho]

:+1:

[amiorin]

:+1:

[pigeonflight]

+1

[defender]

Hi ConnorDoyle A "*" Constraint for unique should exist #846 do you think it will be released at 0.8.1, this bug was already reassign several times :)

thanks.

[drexin]

We can't ensure that a task is running on all slaves in the cluster for following reasons:

1. we can't guarantee that there are enough resources on every machine
2. in a multi framework cluster, we don't know if we will ever receive offers for all the machines
3. we could never tell if it has been deployed successfully, because we don't know about all the machines in the cluster

I think this would need direct support from Mesos.

[emilevauge]

+1

[drexin]

Moving this to backlog, as it is not possible to implement this in a reliable manner without support from Mesos.

[matt-deboer]

@drexin, you mentioned:

 3. we could never tell if it has been deployed successfully

Can't you just query the Mesos REST api for the cluster topology, and then keep track of whether the offers received (within some time-limit) covered all the hosts?

GET http://{mesos-master}:5050/registrar(1)/registry

You'd probably also have to have some way of listening for changes to the mesos registry such that you could deploy the "once-per-host" apps to newly added hosts.

For the other two,

1. we can't guarantee enough resources on every machine
2. in a multi framework cluster, we don't know if we will ever receive offers for all the machines

...aren't these true for any deployment? I.e., a deployment asks for N instances, there's no guarantee that there are N instances worth of resources available (and that we'll get offers for them)?

Maybe some of the resource contention problem could be alleviated by having some way to load these once-per-host apps on framework startup...(of course, that might require an ordered startup of frameworks--i.e., make marathon start first)?

Am I on the right track here?

[wricardo]

:+1:

[rewiko]

+1

[kolloch]

Basically, we would need to query the Mesos api for the available slaves regularly. We only get events for lost slaves. It's possible but this is the only use case so far.

But much more important: You somehow need to provision your slaves. If you make something that you want to run on all slaves part of your slave provisioning, you are sure that it runs on all slaves. Implementing this in Marathon is complicated AND by nature fragile.

[air]

See #1264 for additional chat on this.

[BrickXu]

join this thread

[clehene]

Here's a the equivalent from Kubernetes - it's called "Daemon Sets" Curious if Kubernetes on Mesos supports this.

As we describe Marathon as "the init or upstart daemon" it would make sense to have this feature ;)

@drexin I understand the complexity, I think the right way to go about this is to figure out what exactly is needed and discuss this with Mesos devs.

[philipnrmn]

Some great documentation on using GROUP_BY and another feature request for this in #3220

[scatterbrain]

+1 really important for many kinds of services (monitoring etc)

[graphex]

+1 a very useful addition. Yes, this could be done with less complexity to Marathon by provisioning all services on all slaves up front, but that just shifts the complexity and burden of updates and redeploys. If a frequently changing app needs to be on each host, having marathon manage redeployment/restarting/configuration in a consistent manner would be very helpful. Would be worth coordinating any necessary changes to Mesos, and appropriate separation of concerns. Is there a corresponding thread in Mesos for this?

[drewrobb]

I have been doing this by setting an autoscaling rule against the metrics we populate for the number of running slaves against an app with a hostname:UNIQUE constraint that we want on every slave. (I also have a simple internally written autoscaling service for our marathon apps). We also configured each slave to have a small amount of resources reserved for a specific role, and that role is only used by apps that need to run on every slave. It has been working pretty nicely in practice.

[mvanholsteijn]

Being able to define a application that needs to run on every slave is essential functionality of a scheduler.

In Nomad these are called system Jobs, in CoreOS Fleet these are called global units.

So, what will the Marathon equivalent be called and how do we activate them?

It would make our live really easy. We need it now.......

[edurdias]

+1

[clehene]

Marathon shouldn't guarantee that resources exist. We can guarantee resources in Mesos through reservations.

The 1-1 framework - role "binding" may become the next issue, though as we'd need / like to ensure resources by workloads (in Marathon) rather than have it for the entire framework.

[vkhatri]

+1

[JamieCressey]

+1 this would be hugely beneficial for monitoring and log extraction etc.

[mimmus]

We need this now :+1:

[fksimon]

+1

[sybrandy]

@JamieCressey is correct. We're in the same situation where we want to make sure telegraf and logspout are running on all of the slaves for metrics and log collection. It would be great if Marathon/Mesos can ensure that we have instances of each on every node in case failed nodes come back online or we add additional nodes to the cluster.

[jdef]

Running daemons on every node in the cluster in an efficient, robust manner will almost certainly require assistance from Mesos. If you want this feature then I encourage you to vote for support in Mesos because without that we're a bit stranded in Marathon-land.

A workaround meanwhile is to use a hostname UNIQUE constraint with instances set to a greater number of agents than you ever expect to have in your cluster.

https://issues.apache.org/jira/browse/MESOS-6595

[silvamerica]

@jdef Does your workaround leave hanging deployments around, since Marathon will never be able to satisfy the number of instances specified?

[jdef]

Using a greater number of instances is not ideal for interaction w/ the deployment manager. If you're running a fairly static cluster then you could set # instances to match the number of agents you expect to have. If you expect to increase the number of agents in the cluster, then increase instances to match just prior to adding the agent(s). It's a workaround, not a "solution".

On Tue, Nov 15, 2016 at 4:31 PM, Nicholas Silva notifications@github.com wrote:

@jdef https://github.com/jdef Does your workaround leave hanging deployments around, since Marathon will never be able to satisfy the number of instances specified?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mesosphere/marathon/issues/846#issuecomment-260775136, or mute the thread https://github.com/notifications/unsubscribe-auth/ACPVLC6ozQWMkivmjwXsXET35dHpIlOgks5q-iSjgaJpZM4DCvNI .

[gengmao]

@jdef probably a dumb question - why Marathon can't deploy a daemon (with instances=* and hostname UNIQUE) to any new agents offered by Mesos? What specific supports have to be provided by Mesos?

[hampsterx]

+1

[dlaidlaw]

+1 1 per instance per role. If an agent has a specific role, and the acceptedResourceRoles matches then deploy there. This is like an autoscaling that is performed as soon as a new instance is detected and before any other deployments can happen on that new instance.

[janisz]

👎 What if there are no resources to run this app on every agent? If service needs to be running on all agents then there are better ways to put it there (upstart/systemd/supervisor + ansible/puppet/chef/salt). Other orchestrators take advantage of "being the only framework" so they can schedule given tasks before node is available for others. In Mesos you can't do that. Easiest way to achieve it with Marathon is to follow this comment https://github.com/mesosphere/marathon/issues/846#issuecomment-204061357 Finally mixing integer and string in one json field in the API is a mistake.

[dangra]

@janisz the resources for these services is something you usually plan and reserve ahead of time. I am not interested on "before any other deployments can happen", just running a service on all agents that meets the criteria is enough for me.

[mimmus]

On CoreOS (systemd-based), I'm solving dropping-in a unit file to run and restart a monitoring Docker container but it would be great having a way to manage this by Marathon.

[tobilg]

Question I see regarding this:

• What should happen if there are agent nodes added to the cluster? Should Marathon automatically be able to schedule the app on these new nodes as well?

This would mean IMO that Marathon would need to to utilize the /slaves endpoint of the masters (I guess it does so already?), and compare the list of slave from the current call to the list of slaves from the last call... Not sure about the implications.

• Concerning mixing string and integer in a property as @janisz said, I think this wouldn't be a good idea either.

This would effectively mean to change the type of the instances property to string, and then testing and potentially parsing it as integer. It could be discussed if the value -1 could replace *, but maybe there's a better idea.

[meichstedt]

Note: This issue has been migrated to https://jira.mesosphere.com/browse/MARATHON-3791. For more information see https://groups.google.com/forum/#!topic/marathon-framework/khtvf-ifnp8.