Applications with persitent local volumes do not start

[Poltergeist]

Adding an application with persistent local volumes does not start on marathon. I have tried multiple different app definitions. Logs and definitions are included below.

• statefull-application-working is a python tool.
• stateful-volumes-working is a simple application with a local volume associated.
• volume is a very simple application which was created with the ui.

Logs

[2016-04-05 13:06:35,274] INFO Processing ForceExpunge(task [stateful-application.49cafa27-fb2c-11e5-9ba8-8a2561d6480a]) for task [stateful-application.49cafa27-fb2c-11e5-9ba8-8a2561d6480a] (mesosphere.marathon.core.launcher.impl.OfferProcessorImpl:ForkJoinPool-2-worker-1)
[2016-04-05 13:06:35,274] INFO Ignoring ForceExpunge for [task [stateful-application.49cafa27-fb2c-11e5-9ba8-8a2561d6480a]], task does not exist (mesosphere.marathon.core.task.tracker.impl.TaskOpProcessorImpl$TaskStateOpResolver:marathon-akka.actor.default-dispatcher-8)
[2016-04-05 13:06:35,274] INFO accepted unreserveAndDestroy for task [stateful-application.49cafa27-fb2c-11e5-9ba8-8a2561d6480a] in offer [9d8f8a58-ef25-4abc-adf0-78e046672dfa-O1421] (mesosphere.marathon.core.matcher.reconcile.impl.OfferMatcherReconciler:ForkJoinPool-2-worker-1)
[2016-04-05 13:06:36,275] INFO Processing ForceExpunge(task [stateful-application.49cafa27-fb2c-11e5-9ba8-8a2561d6480a]) for task [stateful-application.49cafa27-fb2c-11e5-9ba8-8a2561d6480a] (mesosphere.marathon.core.launcher.impl.OfferProcessorImpl:ForkJoinPool-2-worker-7)
[2016-04-05 13:06:36,275] INFO Ignoring ForceExpunge for [task [stateful-application.49cafa27-fb2c-11e5-9ba8-8a2561d6480a]], task does not exist (mesosphere.marathon.core.task.tracker.impl.TaskOpProcessorImpl$TaskStateOpResolver:marathon-akka.actor.default-dispatcher-12)
[2016-04-05 13:06:36,275] INFO accepted unreserveAndDestroy for task [stateful-application.49cafa27-fb2c-11e5-9ba8-8a2561d6480a] in offer [9d8f8a58-ef25-4abc-adf0-78e046672dfa-O1422] (mesosphere.marathon.core.matcher.reconcile.impl.OfferMatcherReconciler:ForkJoinPool-2-worker-9)
[2016-04-05 13:06:37,276] INFO Processing ForceExpunge(task [stateful-application.49cafa27-fb2c-11e5-9ba8-8a2561d6480a]) for task [stateful-application.49cafa27-fb2c-11e5-9ba8-8a2561d6480a] (mesosphere.marathon.core.launcher.impl.OfferProcessorImpl:ForkJoinPool-2-worker-1)
[2016-04-05 13:06:37,276] INFO Ignoring ForceExpunge for [task [stateful-application.49cafa27-fb2c-11e5-9ba8-8a2561d6480a]], task does not exist (mesosphere.marathon.core.task.tracker.impl.TaskOpProcessorImpl$TaskStateOpResolver:marathon-akka.actor.default-dispatcher-12)
[2016-04-05 13:06:37,276] INFO accepted unreserveAndDestroy for task [stateful-application.49cafa27-fb2c-11e5-9ba8-8a2561d6480a] in offer [9d8f8a58-ef25-4abc-adf0-78e046672dfa-O1423] (mesosphere.marathon.core.matcher.reconcile.impl.OfferMatcherReconciler:ForkJoinPool-2-worker-5)
[2016-04-05 13:06:37,294] INFO initiate task reconciliation (mesosphere.marathon.MarathonSchedulerActor:marathon-akka.actor.default-dispatcher-7)
[2016-04-05 13:06:37,294] INFO reconcile [/stateful-application-working] with latest version [2016-04-05T12:49:00.912Z] (mesosphere.marathon.health.MarathonHealthCheckManager$$EnhancerByGuice$$5c8ec9ec:ForkJoinPool-2-worker-11)
[2016-04-05 13:06:37,294] INFO reconcile [/local-volumes-working] with latest version [2016-04-05T12:50:43.341Z] (mesosphere.marathon.health.MarathonHealthCheckManager$$EnhancerByGuice$$5c8ec9ec:ForkJoinPool-2-worker-1)
[2016-04-05 13:06:37,294] INFO reconcile [/volumes] with latest version [2016-04-05T12:55:40.039Z] (mesosphere.marathon.health.MarathonHealthCheckManager$$EnhancerByGuice$$5c8ec9ec:ForkJoinPool-2-worker-7)
[2016-04-05 13:06:37,294] INFO Requesting task reconciliation with the Mesos master (mesosphere.marathon.SchedulerActions:ForkJoinPool-2-worker-15)
[2016-04-05 13:06:37,294] INFO addAllFor [/volumes] version [2016-04-05T12:55:40.039Z] (mesosphere.marathon.health.MarathonHealthCheckManager$$EnhancerByGuice$$5c8ec9ec:ForkJoinPool-2-worker-7)
[2016-04-05 13:06:37,294] INFO addAllFor [/local-volumes-working] version [2016-04-05T12:50:43.341Z] (mesosphere.marathon.health.MarathonHealthCheckManager$$EnhancerByGuice$$5c8ec9ec:ForkJoinPool-2-worker-1)
[2016-04-05 13:06:37,294] INFO task reconciliation has finished (mesosphere.marathon.MarathonSchedulerActor:marathon-akka.actor.default-dispatcher-12)
[2016-04-05 13:06:37,294] INFO addAllFor [/stateful-application-working] version [2016-04-05T12:49:00.912Z] (mesosphere.marathon.health.MarathonHealthCheckManager$$EnhancerByGuice$$5c8ec9ec:ForkJoinPool-2-worker-11)
[2016-04-05 13:06:38,277] INFO Processing ForceExpunge(task [stateful-application.49cafa27-fb2c-11e5-9ba8-8a2561d6480a]) for task [stateful-application.49cafa27-fb2c-11e5-9ba8-8a2561d6480a] (mesosphere.marathon.core.launcher.impl.OfferProcessorImpl:ForkJoinPool-2-worker-9)
[2016-04-05 13:06:38,277] INFO Ignoring ForceExpunge for [task [stateful-application.49cafa27-fb2c-11e5-9ba8-8a2561d6480a]], task does not exist (mesosphere.marathon.core.task.tracker.impl.TaskOpProcessorImpl$TaskStateOpResolver:marathon-akka.actor.default-dispatcher-12)
[2016-04-05 13:06:38,277] INFO accepted unreserveAndDestroy for task [stateful-application.49cafa27-fb2c-11e5-9ba8-8a2561d6480a] in offer [9d8f8a58-ef25-4abc-adf0-78e046672dfa-O1424] (mesosphere.marathon.core.matcher.reconcile.impl.OfferMatcherReconciler:ForkJoinPool-2-worker-9)

App definitions:

{
  "apps": [
    {
      "id": "/stateful-application-working",
      "cmd": "pip install markdoc && markdoc init ./home || echo && echo \"# Started At `date`\" >> ./home/wiki/index.md && cd ./home && markdoc build && (markdoc serve -i 0.0.0.0 -p $PORT0 & while true; do sleep 5; echo `date` >> wiki/index.md; echo >> wiki/index.md; markdoc build; done)",
      "instances": 1,
      "cpus": 0.1,
      "mem": 32,
      "disk": 0,
      "ports": [
        0
      ],
      "maxLaunchDelaySeconds": 3600,

      "container": {
        "type": "MESOS",
        "volumes": [
          {
            "containerPath": "home",
            "mode": "RW",
            "persistent": {
              "size": 100
            }
          }
        ]
      },
      "upgradeStrategy": {
        "minimumHealthCapacity": 0.5,
        "maximumOverCapacity": 0
      },
      "residency": {
        "relaunchEscalationTimeoutSeconds": 10,
        "taskLostBehavior": "WAIT_FOREVER"
      }
    },
    {
      "id": "/local-volumes-working",
      "cmd": "sleep 1000",
      "instances": 1,
      "cpus": 0.1,
      "mem": 32,
      "disk": 0,
      "ports": [
        0
      ],
      "container": {
        "type": "MESOS",
        "volumes": [
          {
            "containerPath": "home",
            "mode": "RW",
            "persistent": {
              "size": 100
            }
          }
        ]
      },
      "upgradeStrategy": {
        "minimumHealthCapacity": 0.5,
        "maximumOverCapacity": 0
      },
      "residency": {
        "relaunchEscalationTimeoutSeconds": 10,
        "taskLostBehavior": "WAIT_FOREVER"
      },
      "versionInfo": {
        "lastScalingAt": "2016-04-05T12:50:43.341Z",
        "lastConfigChangeAt": "2016-04-05T12:50:43.341Z"
      }
    },
    {
      "id": "/volumes",
      "cmd": "sleep 1000;",
      "instances": 1,
      "cpus": 0.1,
      "mem": 128,
      "disk": 0,
      "container": {
        "type": "MESOS",
        "volumes": [
          {
            "containerPath": "home",
            "mode": "RW",
            "persistent": {
              "size": 1024
            }
          }
        ]
      },
      "upgradeStrategy": {
        "minimumHealthCapacity": 0.5,
        "maximumOverCapacity": 0
      },
      "residency": {
        "relaunchEscalationTimeoutSeconds": 10,
        "taskLostBehavior": "WAIT_FOREVER"
      }
    }
  ]
}

[kolloch]

This might be related to #3673

[collinmsn]

@Poltergeist I ran into the same problem, has this bug fixed or any workaround?

[meichstedt]

Note: This issue has been migrated to https://jira.mesosphere.com/browse/MARATHON-1730. For more information see https://groups.google.com/forum/#!topic/marathon-framework/khtvf-ifnp8.