Marathon v1.4.0 /v2/apps PUT REST API Error

[zanes2016]

Marathon v1.4.0 on Ubuntu 14.04 REST API /v2/apps PUT does not seem to work. It always returns error status 422.

Try starting a simple new app:

[ { "id": "/test/sleep120", "cmd": "sleep 120", "cpus": 0.3, "instances": 2, "mem": 9 } ]

And the return message (status 422):

{ "message": null }

The easiest way to reproduce this is with the Marathon UI API Console: http://localhost:8080/api-console/index.html And use the /v2/apps PUT API.

This same job, /test/sleep120, works if it's started using the POST API.

The only error message I see is in syslog

marathon[19778]: [2017-02-18 19:09:09,304] ERROR Exception while processing request (mesosphere.marathon.api.MarathonExceptionMapper$$EnhancerByGuice$$9833dc0d:qtp1923232046-38) marathon[19778]: java.lang.IllegalArgumentException: null marathon[19778]: #011at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) marathon[19778]: #011at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) marathon[19778]: #011at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) marathon[19778]: #011at java.lang.reflect.Method.invoke(Method.java:498) marathon[19778]: #011at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) marathon[19778]: #011at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) marathon[19778]: #011at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) . . . marathon[19778]: #011at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62) marathon[19778]: #011at mesosphere.marathon.api.CacheDisablingFilter.doFilter(CacheDisablingFilter.scala:18) marathon[19778]: #011at mesosphere.marathon.api.CacheDisablingFilter$$EnhancerByGuice$$bbaeb0a8.CGLIB$doFilter$0() marathon[19778]: #011at mesosphere.marathon.api.CacheDisablingFilter$$EnhancerByGuice$$bbaeb0a8$$FastClassByGuice$$ccb95931.invoke() . . .

[zanes2016]

For reproducibility, here is the curl command: curl -X PUT -H "Content-type: application/json" localhost:8080/v2/apps --data '[{"id": "/test/sleep60", "cmd": "sleep 60", "cpus": 0.3, "instances": 2, "mem": 9}]'

While the POST command for the same job succeeds: curl -X POST -H "Content-type: application/json" localhost:8080/v2/apps --data '{"id": "/test/sleep60", "cmd": "sleep 60", "cpus": 0.3, "instances": 2, "mem": 9}'

[janisz]

Can't reproduce with httpie

http PUT localhost:8080/v2/apps/ << EOF
[
{
"id": "/test/sleep120",
"cmd": "sleep 120",
"cpus": 0.3,
"instances": 2,
"mem": 9
}
]
EOF

Got:

HTTP/1.1 200 OK
Cache-Control: no-cache, no-store, must-revalidate
Content-Type: application/json; qs=2
Date: Sat, 18 Feb 2017 19:36:11 GMT
Expires: 0
Marathon-Deployment-Id: 164d0aa7-5c29-4f48-bdd4-10ab02e5691e
Pragma: no-cache
Server: Jetty(9.3.6.v20151106)
Transfer-Encoding: chunked
X-Marathon-Leader: http://localhost:8080

{
    "deploymentId": "164d0aa7-5c29-4f48-bdd4-10ab02e5691e", 
    "version": "2017-02-18T19:36:11.355Z"
}

Edit:

curl -X PUT -H "Content-type: application/json" localhost:8080/v2/apps --data '[{"id": "/test/sleep60", "cmd": "sleep 60", "cpus": 0.3, "instances": 2, "mem": 9}]'
{"version":"2017-02-18T19:38:05.903Z","deploymentId":"cb425bc7-6f77-4d3d-a2cf-8e968bea4f1a"}% 

[janisz]

@zanes2016 how do you run Marathon? How it's configured?

[zanes2016]

Maybe it's something with my environment? When I try with httpie:

http PUT localhost:8080/v2/apps/ << EOF [ { "id": "/test/sleep120", "cmd": "sleep 120", "cpus": 0.3, "instances": 2, "mem": 9 } ] EOF

Response

HTTP/1.1 422 Cache-Control: no-cache, no-store, must-revalidate Content-Type: application/json Date: Sat, 18 Feb 2017 19:38:28 GMT Expires: 0 Pragma: no-cache Server: Jetty(9.3.z-SNAPSHOT) Transfer-Encoding: chunked X-Marathon-Leader: http://ip-172-31-23-127:8080 { "message": null }`

OS:

Distributor ID: Ubuntu Description: Ubuntu 14.04.5 LTS Release: 14.04 Codename: trusty

Marathon Version:

1.4.0-1.0.631.ubuntu1404

Mesos:

1.1.0-2.0.107.ubuntu1404

Zookeeper:

3.4.5+dfsg-1

[zanes2016]

@janisz I start Marathon with Upstart

sudo service marathon start

using the init script that came with the package. I tried to use default configuration, so there's nothing in /etc/default//marathon nor /etc/marathon.

Is there some way I can view/post the Marathon configs that would help?

Edit: I'm running this on a single node/machine deployment.

http GET localhost:8080/v2/info

Returns

HTTP/1.1 200 OK
Cache-Control: no-cache, no-store, must-revalidate
Content-Encoding: gzip
Content-Type: application/json; qs=2
Date: Sat, 18 Feb 2017 19:58:06 GMT
Expires: 0
Pragma: no-cache
Server: Jetty(9.3.z-SNAPSHOT)
Transfer-Encoding: chunked
Vary: Accept-Encoding, User-Agent
X-Marathon-Leader: http://ip-172-31-23-127:8080

{
    "buildref": "unknown", 
    "elected": true, 
    "event_subscriber": null, 
    "frameworkId": "2349febf-38dc-4979-bd31-fc912270241e-0000", 
    "http_config": {
        "http_port": 8080, 
        "https_port": 8443
    }, 
    "leader": "ip-172-31-23-127:8080", 
    "marathon_config": {
        "checkpoint": true, 
        "executor": "//cmd", 
        "failover_timeout": 604800, 
        "features": [], 
        "framework_name": "marathon", 
        "ha": true, 
        "hostname": "ip-172-31-23-127", 
        "leader_proxy_connection_timeout_ms": 5000, 
        "leader_proxy_read_timeout_ms": 10000, 
        "local_port_max": 20000, 
        "local_port_min": 10000, 
        "master": "zk://localhost:2181/mesos", 
        "mesos_leader_ui_url": "http://ip-172-31-23-127.us-west-2.compute.internal:5050/", 
        "mesos_role": null, 
        "mesos_user": "root", 
        "reconciliation_initial_delay": 15000, 
        "reconciliation_interval": 600000, 
        "task_launch_timeout": 300000, 
        "task_reservation_timeout": 20000, 
        "webui_url": null
    }, 
    "name": "marathon", 
    "version": "1.4.0", 
    "zookeeper_config": {
        "zk": "zk://localhost:2181/marathon", 
        "zk_max_versions": 50, 
        "zk_session_timeout": 10000, 
        "zk_timeout": 10000
    }
}

[zanes2016]

Maybe I'm missing something obvious. But I'm able to consistently reproduce the issue in a Docker container (ubuntu 14.04 base image) with the following setup:

@janisz Do you have any suggestions?

Host Machine

docker run -it --name marathon -p 8080:8080 ubuntu:trusty bash

Docker Container

sudo apt-get update
sudo apt-get -y install software-properties-common python-software-properties
sudo add-apt-repository -y ppa:webupd8team/java
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv E56151BF
echo "deb http://repos.mesosphere.com/ubuntu trusty main" |    sudo tee /etc/apt/sources.list.d/mesosphere.list
sudo apt-get -y update
echo debconf shared/accepted-oracle-license-v1-1 select true | sudo debconf-set-selections
echo debconf shared/accepted-oracle-license-v1-1 seen true | sudo debconf-set-selections
sudo apt-get -y install curl httpie oracle-java8-installer zookeeper mesos marathon
sudo service zookeeper start
sudo service marathon start
curl -X PUT -H "Content-type: application/json" localhost:8080/v2/apps --data '[{"id": "/test/sleep60", "cmd": "sleep 60", "cpus": 0.3, "instances": 1, "mem": 9}]'

The last command always returns:

{"message":null}

POST API On the other hand, POST works fine:

curl -X POST -H "Content-type: application/json" localhost:8080/v2/apps --data '{"id": "/test/sleep60", "cmd": "sleep 60", "cpus": 0.3, "instances": 1, "mem": 9}'

Returns

{"id":"/test/sleep60","cmd":"sleep 60","args":null,"user":null,"env":{},"instances":1,"cpus":0.3,"mem":9,"disk":0,"gpus":0,"executor":"","constraints":[],"uris":[],"fetch":[],"storeUrls":[],"backoffSeconds":1,"backoffFactor":1.15,"maxLaunchDelaySeconds":3600,"container":null,"healthChecks":[],"readinessChecks":[],"dependencies":[],"upgradeStrategy":{"minimumHealthCapacity":1,"maximumOverCapacity":1},"labels":{},"ipAddress":null,"version":"2017-02-18T21:31:46.788Z","residency":null,"secrets":{},"taskKillGracePeriodSeconds":null,"unreachableStrategy":{"inactiveAfterSeconds":300,"expungeAfterSeconds":600},"killSelection":"YOUNGEST_FIRST","ports":[0],"portDefinitions":[{"port":0,"protocol":"tcp","name":"default","labels":{}}],"requirePorts":false,"tasksStaged":0,"tasksRunning":0,"tasksHealthy":0,"tasksUnhealthy":0,"deployments":[{"id":"a084daaf-2931-4e6a-9ed1-04c6d468daf4"}],"tasks":[]}

Edit: The same setup above works for PUT and POST with v1.3 (marathon=1.3.10-1.0.627.ubuntu1404) installed. Seems to be an issue only with v1.4.0 (marathon=1.4.0-1.0.631.ubuntu1404).

[janisz]

Probably deb is broken.

[wjoel]

I believe it's not just the deb. I built Marathon 1.4.0 from source (I downloaded the .tar.gz from releases here on Github) and it's not possible to update applications from the UI. I see the same 422 responses. This should be easy to reproduce:

1. Create new application in the UI, name test, 1 CPU, 128 MB RAM, 5 MB disk, command while true; do echo testing; sleep 5; done
2. After the application has started, go to the Configuration tab and click Edit. Change the disk space to 7 MB and click Change and deploy configuration. An error message is shown: There was a problem with your configuration. general: App creation unsuccessful. Check your app settings and try again. and in the web console there's a 422 response to the PUT request.
3. Cancel editing the configuration. Click Scale Application and choose two instances. Click Scale application. Another error message: Error Scaling Application. Error scaling /test: [object Object]. Once again there is a 422 response to the PUT request in the web console.

PS. The above steps work as expected with Marathon 1.3.10, also built from source.

[janisz]

@wjoel My bad. I was testing on Master not on 1.4 release. Can reproduce now. I think this could be related to https://github.com/mesosphere/marathon/issues/5213

[meichstedt]

fixed on 1.4 via e1b79523741bf1c266b953eba1bcf8783f819c3a