Open seanfulton opened 4 years ago
f1-outsourcing
commented
4 years ago I have the same also. I deleted everthing in /var/lib/mesos and /var/lib/zookeeper and still I have this shit. I reported that mesos-dns is not correctly reporting ip addresses almost a year ago and it is still not fixed. I do not get that people are installing this 1GB of DCOS blob with this buggie coding going on there. As if they are a more reliable partner than redhat.
jeschkies
commented
4 years ago Which version are you using?
f1-outsourcing
commented
4 years ago this 1.9.109-1.el7 and 1.9.136-1.el7 the same
jeschkies
commented
4 years ago @f1-outsourcing, you should not blindly delete files. Also, remember that this is all open source. We are happy to accept bug fixes from you.
@seanfulton, sorry for replying so late. Do you still have this issue? If I understood correctly, Marathon fails to load the old state. Was this after an upgrade?
f1-outsourcing
commented
4 years ago I am deleting files and remove configuration options to see if that results to something. Whatever I am changing I am only able to get marathon-1.7.216-9e2a9b579 working with mesos-1.10.0-2.0.1 I have looked at the change log but cannot seem to find a change that breaks.
This is what I posted to the marathon-framework mailing list:
All of a sudden I having problems with marathon ui getting stuck at 'loading' and end points like http://m01.local:8081/v2/info are not responding (http://m01.local:8081/ping gives pong). I have now downgraded the test cluster to one node, running only mesos-master and zookeeper and marathon. Cleaning between tests the /var/lib/zookeeper and the /var/lib/mesos directories. I have also removed many of the configuration options I had, like ssl etc.
I am only able to get to run marathon-1.7.216-9e2a9b579. marathon-1.8.222-86475ddac and marathon-1.10.17-c427ce965 are having the above mentioned errors/problem.
I have been comparing the marathon 1.7 and marathon 1.8 logs and this what I have noticed. There are quite a bit of log statements missing between 'All services up and running. (mesosphere.marathon.MarathonApp:main' and 'akka://marathon/deadLetters' in the 1.8 log.
Anyone had something similar?
[@mesos-master]# rpm -qa | grep java python-javapackages-3.4.1-11.el7.noarch tzdata-java-2020a-1.el7.noarch java-1.8.0-openjdk-headless-1.8.0.252.b09-2.el7_8.x86_64 javapackages-tools-3.4.1-11.el7.noarch
[@mesos-master]# uname -a Linux m01.local 3.10.0-1127.10.1.el7.x86_64 #1 SMP Wed Jun 3 14:28:03 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
[@mesos-master]# cat /etc/redhat-release CentOS Linux release 7.8.2003 (Core)
Jun 7 17:40:59 m01 marathon: [2020-06-07 17:40:59,696] INFO All services up and running. (mesosphere.marathon.MarathonApp:main) Jun 7 17:41:13 m01 marathon: [2020-06-07 17:41:13,833] INFO initiate task reconciliation (mesosphere.marathon.MarathonSchedulerActor:marathon-akka.actor.default-dispatcher-9) Jun 7 17:41:13 m01 marathon: [2020-06-07 17:41:13,854] INFO Requesting task reconciliation with the Mesos master (mesosphere.marathon.SchedulerActions:scheduler-actions-thread-0) Jun 7 17:41:13 m01 mesos-master[11203]: I0607 17:41:13.858621 11227 master.cpp:8846] Performing implicit task state reconciliation for framework f5d67e06-6600-4fb9-94dc-a878be2563be-0000 (marathon) at scheduler-6d98d1e0-a7d2-4517-a0ce-5819a36414c9@192.168.10.151:36941 Jun 7 17:41:13 m01 marathon: [2020-06-07 17:41:13,864] INFO task reconciliation has finished (mesosphere.marathon.MarathonSchedulerActor:marathon-akka.actor.default-dispatcher-4)
Jun 7 17:41:13 m01 marathon: [2020-06-07 17:41:13,879] INFO Message [mesosphere.marathon.MarathonSchedulerActor$TasksReconciled$] from Actor[akka://marathon/user/MarathonScheduler/$a#1746491390] to Actor[akka://marathon/deadLetters] was not delivered. [1] dead letters encountered. If this is not an expected behavior, then [Actor[akka://marathon/deadLetters]] may have terminated unexpectedly, This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. (akka.actor.DeadLetterActorRef:marathon-akka.actor.default-dispatcher-7) Jun 7 17:41:13 m01 marathon: [2020-06-07 17:41:13,910] INFO Prompting Mesos for a heartbeat via explicit task reconciliation (mesosphere.marathon.core.heartbeat.MesosHeartbeatMonitor$$anon$1:marathon-akka.actor.default-dispatcher-7) Jun 7 17:41:13 m01 mesos-master[11203]: I0607 17:41:13.914615 11228 master.cpp:8889] Performing explicit task state reconciliation for 1 tasks of framework f5d67e06-6600-4fb9-94dc-a878be2563be-0000 (marathon) at scheduler-6d98d1e0-a7d2-4517-a0ce-5819a36414c9@192.168.10.151:36941 Jun 7 17:41:13 m01 marathon: [2020-06-07 17:41:13,924] INFO Received fake heartbeat task-status update (mesosphere.marathon.core.heartbeat.MesosHeartbeatMonitor:Thread-13) Jun 7 17:41:28 m01 marathon: [2020-06-07 17:41:28,939] INFO Prompting Mesos for a heartbeat via explicit task reconciliation (mesosphere.marathon.core.heartbeat.MesosHeartbeatMonitor$$anon$1:marathon-akka.actor.default-dispatcher-4) Jun 7 17:41:28 m01 mesos-master[11203]: I0607 17:41:28.946494 11229 master.cpp:8889] Performing explicit task state reconciliation for 1 tasks of framework f5d67e06-6600-4fb9-94dc-a878be2563be-0000 (marathon) at scheduler-6d98d1e0-a7d2-4517-a0ce-5819a36414c9@192.168.10.151:36941 Jun 7 17:41:28 m01 marathon: [2020-06-07 17:41:28,950] INFO Received fake heartbeat task-status update (mesosphere.marathon.core.heartbeat.MesosHeartbeatMonitor:Thread-14)
Jun 7 17:37:02 m01 marathon: [2020-06-07 17:37:02,681] INFO All services up and running. (mesosphere.marathon.MarathonApp:main) Jun 7 17:37:06 m01 marathon: [2020-06-07 17:37:06,222] INFO Received TimedCheck (mesosphere.marathon.core.flow.impl.ReviveOffersActor:marathon-akka.actor.default-dispatcher-8) Jun 7 17:37:06 m01 marathon: [2020-06-07 17:37:06,228] INFO => revive offers NOW, canceling any scheduled revives (mesosphere.marathon.core.flow.impl.ReviveOffersActor:marathon-akka.actor.default-dispatcher-8) Jun 7 17:37:06 m01 mesos-master[10661]: I0607 17:37:06.232568 10690 master.cpp:5521] Processing REVIVE call for framework f2318310-8c7b-438c-9a9d-48fdf1cd0406-0000 (marathon) at scheduler-44c3a1b5-3c08-4fdd-ae79-6a1fd172e3b5@192.168.10.151:40447 Jun 7 17:37:06 m01 mesos-master[10661]: I0607 17:37:06.232730 10690 hierarchical.cpp:1788] Unsuppressed offers and cleared filters for roles { } of framework f2318310-8c7b-438c-9a9d-48fdf1cd0406-0000 Jun 7 17:37:06 m01 marathon: [2020-06-07 17:37:06,235] INFO 2 further revives still needed. Repeating reviveOffers according to --revive_offers_repetitions 3 (mesosphere.marathon.core.flow.impl.ReviveOffersActor:marathon-akka.actor.default-dispatcher-8) Jun 7 17:37:06 m01 marathon: [2020-06-07 17:37:06,238] INFO => Schedule next revive at 2020-06-07T15:37:11.228Z in 4990 milliseconds, adhering to --min_revive_offers_interval 5000 (ms) (mesosphere.marathon.core.flow.impl.ReviveOffersActor:marathon-akka.actor.default-dispatcher-8) Jun 7 17:37:11 m01 marathon: [2020-06-07 17:37:11,240] INFO Received TimedCheck (mesosphere.marathon.core.flow.impl.ReviveOffersActor:marathon-akka.actor.default-dispatcher-5) Jun 7 17:37:11 m01 mesos-master[10661]: I0607 17:37:11.246363 10685 master.cpp:5521] Processing REVIVE call for framework f2318310-8c7b-438c-9a9d-48fdf1cd0406-0000 (marathon) at scheduler-44c3a1b5-3c08-4fdd-ae79-6a1fd172e3b5@192.168.10.151:40447 Jun 7 17:37:11 m01 mesos-master[10661]: I0607 17:37:11.246500 10685 hierarchical.cpp:1788] Unsuppressed offers and cleared filters for roles { } of framework f2318310-8c7b-438c-9a9d-48fdf1cd0406-0000 Jun 7 17:37:11 m01 marathon: [2020-06-07 17:37:11,240] INFO => revive offers NOW, canceling any scheduled revives (mesosphere.marathon.core.flow.impl.ReviveOffersActor:marathon-akka.actor.default-dispatcher-5) Jun 7 17:37:11 m01 marathon: [2020-06-07 17:37:11,241] INFO 1 further revives still needed. Repeating reviveOffers according to --revive_offers_repetitions 3 (mesosphere.marathon.core.flow.impl.ReviveOffersActor:marathon-akka.actor.default-dispatcher-5) Jun 7 17:37:11 m01 marathon: [2020-06-07 17:37:11,241] INFO => Schedule next revive at 2020-06-07T15:37:16.240Z in 4999 milliseconds, adhering to --min_revive_offers_interval 5000 (ms) (mesosphere.marathon.core.flow.impl.ReviveOffersActor:marathon-akka.actor.default-dispatcher-5) Jun 7 17:37:16 m01 marathon: [2020-06-07 17:37:16,261] INFO Received TimedCheck (mesosphere.marathon.core.flow.impl.ReviveOffersActor:marathon-akka.actor.default-dispatcher-8) Jun 7 17:37:16 m01 mesos-master[10661]: I0607 17:37:16.265516 10689 master.cpp:5521] Processing REVIVE call for framework f2318310-8c7b-438c-9a9d-48fdf1cd0406-0000 (marathon) at scheduler-44c3a1b5-3c08-4fdd-ae79-6a1fd172e3b5@192.168.10.151:40447 Jun 7 17:37:16 m01 mesos-master[10661]: I0607 17:37:16.265655 10689 hierarchical.cpp:1788] Unsuppressed offers and cleared filters for roles { * } of framework f2318310-8c7b-438c-9a9d-48fdf1cd0406-0000 Jun 7 17:37:16 m01 marathon: [2020-06-07 17:37:16,261] INFO => revive offers NOW, canceling any scheduled revives (mesosphere.marathon.core.flow.impl.ReviveOffersActor:marathon-akka.actor.default-dispatcher-8) Jun 7 17:37:16 m01 marathon: [2020-06-07 17:37:16,409] INFO initiate task reconciliation (mesosphere.marathon.MarathonSchedulerActor:marathon-akka.actor.default-dispatcher-5) Jun 7 17:37:16 m01 marathon: [2020-06-07 17:37:16,437] INFO Requesting task reconciliation with the Mesos master (mesosphere.marathon.SchedulerActions:scheduler-actions-thread-0) Jun 7 17:37:16 m01 mesos-master[10661]: I0607 17:37:16.441344 10686 master.cpp:8846] Performing implicit task state reconciliation for framework f2318310-8c7b-438c-9a9d-48fdf1cd0406-0000 (marathon) at scheduler-44c3a1b5-3c08-4fdd-ae79-6a1fd172e3b5@192.168.10.151:40447 Jun 7 17:37:16 m01 marathon: [2020-06-07 17:37:16,444] INFO task reconciliation has finished (mesosphere.marathon.MarathonSchedulerActor:marathon-akka.actor.default-dispatcher-2)
Jun 7 17:37:16 m01 marathon: [2020-06-07 17:37:16,459] INFO Message [mesosphere.marathon.MarathonSchedulerActor$TasksReconciled$] from Actor[akka://marathon/user/MarathonScheduler/$a#-463341905] to Actor[akka://marathon/deadLetters] was not delivered. [1] dead letters encountered. If this is not an expected behavior, then [Actor[akka://marathon/deadLetters]] may have terminated unexpectedly, This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. (akka.actor.DeadLetterActorRef:marathon-akka.actor.default-dispatcher-8) Jun 7 17:37:16 m01 marathon: [2020-06-07 17:37:16,502] INFO Prompting Mesos for a heartbeat via explicit task reconciliation (mesosphere.marathon.core.heartbeat.MesosHeartbeatMonitor$$anon$1:marathon-akka.actor.default-dispatcher-5) Jun 7 17:37:16 m01 mesos-master[10661]: I0607 17:37:16.506299 10687 master.cpp:8889] Performing explicit task state reconciliation for 1 tasks of framework f2318310-8c7b-438c-9a9d-48fdf1cd0406-0000 (marathon) at scheduler-44c3a1b5-3c08-4fdd-ae79-6a1fd172e3b5@192.168.10.151:40447 Jun 7 17:37:16 m01 marathon: [2020-06-07 17:37:16,513] INFO Received fake heartbeat task-status update (mesosphere.marathon.core.heartbeat.MesosHeartbeatMonitor:Thread-14) Jun 7 17:37:31 m01 marathon: [2020-06-07 17:37:31,012] INFO Killing overdue instances: (mesosphere.marathon.core.task.jobs.impl.OverdueInstancesActor$Support:scala-execution-context-global-54) Jun 7 17:37:31 m01 marathon: [2020-06-07 17:37:31,018] INFO Kill and forget following instances for reason Overdue: (mesosphere.marathon.core.task.termination.impl.KillServ
jeschkies
commented
4 years ago @f1-outsourcing, could you attach the complete logs of Marathon 1.8 from when you start until you made a request to v2/info.
f1-outsourcing
commented
4 years ago Jun 7 17:40:43 m01 marathon: No start hook file found ($HOOK_MARATHON_START). Proceeding with the start script.
Jun 7 17:40:46 m01 marathon: [2020-06-07 17:40:46,247] INFO Successfully loaded libmesos: version 1.10.0 (mesosphere.marathon.MarathonApp:main)
Jun 7 17:40:48 m01 marathon: [2020-06-07 17:40:48,481] INFO Slf4jLogger started (akka.event.slf4j.Slf4jLogger:marathon-akka.actor.default-dispatcher-6)
Jun 7 17:40:50 m01 marathon: [2020-06-07 17:40:50,642] INFO Starting heartbeat actor (mesosphere.marathon.core.heartbeat.HeartbeatActor:marathon-akka.actor.default-dispatcher-6)
Jun 7 17:40:51 m01 marathon: [2020-06-07 17:40:51,005] INFO Running in ZooKeeper 3.4.x compatibility mode (org.apache.curator.utils.Compatibility:main)
Jun 7 17:40:51 m01 marathon: [2020-06-07 17:40:51,113] INFO Starting (org.apache.curator.framework.imps.CuratorFrameworkImpl:main)
Jun 7 17:40:51 m01 marathon: [2020-06-07 17:40:51,136] INFO Client environment:zookeeper.version=3.4.11-37e277162d567b55a07d1755f0b31c32e93c01a0, built on 11/01/2017 18:06 GMT (org.apache.zookeeper.ZooKeeper:main)
Jun 7 17:40:51 m01 marathon: [2020-06-07 17:40:51,139] INFO Client environment:host.name=m01.local (org.apache.zookeeper.ZooKeeper:main)
Jun 7 17:40:51 m01 marathon: [2020-06-07 17:40:51,141] INFO Client environment:java.version=1.8.0_252 (org.apache.zookeeper.ZooKeeper:main)
Jun 7 17:40:51 m01 marathon: [2020-06-07 17:40:51,143] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper:main)
Jun 7 17:40:51 m01 marathon: [2020-06-07 17:40:51,147] INFO Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre (org.apache.zookeeper.ZooKeeper:main)
Jun 7 17:40:51 m01 marathon: [2020-06-07 17:40:51,150] INFO Client environment:java.class.path=/home/software/marathon-1.8.222-86475ddac/lib/mesosphere.marathon.marathon-1.8.222.jar:/home/software/marathon-1.8.222-86475ddac/lib/mesosphere.marathon.plugin-interface-1.8.222.jar:/home/software/marathon-1.8.222-86475ddac/lib/commons-codec.commons-codec-1.9.jar:/home/software/marathon-1.8.222-86475ddac/lib/org.eclipse.jetty.jetty-http-9.4.8.v20171121.jar:/home/software/marathon-1.8.222-86475ddac/lib/net.logstash.logback.logstash-logback-encoder-4.9.jar:/home/software/marathon-1.8.222-86475ddac/lib/org.apache.yetus.audience-annotations-0.5.0.jar:/home/software/marathon-1.8.222-86475ddac/lib/org.scala-lang.scala-reflect-2.12.7.jar:/home/software/marathon-1.8.222-86475ddac/lib/com.fasterxml.jackson.module.jackson-module-jaxb-annotations-2.9.5.jar:/home/software/marathon-1.8.222-86475ddac/lib/org.scalamacros.resetallattrs_2.12-1.0.0.jar:/home/software/marathon-1.8.222-86475ddac/lib/org.glassfish.jersey.core.jersey-server-2.27.jar:/home/software/marathon-1.8.222-86475ddac/lib/io.netty.netty-3.10.5.Final.jar:/home/software/marathon-1.8.222-86475ddac/lib/com.typesafe.akka.akka-protobuf_2.12-2.5.23.jar:/home/software/marathon-1.8.222-86475ddac/lib/javax.xml.bind.jaxb-api-2.3.1.jar:/home/software/marathon-1.8.222-86475ddac/lib/com.fasterxml.jackson.module.jackson-module-paranamer-2.9.5.jar:/home/software/marathon-1.8.222-86475ddac/lib/com.lightbend.akka.akka-stream-alpakka-s3_2.12-0.14.jar:/home/software/marathon-1.8.222-86475ddac/lib/com.fasterxml.jackson.module.jackson-module-scala_2.12-2.9.5.jar:/home/software/marathon-1.8.222-86475ddac/lib/com.typesafe.play.play-json_2.12-2.6.7.jar:/home/software/marathon-1.8.222-86475ddac/lib/org.glassfish.jersey.media.jersey-media-multipart-2.27.jar:/home/software/marathon-1.8.222-86475ddac/lib/org.glassfish.hk2.hk2-api-2.5.0-b42.jar:/home/software/marathon-1.8.222-86475ddac/lib/org.eclipse.jetty.jetty-servlet-9.4.8.v20171121.jar:/home/software/marathon-1.8.222-86475ddac/lib/org.ap
Jun 7 17:40:51 m01 marathon: [2020-06-07 17:40:51,150] INFO Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper:main)
Jun 7 17:40:51 m01 marathon: [2020-06-07 17:40:51,150] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper:main)
Jun 7 17:40:51 m01 marathon: [2020-06-07 17:40:51,152] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper:main)
Jun 7 17:40:51 m01 marathon: [2020-06-07 17:40:51,152] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper:main)
Jun 7 17:40:51 m01 marathon: [2020-06-07 17:40:51,152] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper:main)
Jun 7 17:40:51 m01 marathon: [2020-06-07 17:40:51,153] INFO Client environment:os.version=3.10.0-1127.10.1.el7.x86_64 (org.apache.zookeeper.ZooKeeper:main)
Jun 7 17:40:51 m01 marathon: [2020-06-07 17:40:51,153] INFO Client environment:user.name=marathon (org.apache.zookeeper.ZooKeeper:main)
Jun 7 17:40:51 m01 marathon: [2020-06-07 17:40:51,156] INFO Client environment:user.home=/home/marathon (org.apache.zookeeper.ZooKeeper:main)
Jun 7 17:40:51 m01 marathon: [2020-06-07 17:40:51,156] INFO Client environment:user.dir=/home/software/marathon-1.8.222-86475ddac (org.apache.zookeeper.ZooKeeper:main)
Jun 7 17:40:51 m01 marathon: [2020-06-07 17:40:51,166] INFO Initiating client connection, connectString=m01.local:2181 sessionTimeout=10000 watcher=org.apache.curator.ConnectionState@516ebdf8 (org.apache.zookeeper.ZooKeeper:main)
Jun 7 17:40:51 m01 marathon: [2020-06-07 17:40:51,259] INFO Default schema (org.apache.curator.framework.imps.CuratorFrameworkImpl:main)
Jun 7 17:40:51 m01 marathon: [2020-06-07 17:40:51,262] INFO Opening socket connection to server m01.local/192.168.10.151:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn:main-SendThread(m01.local:2181))
Jun 7 17:40:51 m01 marathon: [2020-06-07 17:40:51,286] INFO Socket connection established to m01.local/192.168.10.151:2181, initiating session (org.apache.zookeeper.ClientCnxn:main-SendThread(m01.local:2181))
Jun 7 17:40:51 m01 marathon: [2020-06-07 17:40:51,324] INFO Session establishment complete on server m01.local/192.168.10.151:2181, sessionid = 0x1728f7057480004, negotiated timeout = 10000 (org.apache.zookeeper.ClientCnxn:main-SendThread(m01.local:2181))
Jun 7 17:40:51 m01 marathon: [2020-06-07 17:40:51,344] INFO State change: CONNECTED (org.apache.curator.framework.state.ConnectionStateManager:main-EventThread)
Jun 7 17:40:51 m01 marathon: [2020-06-07 17:40:51,426] WARN The version of ZooKeeper being used doesn't support Container nodes. CreateMode.PERSISTENT will be used instead. (org.apache.curator.utils.ZKPaths:main)
Jun 7 17:40:52 m01 marathon: [2020-06-07 17:40:52,383] INFO Started TaskTrackerUpdateStepsProcessorImpl with steps:
Jun 7 17:40:52 m01 marathon: * continueOnError(notifyHealthCheckManager)
Jun 7 17:40:52 m01 marathon: * continueOnError(notifyRateLimiter)
Jun 7 17:40:52 m01 marathon: * continueOnError(notifyLaunchQueue)
Jun 7 17:40:52 m01 marathon: * continueOnError(postTaskStatusEvent)
Jun 7 17:40:52 m01 marathon: * continueOnError(scaleApp) (mesosphere.marathon.core.task.tracker.impl.InstanceTrackerUpdateStepProcessorImpl:main)
Jun 7 17:40:53 m01 marathon: [2020-06-07 17:40:53,128] INFO Loading plugins implementing 'mesosphere.marathon.plugin.task.RunSpecTaskProcessor' from these urls: [] (mesosphere.marathon.core.plugin.impl.PluginManagerImpl:main)
Jun 7 17:40:53 m01 marathon: [2020-06-07 17:40:53,168] INFO Found 0 plugins. (mesosphere.marathon.core.plugin.impl.PluginManagerImpl:main)
Jun 7 17:40:53 m01 marathon: [2020-06-07 17:40:53,173] INFO Loading plugins implementing 'mesosphere.marathon.plugin.validation.RunSpecValidator' from these urls: [] (mesosphere.marathon.core.plugin.impl.PluginManagerImpl:main)
Jun 7 17:40:53 m01 marathon: [2020-06-07 17:40:53,175] INFO Found 0 plugins. (mesosphere.marathon.core.plugin.impl.PluginManagerImpl:main)
Jun 7 17:40:53 m01 marathon: [2020-06-07 17:40:53,316] INFO Loading plugins implementing 'mesosphere.marathon.plugin.auth.Authenticator' from these urls: [] (mesosphere.marathon.core.plugin.impl.PluginManagerImpl:main)
Jun 7 17:40:53 m01 marathon: [2020-06-07 17:40:53,322] INFO Found 0 plugins. (mesosphere.marathon.core.plugin.impl.PluginManagerImpl:main)
Jun 7 17:40:53 m01 marathon: [2020-06-07 17:40:53,333] INFO Loading plugins implementing 'mesosphere.marathon.plugin.auth.Authorizer' from these urls: [] (mesosphere.marathon.core.plugin.impl.PluginManagerImpl:main)
Jun 7 17:40:53 m01 marathon: [2020-06-07 17:40:53,336] INFO Found 0 plugins. (mesosphere.marathon.core.plugin.impl.PluginManagerImpl:main)
Jun 7 17:40:53 m01 marathon: [2020-06-07 17:40:53,372] INFO Loading plugins implementing 'mesosphere.marathon.plugin.scheduler.SchedulerPlugin' from these urls: [] (mesosphere.marathon.core.plugin.impl.PluginManagerImpl:main)
Jun 7 17:40:53 m01 marathon: [2020-06-07 17:40:53,376] INFO Found 0 plugins. (mesosphere.marathon.core.plugin.impl.PluginManagerImpl:main)
Jun 7 17:40:53 m01 marathon: [2020-06-07 17:40:53,537] INFO Started status update processor (mesosphere.marathon.core.task.update.impl.TaskStatusUpdateProcessorImpl:main)
Jun 7 17:40:53 m01 marathon: [2020-06-07 17:40:53,628] INFO All actors suspended:
Jun 7 17:40:53 m01 marathon: * Actor[akka://marathon/user/deploymentManager#1106232599]
Jun 7 17:40:53 m01 marathon: * Actor[akka://marathon/user/offerMatcherManager#-1876253398]
Jun 7 17:40:53 m01 marathon: * Actor[akka://marathon/user/taskKillServiceActor#-778749779]
Jun 7 17:40:53 m01 marathon: * Actor[akka://marathon/user/rateLimiter#-2044772544]
Jun 7 17:40:53 m01 marathon: * Actor[akka://marathon/user/killOverdueStagedTasks#1783287740]
Jun 7 17:40:53 m01 marathon: * Actor[akka://marathon/user/launchQueue#465459569]
Jun 7 17:40:53 m01 marathon: * Actor[akka://marathon/user/expungeOverdueLostTasks#-1395865443]
Jun 7 17:40:53 m01 marathon: * Actor[akka://marathon/user/reviveOffers#-363088573]
Jun 7 17:40:53 m01 marathon: * Actor[akka://marathon/user/instanceTracker#-1547106758]
Jun 7 17:40:53 m01 marathon: * Actor[akka://marathon/user/offerMatcherLaunchTokens#790335717] (mesosphere.marathon.core.leadership.impl.LeadershipCoordinatorActor:marathon-akka.actor.default-dispatcher-5)
Jun 7 17:40:53 m01 marathon: [2020-06-07 17:40:53,671] INFO Created Mesos heartbeat monitor for scheduler mesosphere.marathon.MarathonScheduler@26221bad (mesosphere.marathon.core.heartbeat.MesosHeartbeatMonitor:main)
Jun 7 17:40:53 m01 marathon: [2020-06-07 17:40:53,707] INFO Created Mesos heartbeat monitor for scheduler mesosphere.marathon.MarathonScheduler@26221bad (mesosphere.marathon.core.heartbeat.MesosHeartbeatMonitor:main)
Jun 7 17:40:53 m01 marathon: [2020-06-07 17:40:53,733] INFO History actor ready (mesosphere.marathon.core.history.impl.HistoryActor:marathon-akka.actor.default-dispatcher-2)
Jun 7 17:40:53 m01 marathon: [2020-06-07 17:40:53,881] INFO Logging initialized @9789ms to org.eclipse.jetty.util.log.Slf4jLog (org.eclipse.jetty.util.log:main)
Jun 7 17:40:53 m01 marathon: [2020-06-07 17:40:53,960] INFO Adding HTTP support. (mesosphere.marathon.api.HttpModule:main)
Jun 7 17:40:53 m01 marathon: [2020-06-07 17:40:53,967] INFO No HTTPS support configured. (mesosphere.marathon.api.HttpModule:main)
Jun 7 17:40:54 m01 marathon: [2020-06-07 17:40:54,118] INFO Starting Marathon 1.8.222/86475ddac52b6aa05697be7ea590dde9d288496b with --hostname m01.local --zk zk://m01.local:2181/marathon --http_port 8081 --master zk://m01.local:2181/mesos --logging_level info --mesos_user root (mesosphere.marathon.MarathonApp:main)
Jun 7 17:40:54 m01 marathon: [2020-06-07 17:40:54,280] INFO Loading plugins implementing 'mesosphere.marathon.plugin.http.HttpRequestHandler' from these urls: [] (mesosphere.marathon.core.plugin.impl.PluginManagerImpl:main)
Jun 7 17:40:54 m01 marathon: [2020-06-07 17:40:54,287] INFO Found 0 plugins. (mesosphere.marathon.core.plugin.impl.PluginManagerImpl:main)
Jun 7 17:40:55 m01 marathon: [2020-06-07 17:40:55,617] INFO Now standing by. Closing existing handles and rejecting new. (mesosphere.marathon.core.event.impl.stream.HttpEventStreamActor:marathon-akka.actor.default-dispatcher-6)
Jun 7 17:40:55 m01 marathon: [2020-06-07 17:40:55,711] INFO Starting up HttpServer. (mesosphere.marathon.api.MarathonHttpService:MarathonHttpService STARTING)
Jun 7 17:40:55 m01 marathon: [2020-06-07 17:40:55,717] INFO jetty-9.4.8.v20171121, build timestamp: 2017-11-21T22:27:37+01:00, git hash: 82b8fb23f757335bb3329d540ce37a2a2615f0a8 (org.eclipse.jetty.server.Server:MarathonHttpService STARTING)
Jun 7 17:40:55 m01 marathon: [2020-06-07 17:40:55,720] INFO Starting up (mesosphere.marathon.MarathonSchedulerService:MarathonSchedulerService)
Jun 7 17:40:55 m01 marathon: [2020-06-07 17:40:55,724] INFO Beginning run (mesosphere.marathon.MarathonSchedulerService:MarathonSchedulerService)
Jun 7 17:40:55 m01 marathon: [2020-06-07 17:40:55,782] INFO Starting leader latch (mesosphere.marathon.core.election.CuratorElectionStream$:MarathonSchedulerService)
Jun 7 17:40:55 m01 marathon: [2020-06-07 17:40:55,862] INFO Leader watch loop 1 retrying; waiting for latch initialization. (mesosphere.marathon.core.election.CuratorElectionStream$:pool-1-thread-1)
Jun 7 17:40:55 m01 marathon: [2020-06-07 17:40:55,889] INFO Leader watch loop 1 retrying; waiting for latch initialization. (mesosphere.marathon.core.election.CuratorElectionStream$:pool-1-thread-1)
Jun 7 17:40:55 m01 marathon: [2020-06-07 17:40:55,903] INFO Leader watch loop 1 retrying; waiting for latch initialization. (mesosphere.marathon.core.election.CuratorElectionStream$:pool-1-thread-1)
Jun 7 17:40:55 m01 marathon: [2020-06-07 17:40:55,934] INFO Leader watch loop 1 retrying; waiting for latch initialization. (mesosphere.marathon.core.election.CuratorElectionStream$:pool-1-thread-1)
Jun 7 17:40:56 m01 marathon: [2020-06-07 17:40:56,037] INFO Leader won: m01.local:8081 (mesosphere.marathon.core.election.CuratorElectionStream$:marathon-akka.actor.default-dispatcher-6)
Jun 7 17:40:56 m01 marathon: [2020-06-07 17:40:56,042] INFO As new leader running the driver (mesosphere.marathon.MarathonSchedulerService:pool-1-thread-1)
Jun 7 17:40:57 m01 marathon: [2020-06-07 17:40:57,496] INFO No migration necessary, no version stored (mesosphere.marathon.storage.migration.Migration:scala-execution-context-global-40)
Jun 7 17:40:57 m01 marathon: [2020-06-07 17:40:57,626] INFO Migration successfully applied for version Version(18, 200, 0, PERSISTENCE_STORE) (mesosphere.marathon.storage.migration.Migration:scala-execution-context-global-40)
Jun 7 17:40:57 m01 marathon: [2020-06-07 17:40:57,736] INFO Call preDriverStarts callbacks on (mesosphere.marathon.MarathonSchedulerService:pool-1-thread-1)
Jun 7 17:40:57 m01 marathon: [2020-06-07 17:40:57,737] INFO Finished preDriverStarts callbacks (mesosphere.marathon.MarathonSchedulerService:pool-1-thread-1)
Jun 7 17:40:57 m01 marathon: [2020-06-07 17:40:57,758] INFO ExpungeOverdueLostTasksActor has started (mesosphere.marathon.core.task.jobs.impl.ExpungeOverdueLostTasksActor:marathon-akka.actor.default-dispatcher-5)
Jun 7 17:40:57 m01 marathon: [2020-06-07 17:40:57,760] INFO started RateLimiterActor (mesosphere.marathon.core.launchqueue.impl.RateLimiterActor:marathon-akka.actor.default-dispatcher-4)
Jun 7 17:40:57 m01 marathon: [2020-06-07 17:40:57,832] INFO All actors active:
Jun 7 17:40:57 m01 marathon: * Actor[akka://marathon/user/deploymentManager#1106232599]
Jun 7 17:40:57 m01 marathon: * Actor[akka://marathon/user/offerMatcherManager#-1876253398]
Jun 7 17:40:57 m01 marathon: * Actor[akka://marathon/user/taskKillServiceActor#-778749779]
Jun 7 17:40:57 m01 marathon: * Actor[akka://marathon/user/rateLimiter#-2044772544]
Jun 7 17:40:57 m01 marathon: * Actor[akka://marathon/user/killOverdueStagedTasks#1783287740]
Jun 7 17:40:57 m01 marathon: * Actor[akka://marathon/user/launchQueue#465459569]
Jun 7 17:40:57 m01 marathon: * Actor[akka://marathon/user/expungeOverdueLostTasks#-1395865443]
Jun 7 17:40:57 m01 marathon: * Actor[akka://marathon/user/reviveOffers#-363088573]
Jun 7 17:40:57 m01 marathon: * Actor[akka://marathon/user/instanceTracker#-1547106758]
Jun 7 17:40:57 m01 marathon: * Actor[akka://marathon/user/offerMatcherLaunchTokens#790335717] (mesosphere.marathon.core.leadership.impl.LeadershipCoordinatorActor:marathon-akka.actor.default-dispatcher-7)
Jun 7 17:40:57 m01 marathon: [2020-06-07 17:40:57,845] INFO InstanceTrackerActor is starting. Instances loading initiated. (mesosphere.marathon.core.task.tracker.impl.InstanceTrackerActor:marathon-akka.actor.default-dispatcher-4)
Jun 7 17:40:57 m01 marathon: [2020-06-07 17:40:57,986] INFO Loaded 0 instances (mesosphere.marathon.core.task.tracker.impl.InstancesLoaderImpl:scala-execution-context-global-40)
Jun 7 17:40:58 m01 marathon: [2020-06-07 17:40:58,006] WARN No frameworkId could be read and no instances are defined. This will result in a new frameworkId (mesosphere.marathon.MesosSchedulerDriverFactory$:pool-1-thread-1)
Jun 7 17:40:58 m01 marathon: [2020-06-07 17:40:58,010] INFO Create new Scheduler Driver with frameworkId: None and scheduler mesosphere.marathon.core.heartbeat.MesosHeartbeatMonitor@1850d3c8 (mesosphere.marathon.MarathonSchedulerDriver$:pool-1-thread-1)
Jun 7 17:40:58 m01 marathon: [2020-06-07 17:40:58,021] INFO Instances loading complete. (mesosphere.marathon.core.task.tracker.impl.InstanceTrackerActor:marathon-akka.actor.default-dispatcher-6)
Jun 7 17:40:58 m01 marathon: [2020-06-07 17:40:58,024] INFO Loaded 0 instances. (mesosphere.marathon.core.task.tracker.impl.InstanceTrackerActor:marathon-akka.actor.default-dispatcher-6)
Jun 7 17:40:58 m01 marathon: [2020-06-07 17:40:58,126] INFO PARTITION_AWARE feature enabled. (mesosphere.marathon.MarathonSchedulerDriver$:pool-1-thread-1)
Jun 7 17:40:58 m01 marathon: [2020-06-07 17:40:58,137] INFO REGION_AWARE feature enabled (mesosphere.marathon.MarathonSchedulerDriver$:pool-1-thread-1)
Jun 7 17:40:58 m01 marathon: [2020-06-07 17:40:58,281] WARN A provider mesosphere.marathon.api.v2.QueueResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider mesosphere.marathon.api.v2.QueueResource will be ignored. (org.glassfish.jersey.internal.inject.Providers:MarathonHttpService STARTING)
Jun 7 17:40:58 m01 marathon: [2020-06-07 17:40:58,284] WARN A provider mesosphere.marathon.api.v2.GroupsResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider mesosphere.marathon.api.v2.GroupsResource will be ignored. (org.glassfish.jersey.internal.inject.Providers:MarathonHttpService STARTING)
Jun 7 17:40:58 m01 marathon: [2020-06-07 17:40:58,286] WARN A provider mesosphere.marathon.api.v2.InfoResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider mesosphere.marathon.api.v2.InfoResource will be ignored. (org.glassfish.jersey.internal.inject.Providers:MarathonHttpService STARTING)
Jun 7 17:40:58 m01 marathon: [2020-06-07 17:40:58,289] WARN A provider mesosphere.marathon.api.v2.TasksResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider mesosphere.marathon.api.v2.TasksResource will be ignored. (org.glassfish.jersey.internal.inject.Providers:MarathonHttpService STARTING)
Jun 7 17:40:58 m01 marathon: [2020-06-07 17:40:58,290] WARN A provider mesosphere.marathon.api.v2.LeaderResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider mesosphere.marathon.api.v2.LeaderResource will be ignored. (org.glassfish.jersey.internal.inject.Providers:MarathonHttpService STARTING)
Jun 7 17:40:58 m01 marathon: [2020-06-07 17:40:58,291] WARN A provider mesosphere.marathon.api.v2.PodsResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider mesosphere.marathon.api.v2.PodsResource will be ignored. (org.glassfish.jersey.internal.inject.Providers:MarathonHttpService STARTING)
Jun 7 17:40:58 m01 marathon: [2020-06-07 17:40:58,296] WARN A provider mesosphere.marathon.api.v2.PluginsResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider mesosphere.marathon.api.v2.PluginsResource will be ignored. (org.glassfish.jersey.internal.inject.Providers:MarathonHttpService STARTING)
Jun 7 17:40:58 m01 marathon: [2020-06-07 17:40:58,297] WARN A provider mesosphere.marathon.api.SystemResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider mesosphere.marathon.api.SystemResource will be ignored. (org.glassfish.jersey.internal.inject.Providers:MarathonHttpService STARTING)
Jun 7 17:40:58 m01 marathon: [2020-06-07 17:40:58,297] WARN A provider mesosphere.marathon.api.v2.DeploymentsResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider mesosphere.marathon.api.v2.DeploymentsResource will be ignored. (org.glassfish.jersey.internal.inject.Providers:MarathonHttpService STARTING)
Jun 7 17:40:58 m01 marathon: [2020-06-07 17:40:58,303] WARN A provider mesosphere.marathon.api.v2.AppsResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider mesosphere.marathon.api.v2.AppsResource will be ignored. (org.glassfish.jersey.internal.inject.Providers:MarathonHttpService STARTING)
Jun 7 17:40:58 m01 marathon: [2020-06-07 17:40:58,836] INFO Suppress: There are no pending instances right now. (mesosphere.marathon.core.launchqueue.impl.ReviveOffersStreamLogic$:marathon-akka.actor.default-dispatcher-4)
Jun 7 17:40:58 m01 marathon: [2020-06-07 17:40:58,841] INFO Sending suppress (mesosphere.marathon.core.launchqueue.impl.ReviveOffersActor:marathon-akka.actor.default-dispatcher-4)
Jun 7 17:40:58 m01 marathon: [2020-06-07 17:40:58,892] INFO Heartbeat actor activated (mesosphere.marathon.core.heartbeat.HeartbeatActor:marathon-akka.actor.default-dispatcher-4)
Jun 7 17:40:58 m01 marathon: 2020-06-07 17:40:58,905:11304(0x7f257edc9700):ZOO_INFO@log_env@753: Client environment:zookeeper.version=zookeeper C client 3.4.8
Jun 7 17:40:58 m01 marathon: 2020-06-07 17:40:58,907:11304(0x7f257edc9700):ZOO_INFO@log_env@757: Client environment:host.name=m01.local
Jun 7 17:40:58 m01 marathon: 2020-06-07 17:40:58,909:11304(0x7f257edc9700):ZOO_INFO@log_env@764: Client environment:os.name=Linux
Jun 7 17:40:58 m01 marathon: 2020-06-07 17:40:58,909:11304(0x7f257edc9700):ZOO_INFO@log_env@765: Client environment:os.arch=3.10.0-1127.10.1.el7.x86_64
Jun 7 17:40:58 m01 marathon: 2020-06-07 17:40:58,909:11304(0x7f257edc9700):ZOO_INFO@log_env@766: Client environment:os.version=#1 SMP Wed Jun 3 14:28:03 UTC 2020
Jun 7 17:40:58 m01 marathon: 2020-06-07 17:40:58,909:11304(0x7f257edc9700):ZOO_INFO@log_env@774: Client environment:user.name=(null)
Jun 7 17:40:58 m01 marathon: 2020-06-07 17:40:58,909:11304(0x7f257edc9700):ZOO_INFO@log_env@782: Client environment:user.home=/home/marathon
Jun 7 17:40:58 m01 marathon: 2020-06-07 17:40:58,909:11304(0x7f257edc9700):ZOO_INFO@log_env@794: Client environment:user.dir=/home/software/marathon-1.8.222-86475ddac
Jun 7 17:40:58 m01 marathon: 2020-06-07 17:40:58,909:11304(0x7f257edc9700):ZOO_INFO@zookeeper_init@827: Initiating client connection, host=m01.local:2181 sessionTimeout=10000 watcher=0x7f2585e994e0 sessionId=0 sessionPasswd=<null> context=0x7f25c5b14930 flags=0
Jun 7 17:40:58 m01 marathon: I0607 17:40:58.921898 11398 sched.cpp:239] Version: 1.10.0
Jun 7 17:40:58 m01 marathon: [2020-06-07 17:40:58,936] INFO Starting scheduler actor (mesosphere.marathon.MarathonSchedulerActor:marathon-akka.actor.default-dispatcher-3)
Jun 7 17:40:58 m01 marathon: [2020-06-07 17:40:58,936] INFO Became active. Accepting event streaming requests. (mesosphere.marathon.core.event.impl.stream.HttpEventStreamActor:marathon-akka.actor.default-dispatcher-2)
Jun 7 17:40:58 m01 zookeeper: 2020-06-07 17:40:58,956 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@822] - Connection request from old client /192.168.10.151:37566; will be dropped if server is in r-o mode
Jun 7 17:40:58 m01 marathon: 2020-06-07 17:40:58,959:11304(0x7f257bcc2700):ZOO_INFO@check_events@1764: initiated connection to server [192.168.10.151:2181]
Jun 7 17:40:58 m01 marathon: [2020-06-07 17:40:58,965] INFO Scheduler actor ready (mesosphere.marathon.MarathonSchedulerActor:marathon-akka.actor.default-dispatcher-3)
Jun 7 17:40:58 m01 marathon: 2020-06-07 17:40:58,990:11304(0x7f257bcc2700):ZOO_INFO@check_events@1811: session establishment complete on server [192.168.10.151:2181], sessionId=0x1728f7057480005, negotiated timeout=10000
Jun 7 17:40:58 m01 marathon: I0607 17:40:58.993643 11406 group.cpp:341] Group process (zookeeper-group(1)@192.168.10.151:36941) connected to ZooKeeper
Jun 7 17:40:58 m01 marathon: I0607 17:40:58.995862 11406 group.cpp:831] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
Jun 7 17:40:58 m01 marathon: I0607 17:40:58.998025 11406 group.cpp:419] Trying to create path '/mesos' in ZooKeeper
Jun 7 17:40:59 m01 marathon: I0607 17:40:59.003587 11410 detector.cpp:152] Detected a new leader: (id='0')
Jun 7 17:40:59 m01 marathon: I0607 17:40:59.006004 11406 group.cpp:700] Trying to get '/mesos/json.info_0000000000' in ZooKeeper
Jun 7 17:40:59 m01 marathon: I0607 17:40:59.035372 11409 zookeeper.cpp:262] A new leading master (UPID=master@192.168.10.151:5050) is detected
Jun 7 17:40:59 m01 marathon: I0607 17:40:59.038015 11404 sched.cpp:343] New master detected at master@192.168.10.151:5050
Jun 7 17:40:59 m01 marathon: I0607 17:40:59.041571 11404 sched.cpp:363] No credentials provided. Attempting to register without authentication
Jun 7 17:40:59 m01 mesos-master[11203]: I0607 17:40:59.046126 11233 master.cpp:2901] Received SUBSCRIBE call for framework 'marathon' at scheduler-6d98d1e0-a7d2-4517-a0ce-5819a36414c9@192.168.10.151:36941
Jun 7 17:40:59 m01 mesos-master[11203]: I0607 17:40:59.046856 11233 master.cpp:2983] Subscribing framework marathon with checkpointing enabled and capabilities [ PARTITION_AWARE, REGION_AWARE ]
Jun 7 17:40:59 m01 mesos-master[11203]: I0607 17:40:59.051831 11233 master.cpp:9897] Adding framework f5d67e06-6600-4fb9-94dc-a878be2563be-0000 (marathon) at scheduler-6d98d1e0-a7d2-4517-a0ce-5819a36414c9@192.168.10.151:36941 with roles { } suppressed
Jun 7 17:40:59 m01 mesos-master[11203]: I0607 17:40:59.056349 11228 hierarchical.cpp:767] Added framework f5d67e06-6600-4fb9-94dc-a878be2563be-0000
Jun 7 17:40:59 m01 marathon: I0607 17:40:59.060653 11403 sched.cpp:751] Framework registered with f5d67e06-6600-4fb9-94dc-a878be2563be-0000
Jun 7 17:40:59 m01 marathon: [2020-06-07 17:40:59,095] INFO Registered heartbeat monitor (mesosphere.marathon.core.heartbeat.MesosHeartbeatMonitor:Thread-12)
Jun 7 17:40:59 m01 marathon: [2020-06-07 17:40:59,098] INFO Registered as f5d67e06-6600-4fb9-94dc-a878be2563be-0000 to master 'f5d67e06-6600-4fb9-94dc-a878be2563be' (mesosphere.marathon.MarathonScheduler:Thread-12)
Jun 7 17:40:59 m01 marathon: [2020-06-07 17:40:59,100] INFO Mesos Master version 1.10.0 (mesosphere.marathon.MarathonScheduler:Thread-12)
Jun 7 17:40:59 m01 marathon: [2020-06-07 17:40:59,638] WARN The following warnings have been detected: WARNING: The (sub)resource method redirectUI in mesosphere.marathon.api.SystemResource contains empty path annotation.
Jun 7 17:40:59 m01 marathon: (org.glassfish.jersey.internal.Errors:MarathonHttpService STARTING)
Jun 7 17:40:59 m01 marathon: [2020-06-07 17:40:59,655] INFO Started o.e.j.s.ServletContextHandler@65b2a783{/,null,AVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:MarathonHttpService STARTING)
Jun 7 17:40:59 m01 marathon: [2020-06-07 17:40:59,690] INFO Started ServerConnector@4f89a524{HTTP/1.1,[http/1.1]}{0.0.0.0:8081} (org.eclipse.jetty.server.AbstractConnector:MarathonHttpService STARTING)
Jun 7 17:40:59 m01 marathon: [2020-06-07 17:40:59,693] INFO Started @15602ms (org.eclipse.jetty.server.Server:MarathonHttpService STARTING)
Jun 7 17:40:59 m01 marathon: [2020-06-07 17:40:59,696] INFO All services up and running. (mesosphere.marathon.MarathonApp:main)
Jun 7 17:41:13 m01 marathon: [2020-06-07 17:41:13,833] INFO initiate task reconciliation (mesosphere.marathon.MarathonSchedulerActor:marathon-akka.actor.default-dispatcher-9)
Jun 7 17:41:13 m01 marathon: [2020-06-07 17:41:13,854] INFO Requesting task reconciliation with the Mesos master (mesosphere.marathon.SchedulerActions:scheduler-actions-thread-0)
Jun 7 17:41:13 m01 mesos-master[11203]: I0607 17:41:13.858621 11227 master.cpp:8846] Performing implicit task state reconciliation for framework f5d67e06-6600-4fb9-94dc-a878be2563be-0000 (marathon) at scheduler-6d98d1e0-a7d2-4517-a0ce-5819a36414c9@192.168.10.151:36941
Jun 7 17:41:13 m01 marathon: [2020-06-07 17:41:13,864] INFO task reconciliation has finished (mesosphere.marathon.MarathonSchedulerActor:marathon-akka.actor.default-dispatcher-4)
Jun 7 17:41:13 m01 marathon: [2020-06-07 17:41:13,879] INFO Message [mesosphere.marathon.MarathonSchedulerActor$TasksReconciled$] from Actor[akka://marathon/user/MarathonScheduler/$a#1746491390] to Actor[akka://marathon/deadLetters] was not delivered. [1] dead letters encountered. If this is not an expected behavior, then [Actor[akka://marathon/deadLetters]] may have terminated unexpectedly, This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. (akka.actor.DeadLetterActorRef:marathon-akka.actor.default-dispatcher-7)
Jun 7 17:41:13 m01 marathon: [2020-06-07 17:41:13,910] INFO Prompting Mesos for a heartbeat via explicit task reconciliation (mesosphere.marathon.core.heartbeat.MesosHeartbeatMonitor$$anon$1:marathon-akka.actor.default-dispatcher-7)
Jun 7 17:41:13 m01 mesos-master[11203]: I0607 17:41:13.914615 11228 master.cpp:8889] Performing explicit task state reconciliation for 1 tasks of framework f5d67e06-6600-4fb9-94dc-a878be2563be-0000 (marathon) at scheduler-6d98d1e0-a7d2-4517-a0ce-5819a36414c9@192.168.10.151:36941
Jun 7 17:41:13 m01 marathon: [2020-06-07 17:41:13,924] INFO Received fake heartbeat task-status update (mesosphere.marathon.core.heartbeat.MesosHeartbeatMonitor:Thread-13)
Jun 7 17:41:28 m01 marathon: [2020-06-07 17:41:28,939] INFO Prompting Mesos for a heartbeat via explicit task reconciliation (mesosphere.marathon.core.heartbeat.MesosHeartbeatMonitor$$anon$1:marathon-akka.actor.default-dispatcher-4)
Jun 7 17:41:28 m01 mesos-master[11203]: I0607 17:41:28.946494 11229 master.cpp:8889] Performing explicit task state reconciliation for 1 tasks of framework f5d67e06-6600-4fb9-94dc-a878be2563be-0000 (marathon) at scheduler-6d98d1e0-a7d2-4517-a0ce-5819a36414c9@192.168.10.151:36941
Jun 7 17:41:28 m01 marathon: [2020-06-07 17:41:28,950] INFO Received fake heartbeat task-status update (mesosphere.marathon.core.heartbeat.MesosHeartbeatMonitor:Thread-14)
@f1-outsourcing, could you attach the complete logs of Marathon 1.8 from when you start until you made a request to
v2/info.
Is this useful?
f1-outsourcing
commented
4 years ago Is this being looked at still?
f1-outsourcing
commented
4 years ago If your strategy at D2iQ/mesosphere is to give 'shitty' support to marathon, because you want to push people into using DCOS. You should consider there is a flip side to that approach, I perceive this as:
If you do not have time to resolve this probably simple issue, how can users of DCOS trust you to do a good job maintaining this 1GB blob install you offer? I prefer to run your mesos and marathon on CentOS/RedHat because I trust RedHat to do a better job at back porting bugs than you do. EL is their core business, it is not your core business maintaining packages/distributions. There is no way for me to check if your installed binaries are up to date, and you even take the time to update them.
I assume this issue I have, is maybe related to an incompatibility somewhere between mesos and marathon. So that could indicate that in your DCOS blob you are maybe mixing java versions, or using leading edge software releases instead of LTS versions. At least have a policy that is deviates from that of LTS distributions, which again of course does not make sense from a security and stability perspective. The fact that you advertise in mesos that you are cni 0.3 compatible, which you are not[2]. Is only contributing to the perception you do not mind deviating from best practices.
Someone else reported the same issue[1] in March on your JIRA and his work-a-round of downgrading to Marathon 1.7. He also did not get any attention for 6 month's. Whether or not your software is open source, you should attend to such issues quicker, where people need to downgrade so many versions. Otherwise just make this closed source, or withdraw it from public support, so people are not wasting time trying/using it.
f1-outsourcing
commented
4 years ago Also not working Downgrading mesos to mesos-1.7.2-2.0.1.x86_64 and downgrading marathon to sbt -mem 1024 'run --master m01.local:5050 --zk zk://m01.local:2181/v18207 --framework_name v18207 --http_port 7070'
What about this message:
[info] [2020-07-18 13:14:42,675] INFO Message [mesosphere.marathon.MarathonSchedulerActor$TasksReconciled$] from Actor[akka://marathon/user/MarathonScheduler/$a#-1125671270] to Actor[akka://marathon/deadLetters] was not delivered. [1] dead letters encountered. If this is not an expected behavior, then [Actor[akka://marathon/deadLetters]] may have terminated unexpectedly, This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. (akka.actor.DeadLetterActorRef:marathon-akka.actor.default-dispatcher-8)
f1-outsourcing
commented
4 years ago debug log of http://test2.local:7070/v2/info
[info] [2020-07-18 13:26:09,845] DEBUG Current State: LaunchTokens:100 OffersWanted:false Matchers:0 OfferQueues:0 UnprocessedOffers:0 (mesosphere.marathon.core.matcher.manager.impl.OfferMatcherManagerActor:marathon-akka.actor.default-dispatcher-9)
[info] [2020-07-18 13:26:09,845] DEBUG Reject Elapsed offers. Unprocessed: 0 Overdue:0 (mesosphere.marathon.core.matcher.manager.impl.OfferMatcherManagerActor:marathon-akka.actor.default-dispatcher-9)
[info] [2020-07-18 13:26:11,788] DEBUG Queued change org.eclipse.jetty.io.ManagedSelector$Accept@2b70db91 on org.eclipse.jetty.io.ManagedSelector@4752a8a1 id=0 keys=0 selected=0 actions=0 (org.eclipse.jetty.io.ManagedSelector:qtp1021656938-31-acceptor-0@654d6319-ServerConnector@2a2843ec{HTTP/1.1,[http/1.1]}{0.0.0.0:7070})
[info] [2020-07-18 13:26:11,789] DEBUG Selector sun.nio.ch.EPollSelectorImpl@3c2dd5d9 woken up from select, 0/0 selected (org.eclipse.jetty.io.ManagedSelector:qtp1021656938-30)
[info] [2020-07-18 13:26:11,790] DEBUG Selector sun.nio.ch.EPollSelectorImpl@3c2dd5d9 processing 0 keys, 1 actions (org.eclipse.jetty.io.ManagedSelector:qtp1021656938-30)
[info] [2020-07-18 13:26:11,790] DEBUG action=org.eclipse.jetty.io.ManagedSelector$Accept@2b70db91 wakeup=false (org.eclipse.jetty.io.ManagedSelector:qtp1021656938-30)
[info] [2020-07-18 13:26:11,790] DEBUG EatWhatYouKill@2d2e8982/SelectorProducer@2b454e29/PRODUCING/ReservedThreadExecutor@4fbe007b{s=0/1,p=0}@SelectorManager@ServerConnector@2a2843ec{HTTP/1.1,[http/1.1]}{0.0.0.0:7070} t=org.eclipse.jetty.io.ManagedSelector$Accept@2b70db91/NON_BLOCKING (org.eclipse.jetty.util.thread.strategy.EatWhatYouKill:qtp1021656938-30)
[info] [2020-07-18 13:26:11,791] DEBUG EatWhatYouKill@2d2e8982/SelectorProducer@2b454e29/PRODUCING/ReservedThreadExecutor@4fbe007b{s=0/1,p=0}@SelectorManager@ServerConnector@2a2843ec{HTTP/1.1,[http/1.1]}{0.0.0.0:7070} PC t=org.eclipse.jetty.io.ManagedSelector$Accept@2b70db91 (org.eclipse.jetty.util.thread.strategy.EatWhatYouKill:qtp1021656938-30)
[info] [2020-07-18 13:26:11,792] DEBUG Queued change org.eclipse.jetty.io.ManagedSelector$CreateEndPoint@1acde28a on org.eclipse.jetty.io.ManagedSelector@4752a8a1 id=0 keys=1 selected=0 actions=0 (org.eclipse.jetty.io.ManagedSelector:qtp1021656938-30)
[info] [2020-07-18 13:26:11,792] DEBUG Forcing selection, actions=1 (org.eclipse.jetty.io.ManagedSelector:qtp1021656938-30)
[info] [2020-07-18 13:26:11,794] DEBUG action=null wakeup=true (org.eclipse.jetty.io.ManagedSelector:qtp1021656938-30)
[info] [2020-07-18 13:26:11,797] DEBUG Selector sun.nio.ch.EPollSelectorImpl@3c2dd5d9 waiting on select (org.eclipse.jetty.io.ManagedSelector:qtp1021656938-30)
[info] [2020-07-18 13:26:11,798] DEBUG Selector sun.nio.ch.EPollSelectorImpl@3c2dd5d9 woken up from select, 0/1 selected (org.eclipse.jetty.io.ManagedSelector:qtp1021656938-30)
[info] [2020-07-18 13:26:11,798] DEBUG Selector sun.nio.ch.EPollSelectorImpl@3c2dd5d9 processing 0 keys, 1 actions (org.eclipse.jetty.io.ManagedSelector:qtp1021656938-30)
[info] [2020-07-18 13:26:11,801] DEBUG action=org.eclipse.jetty.io.ManagedSelector$CreateEndPoint@1acde28a wakeup=false (org.eclipse.jetty.io.ManagedSelector:qtp1021656938-30)
[info] [2020-07-18 13:26:11,801] DEBUG EatWhatYouKill@2d2e8982/SelectorProducer@2b454e29/PRODUCING/ReservedThreadExecutor@4fbe007b{s=0/1,p=0}@SelectorManager@ServerConnector@2a2843ec{HTTP/1.1,[http/1.1]}{0.0.0.0:7070} t=org.eclipse.jetty.io.ManagedSelector$CreateEndPoint@1acde28a/BLOCKING (org.eclipse.jetty.util.thread.strategy.EatWhatYouKill:qtp1021656938-30)
[info] [2020-07-18 13:26:11,803] DEBUG Got ping response for sessionid: 0x1735d68ff190022 after 1ms (org.apache.zookeeper.ClientCnxn:main-SendThread(m01.local:2181))
[info] [2020-07-18 13:26:11,806] DEBUG ReservedThreadExecutor@4fbe007b{s=0/1,p=0}@SelectorManager@ServerConnector@2a2843ec{HTTP/1.1,[http/1.1]}{0.0.0.0:7070} tryExecute EatWhatYouKill@2d2e8982/SelectorProducer@2b454e29/PRODUCING/ReservedThreadExecutor@4fbe007b{s=0/1,p=0}@SelectorManager@ServerConnector@2a2843ec{HTTP/1.1,[http/1.1]}{0.0.0.0:7070} (org.eclipse.jetty.util.thread.ReservedThreadExecutor:qtp1021656938-30)
[info] [2020-07-18 13:26:11,808] DEBUG ReservedThreadExecutor@4fbe007b{s=0/1,p=1}@SelectorManager@ServerConnector@2a2843ec{HTTP/1.1,[http/1.1]}{0.0.0.0:7070} startReservedThread p=1 (org.eclipse.jetty.util.thread.ReservedThreadExecutor:qtp1021656938-30)
[info] [2020-07-18 13:26:11,811] DEBUG queue ReservedThreadExecutor@4fbe007b{s=0/1,p=1}@SelectorManager@ServerConnector@2a2843ec{HTTP/1.1,[http/1.1]}{0.0.0.0:7070}@74e52b3f (org.eclipse.jetty.util.thread.QueuedThreadPool:qtp1021656938-30)
[info] [2020-07-18 13:26:11,820] DEBUG run ReservedThreadExecutor@4fbe007b{s=0/1,p=1}@SelectorManager@ServerConnector@2a2843ec{HTTP/1.1,[http/1.1]}{0.0.0.0:7070}@74e52b3f (org.eclipse.jetty.util.thread.QueuedThreadPool:qtp1021656938-32)
[info] [2020-07-18 13:26:11,820] DEBUG ReservedThreadExecutor@4fbe007b{s=1/1,p=1}@SelectorManager@ServerConnector@2a2843ec{HTTP/1.1,[http/1.1]}{0.0.0.0:7070}@74e52b3f started (org.eclipse.jetty.util.thread.ReservedThreadExecutor:qtp1021656938-32)
[info] [2020-07-18 13:26:11,820] DEBUG ReservedThreadExecutor@4fbe007b{s=1/1,p=0}@SelectorManager@ServerConnector@2a2843ec{HTTP/1.1,[http/1.1]}{0.0.0.0:7070}@74e52b3f waiting (org.eclipse.jetty.util.thread.ReservedThreadExecutor:qtp1021656938-32)
[info] [2020-07-18 13:26:11,821] DEBUG EatWhatYouKill@2d2e8982/SelectorProducer@2b454e29/PRODUCING/ReservedThreadExecutor@4fbe007b{s=1/1,p=0}@SelectorManager@ServerConnector@2a2843ec{HTTP/1.1,[http/1.1]}{0.0.0.0:7070} PEC t=org.eclipse.jetty.io.ManagedSelector$CreateEndPoint@1acde28a (org.eclipse.jetty.util.thread.strategy.EatWhatYouKill:qtp1021656938-30)
[info] [2020-07-18 13:26:11,822] DEBUG queue org.eclipse.jetty.io.ManagedSelector$CreateEndPoint@1acde28a (org.eclipse.jetty.util.thread.QueuedThreadPool:qtp1021656938-30)
[info] [2020-07-18 13:26:11,825] DEBUG run org.eclipse.jetty.io.ManagedSelector$CreateEndPoint@1acde28a (org.eclipse.jetty.util.thread.QueuedThreadPool:qtp1021656938-33)
[info] [2020-07-18 13:26:11,829] DEBUG Forcing selection, actions=0 (org.eclipse.jetty.io.ManagedSelector:qtp1021656938-30)
[info] [2020-07-18 13:26:11,829] DEBUG action=null wakeup=false (org.eclipse.jetty.io.ManagedSelector:qtp1021656938-30)
[info] [2020-07-18 13:26:11,831] DEBUG Selector sun.nio.ch.EPollSelectorImpl@3c2dd5d9 waiting on select (org.eclipse.jetty.io.ManagedSelector:qtp1021656938-30)
[info] [2020-07-18 13:26:11,856] DEBUG SocketChannelEndPoint@423be9fa{/192.168.10.219:49734<->/192.168.10.22:7070,OPEN,fill=-,flush=-,to=24/30000}{io=0/0,kio=0,kro=0}->
f1-outsourcing
commented
4 years ago tracelog of 1.8 after deploying a task with 1.7
[info] Loading project definition from /home/software/marathon2/project/project
[info] Loading settings for project marathon2-build from plugins.sbt ... [info] Loading project definition from /home/software/marathon2/type-generator/project
[info] Loading settings for project root from build.sbt ... [info] Loading project definition from /home/software/marathon2/project
[info] Loading settings for project marathon from build.sbt ... [info] Set current project to marathon (in build file:/home/software/marathon2/)
[info] Running (fork) mesosphere.marathon.Main --master m01.local:5050 --zk zk://m01.local:2181/marathon --framework_name marathon --http_port 8080 --logging_level trace --hostname test2.local 2020-07-18 15:23:26,028] INFO Successfully loaded libmesos: version 1.10.0 (mesosphere.marathon.MarathonApp:main)
2020-07-18 15:23:26,957] WARN Specifying a Mesos Master connection without a protocol is deprecated and will likely be prohibited in the future. Please specify a protocol (http://, zk://, https://) (mesosphere.marathon.MarathonConf$:main)
2020-07-18 15:23:28,195] INFO Slf4jLogger started (akka.event.slf4j.Slf4jLogger:marathon-akka.actor.default-dispatcher-3)
2020-07-18 15:23:30,345] INFO Starting heartbeat actor (mesosphere.marathon.core.heartbeat.HeartbeatActor:marathon-akka.actor.default-dispatcher-3)
2020-07-18 15:23:30,784] INFO Running in ZooKeeper 3.4.x compatibility mode (org.apache.curator.utils.Compatibility:main)
2020-07-18 15:23:30,883] INFO Starting (org.apache.curator.framework.imps.CuratorFrameworkImpl:main)
2020-07-18 15:23:30,893] DEBUG Starting (org.apache.curator.CuratorZookeeperClient:main)
2020-07-18 15:23:30,894] DEBUG Starting (org.apache.curator.ConnectionState:main)
2020-07-18 15:23:30,895] DEBUG reset (org.apache.curator.ConnectionState:main)
2020-07-18 15:23:30,914] INFO Client environment:zookeeper.version=3.4.11-37e277162d567b55a07d1755f0b31c32e93c01a0, built on 11/01/2017 18:06 GMT (org.apache.zookeeper.ZooKeeper:main)
2020-07-18 15:23:30,915] INFO Client environment:host.name=test2.local (org.apache.zookeeper.ZooKeeper:main)
2020-07-18 15:23:30,917] INFO Client environment:java.version=1.8.0_252 (org.apache.zookeeper.ZooKeeper:main)
2020-07-18 15:23:30,918] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper:main)
2020-07-18 15:23:30,919] INFO Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.252.b09-2.el7_8.x86_64/jre (org.apache.zookeeper.ZooKeeper:main)
2020-07-18 15:23:30,920] INFO Client environment:java.class.path=/tmp/sbt_dc4cf587/job-1/target/9370c632/marathon_2.12-1.8.207.jar:/tmp/sbt_dc4cf587/target/fba81147/plugin-interface_2.12-1.8.207.jar:/tmp/sbt_dc4cf587/target/9ce04e34/commons-codec-1.9.jar:/tmp/sbt_dc4cf587/target/9879d6c4/jetty-http-9.4.8.v20171121.jar:/tmp/sbt_dc4cf587/target/4c192f4a/logstash-logback-encoder-4.9.jar:/tmp/sbt_dc4cf587/target/55762d31/audience-annotations-0.5.0.jar:/tmp/sbt_dc4cf587/target/c5a8eb12/scala-reflect.jar:/tmp/sbt_dc4cf587/target/187555de/jackson-module-jaxb-annotations-2.9.5.jar:/tmp/sbt_dc4cf587/target/fda78e1e/resetallattrs_2.12-1.0.0.jar:/tmp/sbt_dc4cf587/target/92c158af/jersey-server-2.27.jar:/tmp/sbt_dc4cf587/target/9ca7d55d/netty-3.10.5.Final.jar:/tmp/sbt_dc4cf587/target/8531ad5a/jaxb-api-2.3.1.jar:/tmp/sbt_dc4cf587/target/bbce493f/scala-parser-combinators_2.12-1.1.0.jar:/tmp/sbt_dc4cf587/target/e3016167/jackson-module-paranamer-2.9.5.jar:/tm!
p/sbt_dc4cf587/target/9e0c05fa/akka-stream-alpakka-s3_2.12-0.14.jar:/tmp/sbt_dc4cf587/target/6d25f55a/jackson-module-scala_2.12-2.9.5.jar:/tmp/sbt_dc4cf587/target/7d73a975/play-json_2.12-2.6.7.jar:/tmp/sbt_dc4cf587/target/ebd5d04f/jersey-media-multipart-2.27.jar:/tmp/sbt_dc4cf587/target/9e8c4db1/akka-actor_2.12-2.5.14.jar:/tmp/sbt_dc4cf587/target/7fd4e3d0/hk2-api-2.5.0-b42.jar:/tmp/sbt_dc4cf587/target/bbbb9b5d/jetty-servlet-9.4.8.v20171121.jar:/tmp/sbt_dc4cf587/target/b31526a2/httpcore-4.4.4.jar:/tmp/sbt_dc4cf587/target/6975da39/javax.inject-1.jar:/tmp/sbt_dc4cf587/target/a7976caa/raven-8.0.3.jar:/tmp/sbt_dc4cf587/target/81e24b17/jersey-container-servlet-core-2.27.jar:/tmp/sbt_dc4cf587/target/89507701/guava-20.0.jar:/tmp/sbt_dc4cf587/target/0100b353/mesos-1.8.0.jar:/tmp/sbt_dc4cf587/target/106f2237/jersey-media-jaxb-2.27.jar:/tmp/sbt_dc4cf587/target/9e1ac84e/HdrHistogram-2.1.10.jar:/tmp/sbt_dc4cf587/target/15c5e958/commons-compress-1.13.jar:/tmp/sbt_dc4cf587/target/d3fe2dfa!
/jetty-io-9.4.8.v20171121.jar:/tmp/sbt_dc4cf587/target/66dd6a6!
2/jersey-client-2.27.jar:/tmp/sbt_dc4cf587/target/c845703d/commons-beanutils-1.9.3.jar:/tmp/sbt_dc4cf587/target/e0db9e57/aws-java-sdk-core-1.11.243.jar:/tmp/sbt_dc4cf587/target/8ad72fe3/commons-collections-3.2.2.jar:/tmp/sbt_dc4cf587/target/fbb86b57/scallop_2.12-3.1.2.jar:/tmp/sbt_dc4cf587/target/aa97ee9f/jersey-container-servlet-2.27.jar:/tmp/sbt_dc4cf587/target/7e256d16/play-functional_2.12-2.6.7.jar:/tmp/sbt_dc4cf587/target/2f22c882/jul-to-slf4j-1.7.21.jar:/tmp/sbt_dc4cf587/target/4474ff89/metrics-healthchecks-4.0.2.jar:/tmp/sbt_dc4cf587/target/faddf054/metrics-jvm-4.0.2.jar:/tmp/sbt_dc4cf587/target/a41fe6c2/akka-http_2.12-10.0.11.jar:/tmp/sbt_dc4cf587/target/323964c3/reactive-streams-1.0.2.jar:/tmp/sbt_dc4cf587/target/99a18e9a/jline-0.9.94.jar:/tmp/sbt_dc4cf587/target/34614bd9/jetty-server-9.4.8.v20171121.jar:/tmp/sbt_dc4cf587/target/7c4f3c47/logback-classic-1.2.3.jar:/tmp/sbt_dc4cf587/target/18c09ad0/scala-library.jar:/tmp/sbt_dc4cf587/target/813b4007/scala-async_2.12-!
0.9.7.jar:/tmp/sbt_dc4cf587/target/e22de336/scala-xml_2.12-1.0.6.jar:/tmp/sbt_dc4cf587/target/a22ac510/jackson-core-2.9.5.jar:/tmp/sbt_dc4cf587/target/4b68c2d5/config-1.3.3.jar:/tmp/sbt_dc4cf587/target/9ea08f93/jersey-hk2-2.27.jar:/tmp/sbt_dc4cf587/target/0ffc2cd3/jackson-datatype-jdk8-2.8.9.jar:/tmp/sbt_dc4cf587/target/c5f3355a/curator-x-async-4.0.1.jar:/tmp/sbt_dc4cf587/target/a6982b28/metrics-jetty9-4.0.2.jar:/tmp/sbt_dc4cf587/target/815893df/commons-io-2.6.jar:/tmp/sbt_dc4cf587/target/9525fb6b/scala-java8-compat_2.12-0.9.0.jar:/tmp/sbt_dc4cf587/target/2973d150/junit-4.12.jar:/tmp/sbt_dc4cf587/target/d6ec1a16/jetty-util-9.4.8.v20171121.jar:/tmp/sbt_dc4cf587/target/149700de/akka-parsing_2.12-10.0.11.jar:/tmp/sbt_dc4cf587/target/e74beab6/aopalliance-repackaged-2.5.0-b42.jar:/tmp/sbt_dc4cf587/target/34b64138/jetty-continuation-9.4.8.v20171121.jar:/tmp/sbt_dc4cf587/target/f7b7f3d6/jetty-servlets-9.4.8.v20171121.jar:/tmp/sbt_dc4cf587/target/9802a9a8/akka-http-play-json_2.12-1!
.18.1.jar:/tmp/sbt_dc4cf587/target/9954f282/raven-logback-8.0.3.jar:/tm!
p/sbt_dc4cf587/target/41c913d7/mimepull-1.9.6.jar:/tmp/sbt_dc4cf587/target/138093a4/profiler-1.0.2.jar:/tmp/sbt_dc4cf587/target/44eaf099/javassist-3.22.0-CR2.jar:/tmp/sbt_dc4cf587/target/981f7696/akka-stream_2.12-2.5.14.jar:/tmp/sbt_dc4cf587/target/5613f735/akka-http-xml_2.12-10.0.9.jar:/tmp/sbt_dc4cf587/target/f20d4304/jersey-common-2.27.jar:/tmp/sbt_dc4cf587/target/a4833298/ui-1.3.1.jar:/tmp/sbt_dc4cf587/target/619eba74/paranamer-2.8.jar:/tmp/sbt_dc4cf587/target/ae83b2b7/java-uuid-generator-3.1.4.jar:/tmp/sbt_dc4cf587/target/85262acf/javax.activation-api-1.2.0.jar:/tmp/sbt_dc4cf587/target/ec987884/metrics-core-4.0.2.jar:/tmp/sbt_dc4cf587/target/ee9dacea/ion-java-1.0.2.jar:/tmp/sbt_dc4cf587/target/42a25dc3/hamcrest-core-1.3.jar:/tmp/sbt_dc4cf587/target/f7b520c4/joda-time-2.9.9.jar:/tmp/sbt_dc4cf587/target/8d6f47e7/curator-recipes-4.0.1.jar:/tmp/sbt_dc4cf587/target/060f4378/jackson-datatype-jsr310-2.8.9.jar:/tmp/sbt_dc4cf587/target/b0debcff/akka-stream-alpakka-simple-codecs!
_2.12-0.14.jar:/tmp/sbt_dc4cf587/target/9ec90e5c/akka-http-core_2.12-10.0.11.jar:/tmp/sbt_dc4cf587/target/3da85d2b/curator-framework-4.0.1.jar:/tmp/sbt_dc4cf587/target/eb02b345/metrics-servlets-4.0.2.jar:/tmp/sbt_dc4cf587/target/426a0862/javax.ws.rs-api-2.1.jar:/tmp/sbt_dc4cf587/target/1b74e211/metrics-annotation-4.0.2.jar:/tmp/sbt_dc4cf587/target/8613ae82/validation-api-1.1.0.Final.jar:/tmp/sbt_dc4cf587/target/dc767a0b/zookeeper-3.4.11.jar:/tmp/sbt_dc4cf587/target/e8350eec/jetty-security-9.4.8.v20171121.jar:/tmp/sbt_dc4cf587/target/869c59c0/jgrapht-core-0.9.3.jar:/tmp/sbt_dc4cf587/target/38269f29/jackson-jaxrs-base-2.9.5.jar:/tmp/sbt_dc4cf587/target/f49e7919/accord-api_2.12-0.7.1.jar:/tmp/sbt_dc4cf587/target/da76ca59/slf4j-api-1.7.25.jar:/tmp/sbt_dc4cf587/target/ba9e74b1/jackson-dataformat-cbor-2.6.7.jar:/tmp/sbt_dc4cf587/target/a1dc9750/scala-logging_2.12-3.7.2.jar:/tmp/sbt_dc4cf587/target/ed809d26/macro-compat_2.12-1.1.1.jar:/tmp/sbt_dc4cf587/target/7440a22c/api-console-!
3.0.8-accept.jar:/tmp/sbt_dc4cf587/target/9d6edc0c/hk2-locator-2.5.0-b4!
2.jar:/tmp/sbt_dc4cf587/target/0235ba8b/aopalliance-1.0.jar:/tmp/sbt_dc4cf587/target/34905083/jackson-databind-2.9.5.jar:/tmp/sbt_dc4cf587/target/86434440/logback-core-1.2.3.jar:/tmp/sbt_dc4cf587/target/7eefbf15/accord-core_2.12-0.7.1.jar:/tmp/sbt_dc4cf587/target/98e0b7dc/javax.inject-2.5.0-b42.jar:/tmp/sbt_dc4cf587/target/b42ebb11/metrics-jersey2-4.0.2.jar:/tmp/sbt_dc4cf587/target/e2e76f0f/curator-client-4.0.1.jar:/tmp/sbt_dc4cf587/target/102a344e/hk2-utils-2.5.0-b42.jar:/tmp/sbt_dc4cf587/target/ab9443bc/metrics-json-4.0.2.jar:/tmp/sbt_dc4cf587/target/934c04d3/javax.annotation-api-1.3.2.jar:/tmp/sbt_dc4cf587/target/9056ec9d/jackson-annotations-2.9.5.jar:/tmp/sbt_dc4cf587/target/057fe178/rolling-metrics-2.0.4.jar:/tmp/sbt_dc4cf587/target/38b14ff1/akka-slf4j_2.12-2.5.14.jar:/tmp/sbt_dc4cf587/target/200fb936/protobuf-java-3.5.0.jar:/tmp/sbt_dc4cf587/target/4ed2b2d4/osgi-resource-locator-1.0.1.jar:/tmp/sbt_dc4cf587/target/c80c1d72/jackson-jaxrs-json-provider-2.9.5.jar:/tmp/sbt!
_dc4cf587/target/4d35b3e5/akka-protobuf_2.12-2.5.14.jar:/tmp/sbt_dc4cf587/target/c1fdf6c0/ssl-config-core_2.12-0.2.3.jar:/tmp/sbt_dc4cf587/target/733db77a/httpclient-4.5.2.jar:/tmp/sbt_dc4cf587/target/3cd63d07/javax.servlet-api-3.1.0.jar:/tmp/sbt_dc4cf587/target/eeb69005/guice-4.1.0.jar (org.apache.zookeeper.ZooKeeper:main)
2020-07-18 15:23:30,924] INFO Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper:main)
2020-07-18 15:23:30,928] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper:main)
2020-07-18 15:23:30,928] INFO Client environment:java.compiler=
[[33mwarn] Canceling execution... [[31merror] I0718 15:24:56.504509 1575186 sched.cpp:2166] Asked to stop the driver [[31merror] I0718 15:24:56.504631 1575172 sched.cpp:1204] Stopping framework 5262ced9-70e2-4c0d-9064-ab4173118409-0000 2020-07-18 15:24:56,472] INFO Shutting down services (mesosphere.marathon.MarathonApp:shutdownHook3) [[31merror] Total time: 118 s, completed Jul 18, 2020 3:24:58 PM
f1-outsourcing
commented
4 years ago This is the v2/info request compared between 1.7.236 an 1.10.25
1.7.236
[info] [2020-07-18 23:00:22,571] DEBUG REQUEST for //test2.local:7071/v2/info on HttpChannelOverHttp@1eecfa5c{r=1,c=false,a=IDLE,uri=//test2.local:7071/v2/info}
[info] GET //test2.local:7071/v2/info HTTP/1.1
[info] Host: test2.local:7071
[info] User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0
[info] Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
[info] Accept-Language: en-US,en;q=0.5
[info] Accept-Encoding: gzip, deflate
[info] DNT: 1
[info] Connection: keep-alive
[info] Upgrade-Insecure-Requests: 1
[info] Cache-Control: max-age=0
[info] (org.eclipse.jetty.server.HttpChannel:qtp1640832113-32)
[info] [2020-07-18 23:00:22,571] DEBUG HttpChannelOverHttp@1eecfa5c{r=1,c=false,a=IDLE,uri=//test2.local:7071/v2/info} onContentComplete (org.eclipse.jetty.server.HttpChannel:qtp1640832113-32)
[info] [2020-07-18 23:00:22,571] DEBUG HttpChannelOverHttp@1eecfa5c{r=1,c=false,a=IDLE,uri=//test2.local:7071/v2/info} onRequestComplete (org.eclipse.jetty.server.HttpChannel:qtp1640832113-32)
[info] [2020-07-18 23:00:22,572] DEBUG HttpInputOverHTTP@7a398745[c=0,q=0,[0]=null,s=STREAM] addContent EOF (org.eclipse.jetty.server.HttpInput:qtp1640832113-32)
[info] [2020-07-18 23:00:22,573] DEBUG HttpConnection@df1bc7a[p=HttpParser{s=END,0 of -1},g=HttpGenerator@7caf188e{s=START}]=>HttpChannelOverHttp@1eecfa5c{r=1,c=false,a=IDLE,uri=//test2.local:7071/v2/info}<-SocketChannelEndPoint@3d798ac0{/192.168.10.219:57151<->/192.168.10.22:7071,OPEN,fill=-,flush=-,to=23/30000}{io=0/0,kio=0,kro=1}->HttpConnection@df1bc7a[p=HttpParser{s=END,0 of -1},g=HttpGenerator@7caf188e{s=START}]=>HttpChannelOverHttp@1eecfa5c{r=1,c=false,a=IDLE,uri=//test2.local:7071/v2/info} parsed true HttpParser{s=END,0 of -1} (org.eclipse.jetty.server.HttpConnection:qtp1640832113-32)
[info] [2020-07-18 23:00:22,573] DEBUG releaseRequestBuffer HttpConnection@df1bc7a[p=HttpParser{s=END,0 of -1},g=HttpGenerator@7caf188e{s=START}]=>HttpChannelOverHttp@1eecfa5c{r=1,c=false,a=IDLE,uri=//test2.local:7071/v2/info}<-SocketChannelEndPoint@3d798ac0{/192.168.10.219:57151<->/192.168.10.22:7071,OPEN,fill=-,flush=-,to=23/30000}{io=0/0,kio=0,kro=1}->HttpConnection@df1bc7a[p=HttpParser{s=END,0 of -1},g=HttpGenerator@7caf188e{s=START}]=>HttpChannelOverHttp@1eecfa5c{r=1,c=false,a=IDLE,uri=//test2.local:7071/v2/info} (org.eclipse.jetty.server.HttpConnection:qtp1640832113-32)
[info] [2020-07-18 23:00:22,573] DEBUG HttpChannelOverHttp@1eecfa5c{r=1,c=false,a=IDLE,uri=//test2.local:7071/v2/info} handle //test2.local:7071/v2/info (org.eclipse.jetty.server.HttpChannel:qtp1640832113-32)
[info] [2020-07-18 23:00:22,574] DEBUG handling HttpChannelState@6c10396e{s=IDLE a=NOT_ASYNC i=true r=IDLE w=false} (org.eclipse.jetty.server.HttpChannelState:qtp1640832113-32)
[info] [2020-07-18 23:00:22,574] DEBUG HttpChannelOverHttp@1eecfa5c{r=1,c=false,a=DISPATCHED,uri=//test2.local:7071/v2/info} action DISPATCH (org.eclipse.jetty.server.HttpChannel:qtp1640832113-32)
[info] [2020-07-18 23:00:22,574] DEBUG REQUEST GET /v2/info on HttpChannelOverHttp@1eecfa5c{r=1,c=false,a=DISPATCHED,uri=//test2.local:7071/v2/info} (org.eclipse.jetty.server.Server:qtp1640832113-32)
[info] [2020-07-18 23:00:22,574] DEBUG org.eclipse.jetty.server.handler.gzip.GzipHandler@3c87e6b7 handle Request(GET //test2.local:7071/v2/info)@5f20bc38 in null (org.eclipse.jetty.server.handler.gzip.GzipHandler:qtp1640832113-32)
[info] [2020-07-18 23:00:22,574] DEBUG scope null||/v2/info @ o.e.j.s.ServletContextHandler@b52b755{/,null,AVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:qtp1640832113-32)
[info] [2020-07-18 23:00:22,574] DEBUG context=||/v2/info @ o.e.j.s.ServletContextHandler@b52b755{/,null,AVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:qtp1640832113-32)
[info] [2020-07-18 23:00:22,574] DEBUG servlet ||/v2/info -> org.glassfish.jersey.servlet.ServletContainer-5e8507f1@9bff24ad==org.glassfish.jersey.servlet.ServletContainer,jsp=null,order=-1,inst=true (org.eclipse.jetty.servlet.ServletHandler:qtp1640832113-32)
[info] [2020-07-18 23:00:22,574] DEBUG chain=mesosphere.marathon.api.LeaderProxyFilter-41184371->mesosphere.marathon.api.LimitConcurrentRequestsFilter-608b1fd2->mesosphere.marathon.api.CORSFilter-56382bc9->mesosphere.marathon.api.CacheDisablingFilter-591a4f8e->org.glassfish.jersey.servlet.ServletContainer-5e8507f1@9bff24ad==org.glassfish.jersey.servlet.ServletContainer,jsp=null,order=-1,inst=true (org.eclipse.jetty.servlet.ServletHandler:qtp1640832113-32)
[info] [2020-07-18 23:00:22,575] DEBUG call filter mesosphere.marathon.api.LeaderProxyFilter-41184371 (org.eclipse.jetty.servlet.ServletHandler:qtp1640832113-32)
[info] [2020-07-18 23:00:22,575] DEBUG call filter mesosphere.marathon.api.LimitConcurrentRequestsFilter-608b1fd2 (org.eclipse.jetty.servlet.ServletHandler:qtp1640832113-32)
[info] [2020-07-18 23:00:22,575] DEBUG call filter mesosphere.marathon.api.CORSFilter-56382bc9 (org.eclipse.jetty.servlet.ServletHandler:qtp1640832113-32)
[info] [2020-07-18 23:00:22,575] DEBUG call filter mesosphere.marathon.api.CacheDisablingFilter-591a4f8e (org.eclipse.jetty.servlet.ServletHandler:qtp1640832113-32)
[info] [2020-07-18 23:00:22,575] DEBUG call servlet org.glassfish.jersey.servlet.ServletContainer-5e8507f1@9bff24ad==org.glassfish.jersey.servlet.ServletContainer,jsp=null,order=-1,inst=true (org.eclipse.jetty.servlet.ServletHandler:qtp1640832113-32)
[info] [2020-07-18 23:00:22,585] DEBUG startAsync HttpChannelState@6c10396e{s=DISPATCHED a=NOT_ASYNC i=true r=IDLE w=false} (org.eclipse.jetty.server.HttpChannelState:qtp1640832113-32)
[info] [2020-07-18 23:00:22,594] DEBUG handled=true async=true committed=false on HttpChannelOverHttp@1eecfa5c{r=1,c=false,a=DISPATCHED,uri=//test2.local:7071/v2/info} (org.eclipse.jetty.server.Server:qtp1640832113-32)
[info] [2020-07-18 23:00:22,595] DEBUG unhandle HttpChannelState@6c10396e{s=DISPATCHED a=STARTED i=true r=IDLE w=false} (org.eclipse.jetty.server.HttpChannelState:qtp1640832113-32)
[info] [2020-07-18 23:00:22,596] DEBUG HttpChannelOverHttp@1eecfa5c{r=1,c=false,a=ASYNC_WAIT,uri=//test2.local:7071/v2/info} action WAIT (org.eclipse.jetty.server.HttpChannel:qtp1640832113-32)
[info] [2020-07-18 23:00:22,597] DEBUG HttpChannelOverHttp@1eecfa5c{r=1,c=false,a=ASYNC_WAIT,uri=//test2.local:7071/v2/info} handle exit, result WAIT (org.eclipse.jetty.server.HttpChannel:qtp1640832113-32)
[info] [2020-07-18 23:00:22,598] DEBUG HttpConnection@df1bc7a[p=HttpParser{s=END,0 of -1},g=HttpGenerator@7caf188e{s=START}]=>HttpChannelOverHttp@1eecfa5c{r=1,c=false,a=ASYNC_WAIT,uri=//test2.local:7071/v2/info}<-SocketChannelEndPoint@3d798ac0{/192.168.10.219:57151<->/192.168.10.22:7071,OPEN,fill=-,flush=-,to=48/30000}{io=0/0,kio=0,kro=1}->HttpConnection@df1bc7a[p=HttpParser{s=END,0 of -1},g=HttpGenerator@7caf188e{s=START}]=>HttpChannelOverHttp@1eecfa5c{r=1,c=false,a=ASYNC_WAIT,uri=//test2.local:7071/v2/info} onFillable exit HttpChannelState@6c10396e{s=ASYNC_WAIT a=STARTED i=false r=IDLE w=false} null (org.eclipse.jetty.server.HttpConnection:qtp1640832113-32)
[info] [2020-07-18 23:00:22,598] DEBUG ReservedThreadExecutor@298e6c95{s=1/1,p=0}@SelectorManager@ServerConnector@75ae4a1f{HTTP/1.1,[http/1.1]}{0.0.0.0:7071}@40fe012 size 1 > capacity (org.eclipse.jetty.util.thread.ReservedThreadExecutor:qtp1640832113-32)
[info] [2020-07-18 23:00:22,598] DEBUG ran ReservedThreadExecutor@298e6c95{s=1/1,p=0}@SelectorManager@ServerConnector@75ae4a1f{HTTP/1.1,[http/1.1]}{0.0.0.0:7071}@40fe012 (org.eclipse.jetty.util.thread.QueuedThreadPool:qtp1640832113-32)
[info] [2020-07-18 23:00:22,600] DEBUG org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor@262a7e31 compressing java.util.zip.Deflater@5f15d197 (org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor:scala-execution-context-global-235)
[info] [2020-07-18 23:00:22,600] DEBUG sendResponse info=null content=HeapByteBuffer@2c81139b[p=0,l=755,c=32768,r=755]={<<<\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x00}UM\x8f\x9b0\x10...|\xBbM\xDf\xDf\xFf\x03\x07\xD7aL\x96\x06\x00\x00>>>\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00} complete=true committing=true callback=GzipBufferCB@7e0724e3 (org.eclipse.jetty.server.HttpChannel:scala-execution-context-global-235)
[info] [2020-07-18 23:00:22,600] DEBUG COMMIT for /v2/info on HttpChannelOverHttp@1eecfa5c{r=1,c=true,a=ASYNC_WAIT,uri=//test2.local:7071/v2/info}
[info] 200 OK HTTP/1.1
[info] Date: Sat, 18 Jul 2020 21:00:22 GMT
[info] X-Marathon-Leader: http://test2.local:7071
[info] X-Frame-Options: DENY
[info] X-XSS-Protection: 1; mode=block
[info] Cache-Control: no-cache, no-store, must-revalidate
[info] Pragma: no-cache
[info] Expires: 0
[info] Content-Type: application/json
[info] Vary: Accept-Encoding, User-Agent
[info] Content-Encoding: gzip
[info] (org.eclipse.jetty.server.HttpChannel:scala-execution-context-global-235)
[info] [2020-07-18 23:00:22,600] DEBUG org.eclipse.jetty.server.HttpConnection$SendCallback@721c56b[PROCESSING][i=HTTP/1.1{s=200,h=10,cl=-1},cb=org.eclipse.jetty.server.HttpChannel$CommitCallback@480011e0] generate: NEED_HEADER (null,[p=0,l=755,c=32768,r=755],true)@START (org.eclipse.jetty.server.HttpConnection:scala-execution-context-global-235)
[info] [2020-07-18 23:00:22,600] DEBUG generateHeaders HTTP/1.1{s=200,h=10,cl=-1} last=true content=HeapByteBuffer@2c81139b[p=0,l=755,c=32768,r=755]={<<<\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x00}UM\x8f\x9b0\x10...|\xBbM\xDf\xDf\xFf\x03\x07\xD7aL\x96\x06\x00\x00>>>\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00} (org.eclipse.jetty.http.HttpGenerator:scala-execution-context-global-235)
[info] [2020-07-18 23:00:22,600] DEBUG Date: Sat, 18 Jul 2020 21:00:22 GMT
[info] X-Marathon-Leader: http://test2.local:7071
[info] X-Frame-Options: DENY
[info] X-XSS-Protection: 1; mode=block
[info] Cache-Control: no-cache, no-store, must-revalidate
[info] Pragma: no-cache
[info1.10.25
[2020-07-18 21:40:15,433] DEBUG REQUEST for //test2.local:7070/v2/info on HttpChannelOverHttp@5cb30625{r=1,c=false,a=IDLE,uri=//test2.local:7070/v2/info}
GET //test2.local:7070/v2/info HTTP/1.1
Host: test2.local:7070
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
DNT: 1
Connection: keep-alive
Upgrade-Insecure-Requests: 1
(org.eclipse.jetty.server.HttpChannel:qtp2094310615-27)
[2020-07-18 21:40:15,436] DEBUG HttpChannelOverHttp@5cb30625{r=1,c=false,a=IDLE,uri=//test2.local:7070/v2/info} onContentComplete (org.eclipse.jetty.server.HttpChannel:qtp2094310615-27)
[2020-07-18 21:40:15,438] DEBUG HttpChannelOverHttp@5cb30625{r=1,c=false,a=IDLE,uri=//test2.local:7070/v2/info} onRequestComplete (org.eclipse.jetty.server.HttpChannel:qtp2094310615-27)
[2020-07-18 21:40:15,439] DEBUG HttpInputOverHTTP@653101ee[c=0,q=0,[0]=null,s=STREAM] addContent EOF (org.eclipse.jetty.server.HttpInput:qtp2094310615-27)
[2020-07-18 21:40:15,440] DEBUG HttpConnection@475cea3e[p=HttpParser{s=END,0 of -1},g=HttpGenerator@4d2dab43{s=START}]=>HttpChannelOverHttp@5cb30625{r=1,c=false,a=IDLE,uri=//test2.local:7070/v2/info}<-SocketChannelEndPoint@1afde438{/192.168.10.219:57323<->/192.168.10.22:7070,OPEN,fill=-,flush=-,to=44/30000}{io=0/0,kio=0,kro=1}->HttpConnection@475cea3e[p=HttpParser{s=END,0 of -1},g=HttpGenerator@4d2dab43{s=START}]=>HttpChannelOverHttp@5cb30625{r=1,c=false,a=IDLE,uri=//test2.local:7070/v2/info} parsed true HttpParser{s=END,0 of -1} (org.eclipse.jetty.server.HttpConnection:qtp2094310615-27)
[2020-07-18 21:40:15,441] DEBUG releaseRequestBuffer HttpConnection@475cea3e[p=HttpParser{s=END,0 of -1},g=HttpGenerator@4d2dab43{s=START}]=>HttpChannelOverHttp@5cb30625{r=1,c=false,a=IDLE,uri=//test2.local:7070/v2/info}<-SocketChannelEndPoint@1afde438{/192.168.10.219:57323<->/192.168.10.22:7070,OPEN,fill=-,flush=-,to=45/30000}{io=0/0,kio=0,kro=1}->HttpConnection@475cea3e[p=HttpParser{s=END,0 of -1},g=HttpGenerator@4d2dab43{s=START}]=>HttpChannelOverHttp@5cb30625{r=1,c=false,a=IDLE,uri=//test2.local:7070/v2/info} (org.eclipse.jetty.server.HttpConnection:qtp2094310615-27)
[2020-07-18 21:40:15,442] DEBUG HttpChannelOverHttp@5cb30625{r=1,c=false,a=IDLE,uri=//test2.local:7070/v2/info} handle //test2.local:7070/v2/info (org.eclipse.jetty.server.HttpChannel:qtp2094310615-27)
[2020-07-18 21:40:15,443] DEBUG handling HttpChannelState@29476a6{s=IDLE a=NOT_ASYNC i=true r=IDLE w=false} (org.eclipse.jetty.server.HttpChannelState:qtp2094310615-27)
[2020-07-18 21:40:15,445] DEBUG HttpChannelOverHttp@5cb30625{r=1,c=false,a=DISPATCHED,uri=//test2.local:7070/v2/info} action DISPATCH (org.eclipse.jetty.server.HttpChannel:qtp2094310615-27)
[2020-07-18 21:40:15,454] DEBUG REQUEST GET /v2/info on HttpChannelOverHttp@5cb30625{r=1,c=false,a=DISPATCHED,uri=//test2.local:7070/v2/info} (org.eclipse.jetty.server.Server:qtp2094310615-27)
[2020-07-18 21:40:15,455] DEBUG org.eclipse.jetty.server.handler.gzip.GzipHandler@2d9dff65 handle Request(GET //test2.local:7070/v2/info)@5ea971e5 in null (org.eclipse.jetty.server.handler.gzip.GzipHandler:qtp2094310615-27)
[2020-07-18 21:40:15,457] DEBUG scope null||/v2/info @ o.e.j.s.ServletContextHandler@6920614{/,null,AVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:qtp2094310615-27)
[2020-07-18 21:40:15,459] DEBUG context=||/v2/info @ o.e.j.s.ServletContextHandler@6920614{/,null,AVAILABLE} (org.eclipse.jetty.server.handler.ContextHandler:qtp2094310615-27)
[2020-07-18 21:40:15,460] DEBUG servlet ||/v2/info -> org.glassfish.jersey.servlet.ServletContainer-5d1e0fbb@5dcc1567==org.glassfish.jersey.servlet.ServletContainer,jsp=null,order=-1,inst=true (org.eclipse.jetty.servlet.ServletHandler:qtp2094310615-27)
[2020-07-18 21:40:15,469] DEBUG chain=mesosphere.marathon.api.LeaderProxyFilter-329548d0->mesosphere.marathon.api.LimitConcurrentRequestsFilter-49d30c6f->mesosphere.marathon.api.CORSFilter-1115433e->mesosphere.marathon.api.CacheDisablingFilter-4fdca00a->org.glassfish.jersey.servlet.ServletContainer-5d1e0fbb@5dcc1567==org.glassfish.jersey.servlet.ServletContainer,jsp=null,order=-1,inst=true (org.eclipse.jetty.servlet.ServletHandler:qtp2094310615-27)
[2020-07-18 21:40:15,473] DEBUG call filter mesosphere.marathon.api.LeaderProxyFilter-329548d0 (org.eclipse.jetty.servlet.ServletHandler:qtp2094310615-27)
[2020-07-18 21:40:15,479] DEBUG call filter mesosphere.marathon.api.LimitConcurrentRequestsFilter-49d30c6f (org.eclipse.jetty.servlet.ServletHandler:qtp2094310615-27)
[2020-07-18 21:40:15,480] DEBUG call filter mesosphere.marathon.api.CORSFilter-1115433e (org.eclipse.jetty.servlet.ServletHandler:qtp2094310615-27)
[2020-07-18 21:40:15,490] DEBUG call filter mesosphere.marathon.api.CacheDisablingFilter-4fdca00a (org.eclipse.jetty.servlet.ServletHandler:qtp2094310615-27)
[2020-07-18 21:40:15,491] DEBUG call servlet org.glassfish.jersey.servlet.ServletContainer-5d1e0fbb@5dcc1567==org.glassfish.jersey.servlet.ServletContainer,jsp=null,order=-1,inst=true (org.eclipse.jetty.servlet.ServletHandler:qtp2094310615-27)
[2020-07-18 21:40:15,551] DEBUG Checking for overdue tasks (mesosphere.marathon.core.task.jobs.impl.OverdueInstancesActor$Support:marathon-akka.actor.default-dispatcher-8)
[2020-07-18 21:40:15,606] DEBUG Got ping response for sessionid: 0x17362f333900011 after 1ms (org.apache.zookeeper.ClientCnxn:main-SendThread(m01.local:2181))
[2020-07-18 21:40:15,648] DEBUG startAsync HttpChannelState@29476a6{s=DISPATCHED a=NOT_ASYNC i=true r=IDLE w=false} (org.eclipse.jetty.server.HttpChannelState:qtp2094310615-27)
[2020-07-18 21:40:15,696] DEBUG handled=true async=true committed=false on HttpChannelOverHttp@5cb30625{r=1,c=false,a=DISPATCHED,uri=//test2.local:7070/v2/info} (org.eclipse.jetty.server.Server:qtp2094310615-27)
[2020-07-18 21:40:15,697] DEBUG unhandle HttpChannelState@29476a6{s=DISPATCHED a=STARTED i=true r=IDLE w=false} (org.eclipse.jetty.server.HttpChannelState:qtp2094310615-27)
[2020-07-18 21:40:15,698] DEBUG HttpChannelOverHttp@5cb30625{r=1,c=false,a=ASYNC_WAIT,uri=//test2.local:7070/v2/info} action WAIT (org.eclipse.jetty.server.HttpChannel:qtp2094310615-27)
[2020-07-18 21:40:15,699] DEBUG HttpChannelOverHttp@5cb30625{r=1,c=false,a=ASYNC_WAIT,uri=//test2.local:7070/v2/info} handle exit, result WAIT (org.eclipse.jetty.server.HttpChannel:qtp2094310615-27)
[2020-07-18 21:40:15,701] DEBUG HttpConnection@475cea3e[p=HttpParser{s=END,0 of -1},g=HttpGenerator@4d2dab43{s=START}]=>HttpChannelOverHttp@5cb30625{r=1,c=false,a=ASYNC_WAIT,uri=//test2.local:7070/v2/info}<-SocketChannelEndPoint@1afde438{/192.168.10.219:57323<->/192.168.10.22:7070,OPEN,fill=-,flush=-,to=305/30000}{io=0/0,kio=0,kro=1}->HttpConnection@475cea3e[p=HttpParser{s=END,0 of -1},g=HttpGenerator@4d2dab43{s=START}]=>HttpChannelOverHttp@5cb30625{r=1,c=false,a=ASYNC_WAIT,uri=//test2.local:7070/v2/info} onFillable exit HttpChannelState@29476a6{s=ASYNC_WAIT a=STARTED i=false r=IDLE w=false} null (org.eclipse.jetty.server.HttpConnection:qtp2094310615-27)
[2020-07-18 21:40:15,701] DEBUG ran org.eclipse.jetty.io.ManagedSelector$$Lambda$1423/1947349460@43b0a008 (org.eclipse.jetty.util.thread.QueuedThreadPool:qtp2094310615-27)
[2020-07-18 21:40:16,550] DEBUG Current State: LaunchTokens:100 OffersWanted:false Matchers:0 OfferQueues:0 UnprocessedOffers:0 (mesosphere.marathon.core.matcher.manager.impl.OfferMatcherManagerActor:marathon-akka.actor.default-dispatcher-3)
[2020-07-18 21:40:16,550] DEBUG Reject Elapsed offers. Unprocessed: 0 Overdue:0 (mesosphere.marathon.core.matcher.manager.impl.OfferMatcherManagerActor:marathon-akka.actor.default-dispatcher-3)
[2020-07-18 21:40:18,941] DEBUG Got ping response for sessionid: 0x17362f333900011 after 1ms (org.apache.zookeeper.ClientCnxn:main-SendThread(m01.local:2181))
[2020-07-18 21:40:19,551] DEBUG Current State: LaunchTokens:100 OffersWanted:false Matchers:0 OfferQueues:0 UnprocessedOffers:0 (mesosphere.marathon.core.matcher.manager.impl.OfferMatcherManagerActor:marathon-akka.actor.default-dispatcher-4)
[2020-07-18 21:40:19,551] DEBUG Reject Elapsed offers. Unprocessed: 0 Overdue:0 (mesosphere.marathon.core.matcher.manager.impl.OfferMatcherManagerActor:marathon-akka.actor.default-dispatcher-4)
[2020-07-18 21:40:20,560] DEBUG Checking for overdue tasks (mesosphere.marathon.core.task.jobs.impl.OverdueInstancesActor$Support:marathon-akka.actor.default-dispatcher-3)
[2020-07-18 21:40:22,276] DEBUG Got ping response for sessionid: 0x17362f333900011 after 1ms (org.apache.zookeeper.ClientCnxn:main-SendThread(m01.local:2181))1.10.25 the request ends with these lines
[2020-07-18 21:40:15,701] DEBUG HttpConnection@475cea3e[p=HttpParser{s=END,0 of -1},g=HttpGenerator@4d2dab43{s=START}]=>HttpChannelOverHttp@5cb30625{r=1,c=false,a=ASYNC_WAIT,uri=//test2.local:7070/v2/info}<-SocketChannelEndPoint@1afde438{/192.168.10.219:57323<->/192.168.10.22:7070,OPEN,fill=-,flush=-,to=305/30000}{io=0/0,kio=0,kro=1}->HttpConnection@475cea3e[p=HttpParser{s=END,0 of -1},g=HttpGenerator@4d2dab43{s=START}]=>HttpChannelOverHttp@5cb30625{r=1,c=false,a=ASYNC_WAIT,uri=//test2.local:7070/v2/info} onFillable exit HttpChannelState@29476a6{s=ASYNC_WAIT a=STARTED i=false r=IDLE w=false} null (org.eclipse.jetty.server.HttpConnection:qtp2094310615-27)
[2020-07-18 21:40:15,701] DEBUG ran org.eclipse.jetty.io.ManagedSelector$$Lambda$1423/1947349460@43b0a008 (org.eclipse.jetty.util.thread.QueuedThreadPool:qtp2094310615-27)where 1.7.236 continues like this
[info] [2020-07-18 23:00:22,598] DEBUG HttpConnection@df1bc7a[p=HttpParser{s=END,0 of -1},g=HttpGenerator@7caf188e{s=START}]=>HttpChannelOverHttp@1eecfa5c{r=1,c=false,a=ASYNC_WAIT,uri=//test2.local:7071/v2/info}<-SocketChannelEndPoint@3d798ac0{/192.168.10.219:57151<->/192.168.10.22:7071,OPEN,fill=-,flush=-,to=48/30000}{io=0/0,kio=0,kro=1}->HttpConnection@df1bc7a[p=HttpParser{s=END,0 of -1},g=HttpGenerator@7caf188e{s=START}]=>HttpChannelOverHttp@1eecfa5c{r=1,c=false,a=ASYNC_WAIT,uri=//test2.local:7071/v2/info} onFillable exit HttpChannelState@6c10396e{s=ASYNC_WAIT a=STARTED i=false r=IDLE w=false} null (org.eclipse.jetty.server.HttpConnection:qtp1640832113-32)
[info] [2020-07-18 23:00:22,598] DEBUG ReservedThreadExecutor@298e6c95{s=1/1,p=0}@SelectorManager@ServerConnector@75ae4a1f{HTTP/1.1,[http/1.1]}{0.0.0.0:7071}@40fe012 size 1 > capacity (org.eclipse.jetty.util.thread.ReservedThreadExecutor:qtp1640832113-32)
[info] [2020-07-18 23:00:22,598] DEBUG ran ReservedThreadExecutor@298e6c95{s=1/1,p=0}@SelectorManager@ServerConnector@75ae4a1f{HTTP/1.1,[http/1.1]}{0.0.0.0:7071}@40fe012 (org.eclipse.jetty.util.thread.QueuedThreadPool:qtp1640832113-32)
[info] [2020-07-18 23:00:22,600] DEBUG org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor@262a7e31 compressing java.util.zip.Deflater@5f15d197 (org.eclipse.jetty.server.handler.gzip.GzipHttpOutputInterceptor:scala-execution-context-global-235)
[info] [2020-07-18 23:00:22,600] DEBUG sendResponse info=null content=HeapByteBuffer@2c81139b[p=0,l=755,c=32768,r=755]={<<<\x1f\x8b\x08\x00\x00\x00\x00\x00\x00\x00}UM\x8f\x9b0\x10...|\xBbM\xDf\xDf\xFf\x03\x07\xD7aL\x96\x06\x00\x00>>>\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00} complete=true committing=true callback=GzipBufferCB@7e0724e3 (org.eclipse.jetty.server.HttpChannel:scala-execution-context-global-235)
[info] [2020-07-18 23:00:22,600] DEBUG COMMIT for /v2/info on HttpChannelOverHttp@1eecfa5c{r=1,c=true,a=ASYNC_WAIT,uri=//test2.local:7071/v2/info}
[info] 200 OK HTTP/1.1
[info] Date: Sat, 18 Jul 2020 21:00:22 GMT
[info] X-Marathon-Leader: http://test2.local:7071
[info] X-Frame-Options: DENY
[info] X-XSS-Protection: 1; mode=block
[info] Cache-Control: no-cache, no-store, must-revalidate
[info] Pragma: no-cache
[info] Expires: 0
[info] Content-Type: application/json
[info] Vary: Accept-Encoding, User-Agent
[info] Content-Encoding: gzip
I have a new install of marathon/mesos/zookeeper on centos 7. I am using the RPMs (1.9.109). Everything fires up OK, but when I go to the marathon interface, it spins with Loading Applications ... and never finishes. The only thing I can find in the logs is this on the master:
INFO Found no roles suitable for revive repetition. (mesosphere.marathon.core.launchqueue.impl.ReviveOffersStreamLogic$ReviveRepeaterLogic:marathon-akka.actor.default-dispatcher-3) Feb 08 10:35:25 nj-dcos01-cl01 marathon[581]: [2020-02-08 10:35:25,513] INFO Found no roles suitable for revive repetition. (mesosphere.marathon.core.launchqueue.impl.ReviveOffersStreamLogic$ReviveRepeaterLogic:marathon-akka.actor.default-dispatcher-2) Feb 08 10:35:30 nj-dcos01-cl01 marathon[581]: [2020-02-08 10:35:30,513] INFO Found no roles suitable for revive repetition. (mesosphere.marathon.core.launchqueue.impl.ReviveOffersStreamLogic$ReviveRepeaterLogic:marathon-akka.actor.default-dispatcher-7) Feb 08 10:35:32 nj-dcos01-cl01 marathon[581]: [2020-02-08 10:35:32,723] INFO Prompting Mesos for a heartbeat via explicit task reconciliation (mesosphere.marathon.core.heartbeat.MesosHeartbeatMonitor$$anon$1:marathon-akka.actor.default-dispatcher-8) Feb 08 10:35:32 nj-dcos01-cl01 marathon[581]: [2020-02-08 10:35:32,726] INFO Received fake heartbeat task-status update (mesosphere.marathon.core.heartbeat.MesosHeartbeatMonitor:Thread-172)
Firewall/iptables is off. I can't submit a test job to marathon or create an app. Mesos seems to be working OK. sean