Closed F21 closed 8 years ago
hi @F21, a couple of things here:
Hey @elingg
stdout
of the failing JN launch:I0909 00:17:01.210819 1271 logging.cpp:172] INFO level logging started!
I0909 00:17:01.211256 1271 fetcher.cpp:409] Fetcher Info: {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/20150908-235003-169978048-5050-572-S2\/root","items":[{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"http:\/\/192.168.33.13:31795\/hdfs-mesos-executor-0.1.3.tgz"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"http:\/\/192.168.33.13:31795\/hdfs-site.xml"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"https:\/\/downloads.mesosphere.io\/java\/jre-7u76-linux-x64.tar.gz"}}],"sandbox_directory":"\/tmp\/mesos\/slaves\/20150908-235003-169978048-5050-572-S2\/frameworks\/20150908-235003-169978048-5050-572-0000\/executors\/executor.journalnode.NodeExecutor.1441757820880\/runs\/7d7635ba-e2f5-44c1-8991-3410e35eb245","user":"root"}
I0909 00:17:01.220074 1271 fetcher.cpp:364] Fetching URI 'http://192.168.33.13:31795/hdfs-mesos-executor-0.1.3.tgz'
I0909 00:17:01.220160 1271 fetcher.cpp:238] Fetching directly into the sandbox directory
I0909 00:17:01.220212 1271 fetcher.cpp:176] Fetching URI 'http://192.168.33.13:31795/hdfs-mesos-executor-0.1.3.tgz'
I0909 00:17:01.220265 1271 fetcher.cpp:126] Downloading resource from 'http://192.168.33.13:31795/hdfs-mesos-executor-0.1.3.tgz' to '/tmp/mesos/slaves/20150908-235003-169978048-5050-572-S2/frameworks/20150908-235003-169978048-5050-572-0000/executors/executor.journalnode.NodeExecutor.1441757820880/runs/7d7635ba-e2f5-44c1-8991-3410e35eb245/hdfs-mesos-executor-0.1.3.tgz'
I0909 00:17:04.645876 1271 fetcher.cpp:76] Extracting with command: tar -C '/tmp/mesos/slaves/20150908-235003-169978048-5050-572-S2/frameworks/20150908-235003-169978048-5050-572-0000/executors/executor.journalnode.NodeExecutor.1441757820880/runs/7d7635ba-e2f5-44c1-8991-3410e35eb245' -xf '/tmp/mesos/slaves/20150908-235003-169978048-5050-572-S2/frameworks/20150908-235003-169978048-5050-572-0000/executors/executor.journalnode.NodeExecutor.1441757820880/runs/7d7635ba-e2f5-44c1-8991-3410e35eb245/hdfs-mesos-executor-0.1.3.tgz'
I0909 00:17:05.688472 1271 fetcher.cpp:84] Extracted '/tmp/mesos/slaves/20150908-235003-169978048-5050-572-S2/frameworks/20150908-235003-169978048-5050-572-0000/executors/executor.journalnode.NodeExecutor.1441757820880/runs/7d7635ba-e2f5-44c1-8991-3410e35eb245/hdfs-mesos-executor-0.1.3.tgz' into '/tmp/mesos/slaves/20150908-235003-169978048-5050-572-S2/frameworks/20150908-235003-169978048-5050-572-0000/executors/executor.journalnode.NodeExecutor.1441757820880/runs/7d7635ba-e2f5-44c1-8991-3410e35eb245'
W0909 00:17:05.688606 1271 fetcher.cpp:260] Copying instead of extracting resource from URI with 'extract' flag, because it does not seem to be an archive: http://192.168.33.13:31795/hdfs-mesos-executor-0.1.3.tgz
I0909 00:17:05.688980 1271 fetcher.cpp:441] Fetched 'http://192.168.33.13:31795/hdfs-mesos-executor-0.1.3.tgz' to '/tmp/mesos/slaves/20150908-235003-169978048-5050-572-S2/frameworks/20150908-235003-169978048-5050-572-0000/executors/executor.journalnode.NodeExecutor.1441757820880/runs/7d7635ba-e2f5-44c1-8991-3410e35eb245/hdfs-mesos-executor-0.1.3.tgz'
I0909 00:17:05.689014 1271 fetcher.cpp:364] Fetching URI 'http://192.168.33.13:31795/hdfs-site.xml'
I0909 00:17:05.689040 1271 fetcher.cpp:238] Fetching directly into the sandbox directory
I0909 00:17:05.689079 1271 fetcher.cpp:176] Fetching URI 'http://192.168.33.13:31795/hdfs-site.xml'
I0909 00:17:05.689121 1271 fetcher.cpp:126] Downloading resource from 'http://192.168.33.13:31795/hdfs-site.xml' to '/tmp/mesos/slaves/20150908-235003-169978048-5050-572-S2/frameworks/20150908-235003-169978048-5050-572-0000/executors/executor.journalnode.NodeExecutor.1441757820880/runs/7d7635ba-e2f5-44c1-8991-3410e35eb245/hdfs-site.xml'
W0909 00:17:05.717823 1271 fetcher.cpp:260] Copying instead of extracting resource from URI with 'extract' flag, because it does not seem to be an archive: http://192.168.33.13:31795/hdfs-site.xml
I0909 00:17:05.718186 1271 fetcher.cpp:441] Fetched 'http://192.168.33.13:31795/hdfs-site.xml' to '/tmp/mesos/slaves/20150908-235003-169978048-5050-572-S2/frameworks/20150908-235003-169978048-5050-572-0000/executors/executor.journalnode.NodeExecutor.1441757820880/runs/7d7635ba-e2f5-44c1-8991-3410e35eb245/hdfs-site.xml'
I0909 00:17:05.718291 1271 fetcher.cpp:364] Fetching URI 'https://downloads.mesosphere.io/java/jre-7u76-linux-x64.tar.gz'
I0909 00:17:05.718391 1271 fetcher.cpp:238] Fetching directly into the sandbox directory
I0909 00:17:05.718444 1271 fetcher.cpp:176] Fetching URI 'https://downloads.mesosphere.io/java/jre-7u76-linux-x64.tar.gz'
I0909 00:17:05.718490 1271 fetcher.cpp:126] Downloading resource from 'https://downloads.mesosphere.io/java/jre-7u76-linux-x64.tar.gz' to '/tmp/mesos/slaves/20150908-235003-169978048-5050-572-S2/frameworks/20150908-235003-169978048-5050-572-0000/executors/executor.journalnode.NodeExecutor.1441757820880/runs/7d7635ba-e2f5-44c1-8991-3410e35eb245/jre-7u76-linux-x64.tar.gz'
I0909 00:18:46.927767 1271 fetcher.cpp:76] Extracting with command: tar -C '/tmp/mesos/slaves/20150908-235003-169978048-5050-572-S2/frameworks/20150908-235003-169978048-5050-572-0000/executors/executor.journalnode.NodeExecutor.1441757820880/runs/7d7635ba-e2f5-44c1-8991-3410e35eb245' -xf '/tmp/mesos/slaves/20150908-235003-169978048-5050-572-S2/frameworks/20150908-235003-169978048-5050-572-0000/executors/executor.journalnode.NodeExecutor.1441757820880/runs/7d7635ba-e2f5-44c1-8991-3410e35eb245/jre-7u76-linux-x64.tar.gz'
I0909 00:18:48.727993 1271 fetcher.cpp:84] Extracted '/tmp/mesos/slaves/20150908-235003-169978048-5050-572-S2/frameworks/20150908-235003-169978048-5050-572-0000/executors/executor.journalnode.NodeExecutor.1441757820880/runs/7d7635ba-e2f5-44c1-8991-3410e35eb245/jre-7u76-linux-x64.tar.gz' into '/tmp/mesos/slaves/20150908-235003-169978048-5050-572-S2/frameworks/20150908-235003-169978048-5050-572-0000/executors/executor.journalnode.NodeExecutor.1441757820880/runs/7d7635ba-e2f5-44c1-8991-3410e35eb245'
W0909 00:18:48.728261 1271 fetcher.cpp:260] Copying instead of extracting resource from URI with 'extract' flag, because it does not seem to be an archive: https://downloads.mesosphere.io/java/jre-7u76-linux-x64.tar.gz
I0909 00:18:48.728324 1271 fetcher.cpp:441] Fetched 'https://downloads.mesosphere.io/java/jre-7u76-linux-x64.tar.gz' to '/tmp/mesos/slaves/20150908-235003-169978048-5050-572-S2/frameworks/20150908-235003-169978048-5050-572-0000/executors/executor.journalnode.NodeExecutor.1441757820880/runs/7d7635ba-e2f5-44c1-8991-3410e35eb245/jre-7u76-linux-x64.tar.gz'
The stderr
is empty.
hm, taking a second look, it seems you have the oversubscription module running and it is affecting your containerizer. I've seen these issues with Mesos modules. As you can see below it's giving the unknown container error and abnormal executor error. You should see this error launching various types of tasks including with other frameworks.
I see this in the scheduler log: state=TASK_LOST message='Abnormal executor termination
I see this in the agent log: Querying resource estimator for oversubscribable resources Sep 08 00:50:09 core-04 mesos-slave[1271]: E0908 00:50:09.389519 1278 slave.cpp:3258] Container '126df935-512d-4e4f-b03b-7203f8ff6f75' for executor 'executor.journalnode.NodeExecutor.1441673294675' of framework '20150908-001609-169978048-5050-1255-0001' failed to start: Container destroyed during launch Sep 08 00:50:09 core-04 mesos-slave[1271]: E0908 00:50:09.389919 1278 slave.cpp:3340] Termination of executor 'executor.journalnode.NodeExecutor.1441673294675' of framework '20150908-001609-169978048-5050-1255-0001' failed: Unknown container: 126df935-512d-4e4f-b03b-7203f8ff6f75
I couldn't find a way to disable the oversubscription module, so I built binaries for mesos 0.22.1. I think oversubscription was a new feature in 0.23.
However, once that's done, if I use marathon, the scheduler kept getting killed because it finished downloading the mesos-hdfs framework.
I ended up manually copying mteos-hdfs framework to a node an running it.
Having done that, I am still see JN1 getting killed.
This is the log from the scheduler:
2015-09-10 04:28:41,658:3965(0x7f2f14f72700):ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.5
2015-09-10 04:28:41,658:3965(0x7f2f14f72700):ZOO_INFO@log_env@716: Client environment:host.name=core-02
2015-09-10 04:28:41,658:3965(0x7f2f14f72700):ZOO_INFO@log_env@723: Client environment:os.name=Linux
2015-09-10 04:28:41,658:3965(0x7f2f14f72700):ZOO_INFO@log_env@724: Client environment:os.arch=4.1.6-coreos-r1
2015-09-10 04:28:41,658:3965(0x7f2f14f72700):ZOO_INFO@log_env@725: Client environment:os.version=#2 SMP Thu Sep 3 01:30:12 UTC 2015
2015-09-10 04:28:41,661:3965(0x7f2f14f72700):ZOO_INFO@log_env@733: Client environment:user.name=core
2015-09-10 04:28:41,661:3965(0x7f2f14f72700):ZOO_INFO@log_env@741: Client environment:user.home=/home/core
2015-09-10 04:28:41,661:3965(0x7f2f14f72700):ZOO_INFO@log_env@753: Client environment:user.dir=/tmp/hdfs-mesos-0.1.3
2015-09-10 04:28:41,661:3965(0x7f2f14f72700):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=master.mesos:2181 sessionTimeout=20000 watcher=0x7f2f0666ff50 sessionId=0 sessionPasswd=<null> context=0x7f2ef4001910 flags=0
2015-09-10 04:28:41,669:3965(0x7f2ef3fff700):ZOO_INFO@check_events@1703: initiated connection to server [192.168.33.10:2181]
2015-09-10 04:28:41,709:3965(0x7f2ef3fff700):ZOO_INFO@check_events@1750: session establishment complete on server [192.168.33.10:2181], sessionId=0x14fb5018f750003, negotiated timeout=20000
04:28:41.801 [HdfsScheduler] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Registering without authentication
2015-09-10 04:28:41,817:3965(0x7f2f15f74700):ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.5
2015-09-10 04:28:41,817:3965(0x7f2f15f74700):ZOO_INFO@log_env@716: Client environment:host.name=core-02
04:28:41.817 [main] INFO org.eclipse.jetty.util.log - Logging initialized @1081ms
2015-09-10 04:28:41,818:3965(0x7f2f15f74700):ZOO_INFO@log_env@723: Client environment:os.name=Linux
2015-09-10 04:28:41,818:3965(0x7f2f15f74700):ZOO_INFO@log_env@724: Client environment:os.arch=4.1.6-coreos-r1
2015-09-10 04:28:41,818:3965(0x7f2f15f74700):ZOO_INFO@log_env@725: Client environment:os.version=#2 SMP Thu Sep 3 01:30:12 UTC 2015
I0910 04:28:41.818083 3992 sched.cpp:157] Version: 0.22.1
2015-09-10 04:28:41,827:3965(0x7f2f15f74700):ZOO_INFO@log_env@733: Client environment:user.name=core
2015-09-10 04:28:41,827:3965(0x7f2f15f74700):ZOO_INFO@log_env@741: Client environment:user.home=/home/core
2015-09-10 04:28:41,827:3965(0x7f2f15f74700):ZOO_INFO@log_env@753: Client environment:user.dir=/tmp/hdfs-mesos-0.1.3
2015-09-10 04:28:41,828:3965(0x7f2f15f74700):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=master.mesos:2181 sessionTimeout=10000 watcher=0x7f2f0666ff50 sessionId=0 sessionPasswd=<null> context=0x7f2eec002550 flags=0
2015-09-10 04:28:41,837:3965(0x7f2ef2ffd700):ZOO_INFO@check_events@1703: initiated connection to server [192.168.33.10:2181]
2015-09-10 04:28:41,843:3965(0x7f2ef2ffd700):ZOO_INFO@check_events@1750: session establishment complete on server [192.168.33.10:2181], sessionId=0x14fb5018f750004, negotiated timeout=10000
I0910 04:28:41.844611 3983 group.cpp:313] Group process (group(1)@192.168.33.11:60961) connected to ZooKeeper
I0910 04:28:41.844859 3983 group.cpp:790] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
I0910 04:28:41.845044 3983 group.cpp:385] Trying to create path '/mesos' in ZooKeeper
I0910 04:28:41.871109 3983 detector.cpp:138] Detected a new leader: (id='1')
I0910 04:28:41.872189 3983 group.cpp:659] Trying to get '/mesos/info_0000000001' in ZooKeeper
I0910 04:28:41.879436 3983 detector.cpp:452] A new leading master (UPID=master@192.168.33.10:5050) is detected
I0910 04:28:41.880085 3983 sched.cpp:254] New master detected at master@192.168.33.10:5050
04:28:41.880 [main] INFO org.eclipse.jetty.server.Server - jetty-9.2.z-SNAPSHOT
I0910 04:28:41.881660 3983 sched.cpp:264] No credentials provided. Attempting to register without authentication
I0910 04:28:41.896057 3986 sched.cpp:448] Framework registered with 20150910-020501-169978048-5050-1230-0001
04:28:41.937 [main] INFO o.e.jetty.server.ServerConnector - Started ServerConnector@696dc8c7{HTTP/1.1}{0.0.0.0:8765}
04:28:41.937 [Thread-3] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Registered framework frameworkId=20150910-020501-169978048-5050-1230-0001
04:28:41.938 [main] INFO org.eclipse.jetty.server.Server - Started @1209ms
04:28:41.939 [Thread-3] INFO o.apache.mesos.hdfs.state.LiveState - Acquisition phase is already 'RECONCILING_TASKS'
04:28:42.046 [Thread-13] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 3 offers
04:28:42.058 [Thread-13] INFO o.apache.mesos.hdfs.state.LiveState - Transitioning from acquisition phase 'RECONCILING_TASKS' to 'JOURNAL_NODES'
04:28:42.066 [Thread-13] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:28:42.086 [Thread-13] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Launching node of type journalnode with tasks [journalnode]
04:28:47.842 [Thread-14] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 3 offers
04:28:47.857 [Thread-14] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:28:47.861 [Thread-14] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:28:47.866 [Thread-14] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:28:47.869 [Thread-14] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:28:47.873 [Thread-14] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:28:47.877 [Thread-14] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
2015-09-10 04:28:51,891:3965(0x7f2ef2ffd700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 11ms
04:28:53.870 [Thread-15] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 3 offers
04:28:53.877 [Thread-15] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:28:53.882 [Thread-15] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:28:53.890 [Thread-15] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:28:53.924 [Thread-15] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:28:53.933 [Thread-15] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:28:53.938 [Thread-15] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:28:59.894 [Thread-16] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 3 offers
04:28:59.901 [Thread-16] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:28:59.908 [Thread-16] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:28:59.914 [Thread-16] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:28:59.919 [Thread-16] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:28:59.930 [Thread-16] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:28:59.935 [Thread-16] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:29:05.909 [Thread-17] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 3 offers
04:29:05.943 [Thread-17] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:29:05.952 [Thread-17] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:29:05.957 [Thread-17] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:29:05.966 [Thread-17] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:29:05.984 [Thread-17] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:29:05.989 [Thread-17] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
2015-09-10 04:29:11,929:3965(0x7f2ef2ffd700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 18ms
04:29:11.932 [Thread-18] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 3 offers
04:29:11.984 [Thread-18] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:29:12.002 [Thread-18] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:29:12.015 [Thread-18] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:29:12.038 [Thread-18] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:29:12.087 [Thread-18] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:29:12.283 [Thread-18] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:29:17.952 [Thread-19] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 3 offers
04:29:17.957 [Thread-19] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:29:17.962 [Thread-19] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:29:17.965 [Thread-19] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:29:17.969 [Thread-19] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:29:18.014 [Thread-19] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:29:18.020 [Thread-19] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:29:24.001 [Thread-20] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:29:24.012 [Thread-20] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:29:24.079 [Thread-20] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:29:25.008 [Thread-21] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 2 offers
04:29:25.049 [Thread-21] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:29:25.072 [Thread-21] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:29:25.090 [Thread-21] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:29:25.153 [Thread-21] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
2015-09-10 04:29:25,302:3965(0x7f2ef2ffd700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 23ms
04:29:31.034 [Thread-22] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 3 offers
04:29:31.075 [Thread-22] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:29:31.081 [Thread-22] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:29:31.086 [Thread-22] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:29:31.108 [Thread-22] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:29:31.126 [Thread-22] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:29:31.149 [Thread-22] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
2015-09-10 04:29:31,992:3965(0x7f2ef2ffd700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 18ms
2015-09-10 04:29:35,345:3965(0x7f2ef2ffd700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 19ms
04:29:37.048 [Thread-23] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 3 offers
04:29:37.090 [Thread-23] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:29:37.095 [Thread-23] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:29:37.513 [Thread-23] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:29:37.520 [Thread-23] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:29:37.524 [Thread-23] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:29:37.529 [Thread-23] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:29:43.070 [Thread-24] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 3 offers
04:29:43.082 [Thread-24] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:29:43.110 [Thread-24] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:29:43.170 [Thread-24] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:29:43.212 [Thread-24] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:29:43.263 [Thread-24] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:29:43.308 [Thread-24] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:29:49.096 [Thread-25] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 3 offers
04:29:49.193 [Thread-25] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:29:49.199 [Thread-25] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:29:49.387 [Thread-25] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:29:49.402 [Thread-25] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:29:49.420 [Thread-25] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:29:49.440 [Thread-25] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:29:55.112 [Thread-26] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 3 offers
04:29:55.136 [Thread-26] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:29:55.142 [Thread-26] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:29:55.156 [Thread-26] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:29:55.161 [Thread-26] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:29:55.169 [Thread-26] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:29:55.182 [Thread-26] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:30:01.139 [Thread-27] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 3 offers
04:30:01.158 [Thread-27] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:30:01.175 [Thread-27] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:30:01.179 [Thread-27] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:30:01.189 [Thread-27] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:30:01.193 [Thread-27] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:30:01.199 [Thread-27] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:30:07.160 [Thread-28] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 3 offers
04:30:07.207 [Thread-28] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:30:07.212 [Thread-28] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:30:07.220 [Thread-28] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:30:07.228 [Thread-28] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:30:07.232 [Thread-28] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:30:07.239 [Thread-28] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:30:13.183 [Thread-29] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 2 offers
04:30:13.209 [Thread-29] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:30:13.216 [Thread-29] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:30:13.220 [Thread-29] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:30:13.227 [Thread-29] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:30:14.188 [Thread-30] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:30:14.195 [Thread-30] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:30:14.199 [Thread-30] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:30:19.205 [Thread-31] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 2 offers
04:30:19.211 [Thread-31] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:30:19.216 [Thread-31] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:30:19.222 [Thread-31] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:30:19.226 [Thread-31] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:30:20.228 [Thread-32] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:30:20.232 [Thread-32] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:30:20.239 [Thread-32] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:30:25.238 [Thread-33] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 2 offers
04:30:25.248 [Thread-33] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:30:25.252 [Thread-33] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:30:25.258 [Thread-33] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:30:25.261 [Thread-33] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:30:26.233 [Thread-34] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:30:26.239 [Thread-34] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:30:26.256 [Thread-34] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:30:31.258 [Thread-35] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 2 offers
04:30:31.262 [Thread-35] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:30:31.268 [Thread-35] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:30:31.272 [Thread-35] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:30:31.276 [Thread-35] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:30:32.265 [Thread-36] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:30:32.279 [Thread-36] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:30:32.308 [Thread-36] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:30:37.278 [Thread-37] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 2 offers
04:30:37.283 [Thread-37] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:30:37.301 [Thread-37] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:30:37.309 [Thread-37] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:30:37.313 [Thread-37] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:30:38.288 [Thread-38] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:30:38.307 [Thread-38] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:30:38.409 [Thread-38] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:30:43.297 [Thread-39] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 2 offers
04:30:43.327 [Thread-39] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:30:43.347 [Thread-39] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:30:43.361 [Thread-39] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:30:43.398 [Thread-39] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:30:44.308 [Thread-40] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:30:44.316 [Thread-40] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:30:44.333 [Thread-40] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:30:46.106 [Thread-41] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received status update for taskId=task.journalnode.journalnode.NodeExecutor.1441859322086 state=TASK_LOST message='Abnormal executor termination' stagingTasks.size=1
04:30:46.106 [Thread-41] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Notifying observers
04:30:46.106 [Thread-41] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received task update for: task.journalnode.journalnode.NodeExecutor.1441859322086
04:30:46.108 [Thread-41] INFO o.apache.mesos.hdfs.state.LiveState - Removing running task: value: "task.journalnode.journalnode.NodeExecutor.1441859322086"
04:30:46.177 [Thread-41] INFO o.apache.mesos.hdfs.state.LiveState - Acquisition phase is already 'JOURNAL_NODES'
04:30:46.314 [Thread-42] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:30:46.327 [Thread-42] INFO o.a.m.hdfs.scheduler.HdfsScheduler - [192.168.33.10]
04:30:46.327 [Thread-42] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Launching node of type journalnode with tasks [journalnode]
04:30:49.324 [Thread-43] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:30:49.359 [Thread-43] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:30:49.372 [Thread-43] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:30:50.324 [Thread-44] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:30:50.350 [Thread-44] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:30:50.389 [Thread-44] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:30:52.329 [Thread-45] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:30:52.332 [Thread-45] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:30:52.340 [Thread-45] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:30:55.339 [Thread-46] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:30:55.345 [Thread-46] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:30:55.375 [Thread-46] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:30:56.340 [Thread-47] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:30:56.351 [Thread-47] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:30:56.374 [Thread-47] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:30:58.358 [Thread-48] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:30:58.369 [Thread-48] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:30:58.380 [Thread-48] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:31:01.357 [Thread-49] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:31:01.380 [Thread-49] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:31:01.398 [Thread-49] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:31:03.372 [Thread-50] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:31:03.377 [Thread-50] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:31:03.390 [Thread-50] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:31:04.372 [Thread-51] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:31:04.378 [Thread-51] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:31:04.391 [Thread-51] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:31:08.379 [Thread-52] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:31:08.390 [Thread-52] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:31:08.401 [Thread-52] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:31:09.381 [Thread-53] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:31:09.385 [Thread-53] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:31:09.388 [Thread-53] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:31:10.383 [Thread-54] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:31:10.508 [Thread-54] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:31:10.512 [Thread-54] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:31:14.396 [Thread-55] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:31:14.406 [Thread-55] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:31:14.410 [Thread-55] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:31:15.401 [Thread-56] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:31:15.459 [Thread-56] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:31:15.469 [Thread-56] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:31:17.411 [Thread-57] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:31:17.420 [Thread-57] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:31:17.431 [Thread-57] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:31:20.437 [Thread-58] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:31:20.444 [Thread-58] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:31:20.448 [Thread-58] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:31:21.435 [Thread-59] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:31:21.440 [Thread-59] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:31:21.445 [Thread-59] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:31:23.435 [Thread-60] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:31:23.440 [Thread-60] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:31:23.445 [Thread-60] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:31:26.485 [Thread-61] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:31:26.488 [Thread-61] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:31:26.495 [Thread-61] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:31:27.451 [Thread-62] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:31:27.455 [Thread-62] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:31:27.472 [Thread-62] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:31:28.460 [Thread-63] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:31:28.467 [Thread-63] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:31:28.475 [Thread-63] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
2015-09-10 04:31:32,236:3965(0x7f2ef2ffd700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 71ms
04:31:33.469 [Thread-64] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 2 offers
04:31:33.478 [Thread-64] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:31:33.511 [Thread-64] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:31:33.519 [Thread-64] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:31:33.552 [Thread-64] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:31:34.479 [Thread-65] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:31:34.498 [Thread-65] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:31:34.504 [Thread-65] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
2015-09-10 04:31:38,965:3965(0x7f2ef2ffd700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 56ms
04:31:39.492 [Thread-66] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 2 offers
04:31:39.508 [Thread-66] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:31:39.543 [Thread-66] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:31:39.563 [Thread-66] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:31:39.572 [Thread-66] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:31:40.497 [Thread-67] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:31:40.501 [Thread-67] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:31:40.505 [Thread-67] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:31:45.516 [Thread-68] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:31:45.523 [Thread-68] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:31:45.529 [Thread-68] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:31:46.521 [Thread-69] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 2 offers
04:31:46.530 [Thread-69] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:31:46.540 [Thread-69] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:31:46.546 [Thread-69] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:31:46.551 [Thread-69] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:31:51.545 [Thread-70] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:31:51.553 [Thread-70] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:31:51.561 [Thread-70] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:31:52.539 [Thread-71] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:31:52.554 [Thread-71] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:31:52.576 [Thread-71] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:31:53.546 [Thread-72] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:31:53.585 [Thread-72] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:31:53.590 [Thread-72] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:31:58.571 [Thread-73] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 2 offers
04:31:58.577 [Thread-73] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:31:58.588 [Thread-73] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:31:58.591 [Thread-73] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:31:58.606 [Thread-73] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:31:59.569 [Thread-74] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:31:59.594 [Thread-74] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:31:59.745 [Thread-74] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:32:04.593 [Thread-75] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 2 offers
04:32:04.627 [Thread-75] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:32:04.677 [Thread-75] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:32:04.706 [Thread-75] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:32:04.719 [Thread-75] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:32:05.600 [Thread-76] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:32:05.610 [Thread-76] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:32:05.648 [Thread-76] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
2015-09-10 04:32:09,035:3965(0x7f2ef2ffd700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 26ms
04:32:10.628 [Thread-77] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:32:10.652 [Thread-77] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:32:10.691 [Thread-77] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:32:11.628 [Thread-78] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 2 offers
04:32:11.642 [Thread-78] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:32:11.665 [Thread-78] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:32:11.702 [Thread-78] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:32:11.709 [Thread-78] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:32:16.637 [Thread-79] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:32:16.720 [Thread-79] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:32:16.765 [Thread-79] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:32:17.644 [Thread-80] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:32:17.759 [Thread-80] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:32:17.781 [Thread-80] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:32:18.644 [Thread-81] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:32:18.693 [Thread-81] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:32:18.762 [Thread-81] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
2015-09-10 04:32:19,080:3965(0x7f2ef2ffd700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 36ms
04:32:22.660 [Thread-82] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:32:22.682 [Thread-82] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:32:22.718 [Thread-82] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:32:23.664 [Thread-83] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:32:23.699 [Thread-83] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:32:23.712 [Thread-83] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:32:24.668 [Thread-84] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:32:24.679 [Thread-84] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:32:24.711 [Thread-84] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:32:28.679 [Thread-85] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:32:28.694 [Thread-85] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:32:28.744 [Thread-85] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:32:29.684 [Thread-86] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:32:29.827 [Thread-86] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:32:29.860 [Thread-86] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:32:30.701 [Thread-87] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:32:30.734 [Thread-87] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:32:30.783 [Thread-87] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:32:34.711 [Thread-88] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:32:34.779 [Thread-88] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:32:34.971 [Thread-88] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:32:35.716 [Thread-89] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:32:35.791 [Thread-89] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:32:35.809 [Thread-89] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:32:36.720 [Thread-90] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:32:36.752 [Thread-90] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:32:36.786 [Thread-90] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:32:41.741 [Thread-91] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:32:41.991 [Thread-91] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:32:42.004 [Thread-91] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:32:42.747 [Thread-92] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 2 offers
04:32:42.761 [Thread-92] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:32:42.777 [Thread-92] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:32:42.804 [Thread-92] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:32:42.810 [Thread-92] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:32:47.765 [Thread-93] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:32:47.779 [Thread-93] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:32:47.790 [Thread-93] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:32:48.769 [Thread-94] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 2 offers
04:32:48.782 [Thread-94] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:32:48.791 [Thread-94] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:32:48.808 [Thread-94] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:32:48.851 [Thread-94] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
2015-09-10 04:32:52,473:3965(0x7f2ef2ffd700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 17ms
04:32:53.786 [Thread-95] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:32:53.813 [Thread-95] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:32:53.852 [Thread-95] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:32:54.793 [Thread-96] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:32:54.798 [Thread-96] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:32:54.808 [Thread-96] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
04:32:55.794 [Thread-97] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Received 1 offers
04:32:55.804 [Thread-97] INFO o.a.m.hdfs.scheduler.HdfsScheduler - []
04:32:55.809 [Thread-97] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Already running 1 journalnodes
The stderr
from the JN1 task:
I0910 04:28:42.494109 6096 logging.cpp:172] INFO level logging started!
I0910 04:28:42.500326 6096 fetcher.cpp:214] Fetching URI 'http://192.168.33.11:8765/hdfs-mesos-executor-0.1.3.tgz'
I0910 04:28:42.500411 6096 fetcher.cpp:125] Fetching URI 'http://192.168.33.11:8765/hdfs-mesos-executor-0.1.3.tgz' with os::net
I0910 04:28:42.500459 6096 fetcher.cpp:135] Downloading 'http://192.168.33.11:8765/hdfs-mesos-executor-0.1.3.tgz' to '/tmp/mesos/slaves/20150910-020501-169978048-5050-1230-S0/frameworks/20150910-020501-169978048-5050-1230-0001/executors/executor.journalnode.NodeExecutor.1441859322086/runs/851cfc25-42a6-45df-9c4f-0e52138fed2e/hdfs-mesos-executor-0.1.3.tgz'
I0910 04:28:47.882668 6096 fetcher.cpp:78] Extracted resource '/tmp/mesos/slaves/20150910-020501-169978048-5050-1230-S0/frameworks/20150910-020501-169978048-5050-1230-0001/executors/executor.journalnode.NodeExecutor.1441859322086/runs/851cfc25-42a6-45df-9c4f-0e52138fed2e/hdfs-mesos-executor-0.1.3.tgz' into '/tmp/mesos/slaves/20150910-020501-169978048-5050-1230-S0/frameworks/20150910-020501-169978048-5050-1230-0001/executors/executor.journalnode.NodeExecutor.1441859322086/runs/851cfc25-42a6-45df-9c4f-0e52138fed2e'
I0910 04:28:47.888000 6096 fetcher.cpp:214] Fetching URI 'http://192.168.33.11:8765/hdfs-site.xml'
I0910 04:28:47.888078 6096 fetcher.cpp:125] Fetching URI 'http://192.168.33.11:8765/hdfs-site.xml' with os::net
I0910 04:28:47.888118 6096 fetcher.cpp:135] Downloading 'http://192.168.33.11:8765/hdfs-site.xml' to '/tmp/mesos/slaves/20150910-020501-169978048-5050-1230-S0/frameworks/20150910-020501-169978048-5050-1230-0001/executors/executor.journalnode.NodeExecutor.1441859322086/runs/851cfc25-42a6-45df-9c4f-0e52138fed2e/hdfs-site.xml'
I0910 04:28:47.932752 6096 fetcher.cpp:214] Fetching URI 'https://downloads.mesosphere.io/java/jre-7u76-linux-x64.tar.gz'
I0910 04:28:47.932917 6096 fetcher.cpp:125] Fetching URI 'https://downloads.mesosphere.io/java/jre-7u76-linux-x64.tar.gz' with os::net
I0910 04:28:47.932968 6096 fetcher.cpp:135] Downloading 'https://downloads.mesosphere.io/java/jre-7u76-linux-x64.tar.gz' to '/tmp/mesos/slaves/20150910-020501-169978048-5050-1230-S0/frameworks/20150910-020501-169978048-5050-1230-0001/executors/executor.journalnode.NodeExecutor.1441859322086/runs/851cfc25-42a6-45df-9c4f-0e52138fed2e/jre-7u76-linux-x64.tar.gz'
I0910 04:30:46.004843 6096 fetcher.cpp:78] Extracted resource '/tmp/mesos/slaves/20150910-020501-169978048-5050-1230-S0/frameworks/20150910-020501-169978048-5050-1230-0001/executors/executor.journalnode.NodeExecutor.1441859322086/runs/851cfc25-42a6-45df-9c4f-0e52138fed2e/jre-7u76-linux-x64.tar.gz' into '/tmp/mesos/slaves/20150910-020501-169978048-5050-1230-S0/frameworks/20150910-020501-169978048-5050-1230-0001/executors/executor.journalnode.NodeExecutor.1441859322086/runs/851cfc25-42a6-45df-9c4f-0e52138fed2e'
stdout
is empty.
Other frameworks seems to be running fine, I tried Elasticsearch and Kafka.
If you are running other frameworks on your cluster, including ones that run in docker, it could be related to a Mesos bug where with systemd and the containerizer where if you run multiple frameworks (that run in docker and outside docker), executors and tasks get killed.
If you keep seeing something like that, Sep 08 00:50:09 core-04 mesos-slave[1271]: E0908 00:50:09.389919 1278 slave.cpp:3340] Termination of executor 'executor.journalnode.NodeExecutor.1441673294675' of framework '20150908-001609-169978048-5050-1255-0001' failed: Unknown container: 126df935-512d-4e4f-b03b-7203f8ff6f75
it is a sure sign of containerizer issues.
In the above tests, I ran the hdfs framework by it self (after stopping and removing all other frameworks) and in fresh clusters.
Is there anyway to get more debug info out of the framework or mesos to work out the exact root cause?
I will also set up a Ubuntu cluster to test and see if it suffers from the same problem.
Here are my results from setting up a Ubuntu cluster containing 4 nodes: 1 master/slave and 3 slaves.
Mesos is 0.23.0 and marathon is 0.10.1.
In this case, I am still seeing the JournalNode task being lost (the same situation as when I was usinng CoreOS).
The logs from the mesos-slave where the scheduler is launched:
Sep 11 09:40:58 mesos-slave-03 mesos-slave[733]: I0911 09:40:58.568208 884 slave.cpp:4193] Received oversubscribable resources from the resource estimator
Sep 11 09:41:13 mesos-slave-03 mesos-slave[733]: I0911 09:41:13.573972 885 slave.cpp:4179] Querying resource estimator for oversubscribable resources
Sep 11 09:41:13 mesos-slave-03 mesos-slave[733]: I0911 09:41:13.574148 885 slave.cpp:4193] Received oversubscribable resources from the resource estimator
Sep 11 09:41:28 mesos-slave-03 mesos-slave[733]: I0911 09:41:28.579911 886 slave.cpp:4179] Querying resource estimator for oversubscribable resources
Sep 11 09:41:28 mesos-slave-03 mesos-slave[733]: I0911 09:41:28.580255 886 slave.cpp:4193] Received oversubscribable resources from the resource estimator
Sep 11 09:41:43 mesos-slave-03 mesos-slave[733]: I0911 09:41:43.585804 887 slave.cpp:4179] Querying resource estimator for oversubscribable resources
Sep 11 09:41:43 mesos-slave-03 mesos-slave[733]: I0911 09:41:43.586032 887 slave.cpp:4193] Received oversubscribable resources from the resource estimator
Sep 11 09:41:43 mesos-slave-03 mesos-slave[733]: I0911 09:41:43.957613 883 slave.cpp:1244] Got assigned task task.journalnode.journalnode.NodeExecutor.1441964503869 for framework 20150911-055603-169978048-5050-644-0000
Sep 11 09:41:43 mesos-slave-03 mesos-slave[733]: I0911 09:41:43.958948 883 slave.cpp:1355] Launching task task.journalnode.journalnode.NodeExecutor.1441964503869 for framework 20150911-055603-169978048-5050-644-0000
Sep 11 09:41:43 mesos-slave-03 mesos-slave[733]: I0911 09:41:43.963919 883 slave.cpp:4733] Launching executor executor.journalnode.NodeExecutor.1441964503869 of framework 20150911-055603-169978048-5050-644-0000 with resources cpus(*):0.5; mem(*):256 in work directory '/tmp/mesos/slaves/20150911-055603-169978048-5050-644-S3/frameworks/20150911-055603-169978048-5050-644-0000/executors/executor.journalnode.NodeExecutor.1441964503869/runs/c9c3afd6-af08-4af0-a2b6-5c17280c9239'
Sep 11 09:41:43 mesos-slave-03 mesos-slave[733]: I0911 09:41:43.964347 885 docker.cpp:739] No container info found, skipping launch
Sep 11 09:41:43 mesos-slave-03 mesos-slave[733]: I0911 09:41:43.964629 883 slave.cpp:1573] Queuing task 'task.journalnode.journalnode.NodeExecutor.1441964503869' for executor executor.journalnode.NodeExecutor.1441964503869 of framework '20150911-055603-169978048-5050-644-0000
Sep 11 09:41:43 mesos-slave-03 mesos-slave[733]: I0911 09:41:43.965713 885 containerizer.cpp:534] Starting container 'c9c3afd6-af08-4af0-a2b6-5c17280c9239' for executor 'executor.journalnode.NodeExecutor.1441964503869' of framework '20150911-055603-169978048-5050-644-0000'
Sep 11 09:41:43 mesos-slave-03 mesos-slave[733]: I0911 09:41:43.967423 887 launcher.cpp:131] Forked child with pid '3106' for container 'c9c3afd6-af08-4af0-a2b6-5c17280c9239'
Sep 11 09:41:43 mesos-slave-03 mesos-slave[733]: I0911 09:41:43.967866 887 containerizer.cpp:770] Checkpointing executor's forked pid 3106 to '/tmp/mesos/meta/slaves/20150911-055603-169978048-5050-644-S3/frameworks/20150911-055603-169978048-5050-644-0000/executors/executor.journalnode.NodeExecutor.1441964503869/runs/c9c3afd6-af08-4af0-a2b6-5c17280c9239/pids/forked.pid'
Sep 11 09:41:46 mesos-slave-03 mesos-slave[733]: I0911 09:41:46.034028 886 slave.cpp:3842] Current disk usage 7.30%. Max allowed age: 5.788928537733993days
Sep 11 09:41:58 mesos-slave-03 mesos-slave[733]: I0911 09:41:58.587863 889 slave.cpp:4179] Querying resource estimator for oversubscribable resources
Sep 11 09:41:58 mesos-slave-03 mesos-slave[733]: I0911 09:41:58.588443 889 slave.cpp:4193] Received oversubscribable resources from the resource estimator
Sep 11 09:42:13 mesos-slave-03 mesos-slave[733]: I0911 09:42:13.588912 883 slave.cpp:4179] Querying resource estimator for oversubscribable resources
Sep 11 09:42:13 mesos-slave-03 mesos-slave[733]: I0911 09:42:13.589126 883 slave.cpp:4193] Received oversubscribable resources from the resource estimator
Sep 11 09:42:28 mesos-slave-03 mesos-slave[733]: I0911 09:42:28.590184 884 slave.cpp:4179] Querying resource estimator for oversubscribable resources
Sep 11 09:42:28 mesos-slave-03 mesos-slave[733]: I0911 09:42:28.590517 884 slave.cpp:4193] Received oversubscribable resources from the resource estimator
Sep 11 09:42:43 mesos-slave-03 mesos-slave[733]: I0911 09:42:43.592941 890 slave.cpp:4179] Querying resource estimator for oversubscribable resources
Sep 11 09:42:43 mesos-slave-03 mesos-slave[733]: I0911 09:42:43.593324 890 slave.cpp:4193] Received oversubscribable resources from the resource estimator
Sep 11 09:42:43 mesos-slave-03 mesos-slave[733]: I0911 09:42:43.966051 890 slave.cpp:3798] Terminating executor executor.journalnode.NodeExecutor.1441964503869 of framework 20150911-055603-169978048-5050-644-0000 because it did not register within 1mins
Sep 11 09:42:43 mesos-slave-03 mesos-slave[733]: I0911 09:42:43.967139 890 containerizer.cpp:1001] Destroying container 'c9c3afd6-af08-4af0-a2b6-5c17280c9239'
Sep 11 09:42:44 mesos-slave-03 mesos-slave[733]: I0911 09:42:44.087127 883 containerizer.cpp:1188] Executor for container 'c9c3afd6-af08-4af0-a2b6-5c17280c9239' has exited
Sep 11 09:42:46 mesos-slave-03 mesos-slave[733]: I0911 09:42:46.036401 887 slave.cpp:3842] Current disk usage 7.38%. Max allowed age: 5.783462926418901days
Sep 11 09:42:58 mesos-slave-03 mesos-slave[733]: I0911 09:42:58.595213 883 slave.cpp:4179] Querying resource estimator for oversubscribable resources
Sep 11 09:42:58 mesos-slave-03 mesos-slave[733]: I0911 09:42:58.595502 883 slave.cpp:4193] Received oversubscribable resources from the resource estimator
Sep 11 09:43:13 mesos-slave-03 mesos-slave[733]: I0911 09:43:13.597286 887 slave.cpp:4179] Querying resource estimator for oversubscribable resources
Sep 11 09:43:13 mesos-slave-03 mesos-slave[733]: I0911 09:43:13.597537 887 slave.cpp:4193] Received oversubscribable resources from the resource estimator
Sep 11 09:43:15 mesos-slave-03 mesos-slave[733]: E0911 09:43:15.821022 888 slave.cpp:3258] Container 'c9c3afd6-af08-4af0-a2b6-5c17280c9239' for executor 'executor.journalnode.NodeExecutor.1441964503869' of framework '20150911-055603-169978048-5050-644-0000' failed to start: Container destroyed during launch
Sep 11 09:43:15 mesos-slave-03 mesos-slave[733]: E0911 09:43:15.821300 888 slave.cpp:3340] Termination of executor 'executor.journalnode.NodeExecutor.1441964503869' of framework '20150911-055603-169978048-5050-644-0000' failed: Unknown container: c9c3afd6-af08-4af0-a2b6-5c17280c9239
Sep 11 09:43:15 mesos-slave-03 mesos-slave[733]: W0911 09:43:15.821337 890 composing.cpp:520] Container 'c9c3afd6-af08-4af0-a2b6-5c17280c9239' is already destroyed
Sep 11 09:43:15 mesos-slave-03 mesos-slave[733]: I0911 09:43:15.823056 888 slave.cpp:2671] Handling status update TASK_LOST (UUID: a0efed75-b4cc-4216-8dc4-cc21c5020077) for task task.journalnode.journalnode.NodeExecutor.1441964503869 of framework 20150911-055603-169978048-5050-644-0000 from @0.0.0.0:0
Sep 11 09:43:15 mesos-slave-03 mesos-slave[733]: W0911 09:43:15.823374 887 containerizer.cpp:885] Ignoring update for unknown container: c9c3afd6-af08-4af0-a2b6-5c17280c9239
Sep 11 09:43:15 mesos-slave-03 mesos-slave[733]: I0911 09:43:15.823571 887 status_update_manager.cpp:322] Received status update TASK_LOST (UUID: a0efed75-b4cc-4216-8dc4-cc21c5020077) for task task.journalnode.journalnode.NodeExecutor.1441964503869 of framework 20150911-055603-169978048-5050-644-0000
Sep 11 09:43:15 mesos-slave-03 mesos-slave[733]: I0911 09:43:15.823814 887 status_update_manager.cpp:826] Checkpointing UPDATE for status update TASK_LOST (UUID: a0efed75-b4cc-4216-8dc4-cc21c5020077) for task task.journalnode.journalnode.NodeExecutor.1441964503869 of framework 20150911-055603-169978048-5050-644-0000
Sep 11 09:43:15 mesos-slave-03 mesos-slave[733]: I0911 09:43:15.827191 887 slave.cpp:2926] Forwarding the update TASK_LOST (UUID: a0efed75-b4cc-4216-8dc4-cc21c5020077) for task task.journalnode.journalnode.NodeExecutor.1441964503869 of framework 20150911-055603-169978048-5050-644-0000 to master@192.168.33.10:5050
Sep 11 09:43:15 mesos-slave-03 mesos-slave[733]: I0911 09:43:15.850566 884 status_update_manager.cpp:394] Received status update acknowledgement (UUID: a0efed75-b4cc-4216-8dc4-cc21c5020077) for task task.journalnode.journalnode.NodeExecutor.1441964503869 of framework 20150911-055603-169978048-5050-644-0000
Sep 11 09:43:15 mesos-slave-03 mesos-slave[733]: I0911 09:43:15.850788 884 status_update_manager.cpp:826] Checkpointing ACK for status update TASK_LOST (UUID: a0efed75-b4cc-4216-8dc4-cc21c5020077) for task task.journalnode.journalnode.NodeExecutor.1441964503869 of framework 20150911-055603-169978048-5050-644-0000
Sep 11 09:43:15 mesos-slave-03 mesos-slave[733]: I0911 09:43:15.853552 884 slave.cpp:3460] Cleaning up executor 'executor.journalnode.NodeExecutor.1441964503869' of framework 20150911-055603-169978048-5050-644-0000
Sep 11 09:43:15 mesos-slave-03 mesos-slave[733]: I0911 09:43:15.854105 884 slave.cpp:3549] Cleaning up framework 20150911-055603-169978048-5050-644-0000
Sep 11 09:43:15 mesos-slave-03 mesos-slave[733]: I0911 09:43:15.854246 884 gc.cpp:56] Scheduling '/tmp/mesos/slaves/20150911-055603-169978048-5050-644-S3/frameworks/20150911-055603-169978048-5050-644-0000/executors/executor.journalnode.NodeExecutor.1441964503869/runs/c9c3afd6-af08-4af0-a2b6-5c17280c9239' for gc 6.99999011670222days in the future
Sep 11 09:43:15 mesos-slave-03 mesos-slave[733]: I0911 09:43:15.854334 884 gc.cpp:56] Scheduling '/tmp/mesos/slaves/20150911-055603-169978048-5050-644-S3/frameworks/20150911-055603-169978048-5050-644-0000/executors/executor.journalnode.NodeExecutor.1441964503869' for gc 6.99999011571852days in the future
Sep 11 09:43:15 mesos-slave-03 mesos-slave[733]: I0911 09:43:15.854410 884 gc.cpp:56] Scheduling '/tmp/mesos/meta/slaves/20150911-055603-169978048-5050-644-S3/frameworks/20150911-055603-169978048-5050-644-0000/executors/executor.journalnode.NodeExecutor.1441964503869/runs/c9c3afd6-af08-4af0-a2b6-5c17280c9239' for gc 6.99999011518519days in the future
Sep 11 09:43:15 mesos-slave-03 mesos-slave[733]: I0911 09:43:15.854480 884 gc.cpp:56] Scheduling '/tmp/mesos/meta/slaves/20150911-055603-169978048-5050-644-S3/frameworks/20150911-055603-169978048-5050-644-0000/executors/executor.journalnode.NodeExecutor.1441964503869' for gc 6.99999011474667days in the future
Sep 11 09:43:15 mesos-slave-03 mesos-slave[733]: I0911 09:43:15.854548 884 gc.cpp:56] Scheduling '/tmp/mesos/slaves/20150911-055603-169978048-5050-644-S3/frameworks/20150911-055603-169978048-5050-644-0000' for gc 6.99999011355556days in the future
Sep 11 09:43:15 mesos-slave-03 mesos-slave[733]: I0911 09:43:15.854614 884 gc.cpp:56] Scheduling '/tmp/mesos/meta/slaves/20150911-055603-169978048-5050-644-S3/frameworks/20150911-055603-169978048-5050-644-0000' for gc 6.99999011315852days in the future
Sep 11 09:43:15 mesos-slave-03 mesos-slave[733]: I0911 09:43:15.854683 884 status_update_manager.cpp:284] Closing status update streams for framework 20150911-055603-169978048-5050-644-0000
Sep 11 09:43:15 mesos-slave-03 mesos-slave[733]: I0911 09:43:15.892761 884 slave.cpp:1244] Got assigned task task.journalnode.journalnode.NodeExecutor.1441964595884 for framework 20150911-055603-169978048-5050-644-0000
Sep 11 09:43:15 mesos-slave-03 mesos-slave[733]: I0911 09:43:15.893767 884 gc.cpp:84] Unscheduling '/tmp/mesos/slaves/20150911-055603-169978048-5050-644-S3/frameworks/20150911-055603-169978048-5050-644-0000' from gc
Sep 11 09:43:15 mesos-slave-03 mesos-slave[733]: I0911 09:43:15.894336 888 gc.cpp:84] Unscheduling '/tmp/mesos/meta/slaves/20150911-055603-169978048-5050-644-S3/frameworks/20150911-055603-169978048-5050-644-0000' from gc
Sep 11 09:43:15 mesos-slave-03 mesos-slave[733]: I0911 09:43:15.894470 888 slave.cpp:1355] Launching task task.journalnode.journalnode.NodeExecutor.1441964595884 for framework 20150911-055603-169978048-5050-644-0000
Sep 11 09:43:15 mesos-slave-03 mesos-slave[733]: I0911 09:43:15.898707 888 slave.cpp:4733] Launching executor executor.journalnode.NodeExecutor.1441964595884 of framework 20150911-055603-169978048-5050-644-0000 with resources cpus(*):0.5; mem(*):256 in work directory '/tmp/mesos/slaves/20150911-055603-169978048-5050-644-S3/frameworks/20150911-055603-169978048-5050-644-0000/executors/executor.journalnode.NodeExecutor.1441964595884/runs/fecea3fc-87f2-404b-943f-08230a1f7938'
Sep 11 09:43:15 mesos-slave-03 mesos-slave[733]: I0911 09:43:15.899261 888 slave.cpp:1573] Queuing task 'task.journalnode.journalnode.NodeExecutor.1441964595884' for executor executor.journalnode.NodeExecutor.1441964595884 of framework '20150911-055603-169978048-5050-644-0000
Sep 11 09:43:15 mesos-slave-03 mesos-slave[733]: I0911 09:43:15.899538 883 docker.cpp:739] No container info found, skipping launch
Sep 11 09:43:15 mesos-slave-03 mesos-slave[733]: I0911 09:43:15.899785 883 containerizer.cpp:534] Starting container 'fecea3fc-87f2-404b-943f-08230a1f7938' for executor 'executor.journalnode.NodeExecutor.1441964595884' of framework '20150911-055603-169978048-5050-644-0000'
Sep 11 09:43:15 mesos-slave-03 mesos-slave[733]: I0911 09:43:15.901474 883 launcher.cpp:131] Forked child with pid '3243' for container 'fecea3fc-87f2-404b-943f-08230a1f7938'
Sep 11 09:43:15 mesos-slave-03 mesos-slave[733]: I0911 09:43:15.901763 883 containerizer.cpp:770] Checkpointing executor's forked pid 3243 to '/tmp/mesos/meta/slaves/20150911-055603-169978048-5050-644-S3/frameworks/20150911-055603-169978048-5050-644-0000/executors/executor.journalnode.NodeExecutor.1441964595884/runs/fecea3fc-87f2-404b-943f-08230a1f7938/pids/forked.pid'
Sep 11 09:43:28 mesos-slave-03 mesos-slave[733]: I0911 09:43:28.599494 884 slave.cpp:4179] Querying resource estimator for oversubscribable resources
Sep 11 09:43:28 mesos-slave-03 mesos-slave[733]: I0911 09:43:28.599934 884 slave.cpp:4193] Received oversubscribable resources from the resource estimator
Sep 11 09:43:43 mesos-slave-03 mesos-slave[733]: I0911 09:43:43.603425 883 slave.cpp:4179] Querying resource estimator for oversubscribable resources
Sep 11 09:43:43 mesos-slave-03 mesos-slave[733]: I0911 09:43:43.604190 883 slave.cpp:4193] Received oversubscribable resources from the resource estimator
Sep 11 09:43:46 mesos-slave-03 mesos-slave[733]: I0911 09:43:46.037075 887 slave.cpp:3842] Current disk usage 8.06%. Max allowed age: 5.735835403677870days
Sep 11 09:43:58 mesos-slave-03 mesos-slave[733]: I0911 09:43:58.604714 890 slave.cpp:4179] Querying resource estimator for oversubscribable resources
Sep 11 09:43:58 mesos-slave-03 mesos-slave[733]: I0911 09:43:58.605131 890 slave.cpp:4193] Received oversubscribable resources from the resource estimator
Sep 11 09:44:13 mesos-slave-03 mesos-slave[733]: I0911 09:44:13.606503 884 slave.cpp:4179] Querying resource estimator for oversubscribable resources
Sep 11 09:44:13 mesos-slave-03 mesos-slave[733]: I0911 09:44:13.606879 884 slave.cpp:4193] Received oversubscribable resources from the resource estimator
Sep 11 09:44:15 mesos-slave-03 mesos-slave[733]: I0911 09:44:15.900657 886 slave.cpp:3798] Terminating executor executor.journalnode.NodeExecutor.1441964595884 of framework 20150911-055603-169978048-5050-644-0000 because it did not register within 1mins
Sep 11 09:44:15 mesos-slave-03 mesos-slave[733]: I0911 09:44:15.901098 886 containerizer.cpp:1001] Destroying container 'fecea3fc-87f2-404b-943f-08230a1f7938'
Sep 11 09:44:16 mesos-slave-03 mesos-slave[733]: I0911 09:44:16.005170 889 containerizer.cpp:1188] Executor for container 'fecea3fc-87f2-404b-943f-08230a1f7938' has exited
Sep 11 09:44:28 mesos-slave-03 mesos-slave[733]: I0911 09:44:28.610803 889 slave.cpp:4179] Querying resource estimator for oversubscribable resources
Sep 11 09:44:28 mesos-slave-03 mesos-slave[733]: I0911 09:44:28.611074 889 slave.cpp:4193] Received oversubscribable resources from the resource estimator
Sep 11 09:44:43 mesos-slave-03 mesos-slave[733]: I0911 09:44:43.612027 890 slave.cpp:4179] Querying resource estimator for oversubscribable resources
Sep 11 09:44:43 mesos-slave-03 mesos-slave[733]: I0911 09:44:43.612483 890 slave.cpp:4193] Received oversubscribable resources from the resource estimator
Sep 11 09:44:46 mesos-slave-03 mesos-slave[733]: I0911 09:44:46.044634 885 slave.cpp:3842] Current disk usage 8.13%. Max allowed age: 5.730711694039757days
Sep 11 09:44:55 mesos-slave-03 mesos-slave[733]: E0911 09:44:55.962294 884 slave.cpp:3258] Container 'fecea3fc-87f2-404b-943f-08230a1f7938' for executor 'executor.journalnode.NodeExecutor.1441964595884' of framework '20150911-055603-169978048-5050-644-0000' failed to start: Container destroyed during launch
Sep 11 09:44:55 mesos-slave-03 mesos-slave[733]: E0911 09:44:55.962698 884 slave.cpp:3340] Termination of executor 'executor.journalnode.NodeExecutor.1441964595884' of framework '20150911-055603-169978048-5050-644-0000' failed: Unknown container: fecea3fc-87f2-404b-943f-08230a1f7938
Sep 11 09:44:55 mesos-slave-03 mesos-slave[733]: W0911 09:44:55.962759 890 composing.cpp:520] Container 'fecea3fc-87f2-404b-943f-08230a1f7938' is already destroyed
Sep 11 09:44:55 mesos-slave-03 mesos-slave[733]: I0911 09:44:55.964284 884 slave.cpp:2671] Handling status update TASK_LOST (UUID: 746ef490-057b-41e1-ae93-fb8f984b6815) for task task.journalnode.journalnode.NodeExecutor.1441964595884 of framework 20150911-055603-169978048-5050-644-0000 from @0.0.0.0:0
Sep 11 09:44:55 mesos-slave-03 mesos-slave[733]: W0911 09:44:55.964607 883 containerizer.cpp:885] Ignoring update for unknown container: fecea3fc-87f2-404b-943f-08230a1f7938
Sep 11 09:44:55 mesos-slave-03 mesos-slave[733]: I0911 09:44:55.964953 884 status_update_manager.cpp:322] Received status update TASK_LOST (UUID: 746ef490-057b-41e1-ae93-fb8f984b6815) for task task.journalnode.journalnode.NodeExecutor.1441964595884 of framework 20150911-055603-169978048-5050-644-0000
Sep 11 09:44:55 mesos-slave-03 mesos-slave[733]: I0911 09:44:55.965289 884 status_update_manager.cpp:826] Checkpointing UPDATE for status update TASK_LOST (UUID: 746ef490-057b-41e1-ae93-fb8f984b6815) for task task.journalnode.journalnode.NodeExecutor.1441964595884 of framework 20150911-055603-169978048-5050-644-0000
Sep 11 09:44:55 mesos-slave-03 mesos-slave[733]: I0911 09:44:55.970175 884 slave.cpp:2926] Forwarding the update TASK_LOST (UUID: 746ef490-057b-41e1-ae93-fb8f984b6815) for task task.journalnode.journalnode.NodeExecutor.1441964595884 of framework 20150911-055603-169978048-5050-644-0000 to master@192.168.33.10:5050
Sep 11 09:44:55 mesos-slave-03 mesos-slave[733]: I0911 09:44:55.995640 885 status_update_manager.cpp:394] Received status update acknowledgement (UUID: 746ef490-057b-41e1-ae93-fb8f984b6815) for task task.journalnode.journalnode.NodeExecutor.1441964595884 of framework 20150911-055603-169978048-5050-644-0000
Sep 11 09:44:55 mesos-slave-03 mesos-slave[733]: I0911 09:44:55.996058 885 status_update_manager.cpp:826] Checkpointing ACK for status update TASK_LOST (UUID: 746ef490-057b-41e1-ae93-fb8f984b6815) for task task.journalnode.journalnode.NodeExecutor.1441964595884 of framework 20150911-055603-169978048-5050-644-0000
Sep 11 09:44:55 mesos-slave-03 mesos-slave[733]: I0911 09:44:55.998778 885 slave.cpp:3460] Cleaning up executor 'executor.journalnode.NodeExecutor.1441964595884' of framework 20150911-055603-169978048-5050-644-0000
Sep 11 09:44:55 mesos-slave-03 mesos-slave[733]: I0911 09:44:55.999249 887 gc.cpp:56] Scheduling '/tmp/mesos/slaves/20150911-055603-169978048-5050-644-S3/frameworks/20150911-055603-169978048-5050-644-0000/executors/executor.journalnode.NodeExecutor.1441964595884/runs/fecea3fc-87f2-404b-943f-08230a1f7938' for gc 6.99998843677037days in the future
Sep 11 09:44:55 mesos-slave-03 mesos-slave[733]: I0911 09:44:55.999308 885 slave.cpp:3549] Cleaning up framework 20150911-055603-169978048-5050-644-0000
Sep 11 09:44:55 mesos-slave-03 mesos-slave[733]: I0911 09:44:55.999354 887 gc.cpp:56] Scheduling '/tmp/mesos/slaves/20150911-055603-169978048-5050-644-S3/frameworks/20150911-055603-169978048-5050-644-0000/executors/executor.journalnode.NodeExecutor.1441964595884' for gc 6.99998843509333days in the future
Sep 11 09:44:55 mesos-slave-03 mesos-slave[733]: I0911 09:44:55.999428 887 gc.cpp:56] Scheduling '/tmp/mesos/meta/slaves/20150911-055603-169978048-5050-644-S3/frameworks/20150911-055603-169978048-5050-644-0000/executors/executor.journalnode.NodeExecutor.1441964595884/runs/fecea3fc-87f2-404b-943f-08230a1f7938' for gc 6.99998843453926days in the future
Sep 11 09:44:55 mesos-slave-03 mesos-slave[733]: I0911 09:44:55.999544 887 gc.cpp:56] Scheduling '/tmp/mesos/meta/slaves/20150911-055603-169978048-5050-644-S3/frameworks/20150911-055603-169978048-5050-644-0000/executors/executor.journalnode.NodeExecutor.1441964595884' for gc 6.99998843412148days in the future
Sep 11 09:44:56 mesos-slave-03 mesos-slave[733]: I0911 09:44:55.999615 887 gc.cpp:56] Scheduling '/tmp/mesos/slaves/20150911-055603-169978048-5050-644-S3/frameworks/20150911-055603-169978048-5050-644-0000' for gc 6.99998843276741days in the future
Sep 11 09:44:56 mesos-slave-03 mesos-slave[733]: I0911 09:44:56.000236 887 gc.cpp:56] Scheduling '/tmp/mesos/meta/slaves/20150911-055603-169978048-5050-644-S3/frameworks/20150911-055603-169978048-5050-644-0000' for gc 6.99998843239704days in the future
Sep 11 09:44:56 mesos-slave-03 mesos-slave[733]: I0911 09:44:55.999428 889 status_update_manager.cpp:284] Closing status update streams for framework 20150911-055603-169978048-5050-644-0000
Sep 11 09:44:56 mesos-slave-03 mesos-slave[733]: I0911 09:44:56.233333 890 slave.cpp:1244] Got assigned task task.journalnode.journalnode.NodeExecutor.1441964696232 for framework 20150911-055603-169978048-5050-644-0000
Sep 11 09:44:56 mesos-slave-03 mesos-slave[733]: I0911 09:44:56.234222 890 gc.cpp:84] Unscheduling '/tmp/mesos/slaves/20150911-055603-169978048-5050-644-S3/frameworks/20150911-055603-169978048-5050-644-0000' from gc
Sep 11 09:44:56 mesos-slave-03 mesos-slave[733]: I0911 09:44:56.234328 890 gc.cpp:84] Unscheduling '/tmp/mesos/meta/slaves/20150911-055603-169978048-5050-644-S3/frameworks/20150911-055603-169978048-5050-644-0000' from gc
Sep 11 09:44:56 mesos-slave-03 mesos-slave[733]: I0911 09:44:56.234475 889 slave.cpp:1355] Launching task task.journalnode.journalnode.NodeExecutor.1441964696232 for framework 20150911-055603-169978048-5050-644-0000
Sep 11 09:44:56 mesos-slave-03 mesos-slave[733]: I0911 09:44:56.239325 889 slave.cpp:4733] Launching executor executor.journalnode.NodeExecutor.1441964696232 of framework 20150911-055603-169978048-5050-644-0000 with resources cpus(*):0.5; mem(*):256 in work directory '/tmp/mesos/slaves/20150911-055603-169978048-5050-644-S3/frameworks/20150911-055603-169978048-5050-644-0000/executors/executor.journalnode.NodeExecutor.1441964696232/runs/f583dcb6-5a78-4d86-9576-da09527a5f6c'
Sep 11 09:44:56 mesos-slave-03 mesos-slave[733]: I0911 09:44:56.240191 888 docker.cpp:739] No container info found, skipping launch
Sep 11 09:44:56 mesos-slave-03 mesos-slave[733]: I0911 09:44:56.240340 889 slave.cpp:1573] Queuing task 'task.journalnode.journalnode.NodeExecutor.1441964696232' for executor executor.journalnode.NodeExecutor.1441964696232 of framework '20150911-055603-169978048-5050-644-0000
Sep 11 09:44:56 mesos-slave-03 mesos-slave[733]: I0911 09:44:56.240444 886 containerizer.cpp:534] Starting container 'f583dcb6-5a78-4d86-9576-da09527a5f6c' for executor 'executor.journalnode.NodeExecutor.1441964696232' of framework '20150911-055603-169978048-5050-644-0000'
Sep 11 09:44:56 mesos-slave-03 mesos-slave[733]: I0911 09:44:56.242135 888 launcher.cpp:131] Forked child with pid '3289' for container 'f583dcb6-5a78-4d86-9576-da09527a5f6c'
Sep 11 09:44:56 mesos-slave-03 mesos-slave[733]: I0911 09:44:56.242538 888 containerizer.cpp:770] Checkpointing executor's forked pid 3289 to '/tmp/mesos/meta/slaves/20150911-055603-169978048-5050-644-S3/frameworks/20150911-055603-169978048-5050-644-0000/executors/executor.journalnode.NodeExecutor.1441964696232/runs/f583dcb6-5a78-4d86-9576-da09527a5f6c/pids/forked.pid'
Sep 11 09:44:58 mesos-slave-03 mesos-slave[733]: I0911 09:44:58.616075 889 slave.cpp:4179] Querying resource estimator for oversubscribable resources
Sep 11 09:44:58 mesos-slave-03 mesos-slave[733]: I0911 09:44:58.616453 889 slave.cpp:4193] Received oversubscribable resources from the resource estimator
Sep 11 09:45:13 mesos-slave-03 mesos-slave[733]: I0911 09:45:13.617174 885 slave.cpp:4179] Querying resource estimator for oversubscribable resources
Sep 11 09:45:13 mesos-slave-03 mesos-slave[733]: I0911 09:45:13.617477 885 slave.cpp:4193] Received oversubscribable resources from the resource estimator
Sep 11 09:45:28 mesos-slave-03 mesos-slave[733]: I0911 09:45:28.618423 887 slave.cpp:4179] Querying resource estimator for oversubscribable resources
Sep 11 09:45:28 mesos-slave-03 mesos-slave[733]: I0911 09:45:28.618767 887 slave.cpp:4193] Received oversubscribable resources from the resource estimator
Sep 11 09:45:43 mesos-slave-03 mesos-slave[733]: I0911 09:45:43.619093 883 slave.cpp:4179] Querying resource estimator for oversubscribable resources
Sep 11 09:45:43 mesos-slave-03 mesos-slave[733]: I0911 09:45:43.619428 883 slave.cpp:4193] Received oversubscribable resources from the resource estimator
Sep 11 09:45:46 mesos-slave-03 mesos-slave[733]: I0911 09:45:46.047137 887 slave.cpp:3842] Current disk usage 8.82%. Max allowed age: 5.682946585110810days
Sep 11 09:45:56 mesos-slave-03 mesos-slave[733]: I0911 09:45:56.242110 888 slave.cpp:3798] Terminating executor executor.journalnode.NodeExecutor.1441964696232 of framework 20150911-055603-169978048-5050-644-0000 because it did not register within 1mins
Sep 11 09:45:56 mesos-slave-03 mesos-slave[733]: I0911 09:45:56.242458 888 containerizer.cpp:1001] Destroying container 'f583dcb6-5a78-4d86-9576-da09527a5f6c'
Sep 11 09:45:56 mesos-slave-03 mesos-slave[733]: I0911 09:45:56.355291 883 containerizer.cpp:1188] Executor for container 'f583dcb6-5a78-4d86-9576-da09527a5f6c' has exited
Sep 11 09:45:58 mesos-slave-03 mesos-slave[733]: I0911 09:45:58.621155 884 slave.cpp:4179] Querying resource estimator for oversubscribable resources
Sep 11 09:45:58 mesos-slave-03 mesos-slave[733]: I0911 09:45:58.621547 884 slave.cpp:4193] Received oversubscribable resources from the resource estimator
Sep 11 09:46:13 mesos-slave-03 mesos-slave[733]: I0911 09:46:13.623869 883 slave.cpp:4179] Querying resource estimator for oversubscribable resources
Sep 11 09:46:13 mesos-slave-03 mesos-slave[733]: I0911 09:46:13.624193 883 slave.cpp:4193] Received oversubscribable resources from the resource estimator
Sep 11 09:46:27 mesos-slave-03 mesos-slave[733]: E0911 09:46:27.744279 884 slave.cpp:3258] Container 'f583dcb6-5a78-4d86-9576-da09527a5f6c' for executor 'executor.journalnode.NodeExecutor.1441964696232' of framework '20150911-055603-169978048-5050-644-0000' failed to start: Container destroyed during launch
Sep 11 09:46:27 mesos-slave-03 mesos-slave[733]: E0911 09:46:27.744730 884 slave.cpp:3340] Termination of executor 'executor.journalnode.NodeExecutor.1441964696232' of framework '20150911-055603-169978048-5050-644-0000' failed: Unknown container: f583dcb6-5a78-4d86-9576-da09527a5f6c
Sep 11 09:46:27 mesos-slave-03 mesos-slave[733]: W0911 09:46:27.744840 887 composing.cpp:520] Container 'f583dcb6-5a78-4d86-9576-da09527a5f6c' is already destroyed
Sep 11 09:46:27 mesos-slave-03 mesos-slave[733]: I0911 09:46:27.746973 884 slave.cpp:2671] Handling status update TASK_LOST (UUID: 69765d46-b587-47c3-b6d7-789e03081a4c) for task task.journalnode.journalnode.NodeExecutor.1441964696232 of framework 20150911-055603-169978048-5050-644-0000 from @0.0.0.0:0
Sep 11 09:46:27 mesos-slave-03 mesos-slave[733]: W0911 09:46:27.747392 884 containerizer.cpp:885] Ignoring update for unknown container: f583dcb6-5a78-4d86-9576-da09527a5f6c
Sep 11 09:46:27 mesos-slave-03 mesos-slave[733]: I0911 09:46:27.748229 888 status_update_manager.cpp:322] Received status update TASK_LOST (UUID: 69765d46-b587-47c3-b6d7-789e03081a4c) for task task.journalnode.journalnode.NodeExecutor.1441964696232 of framework 20150911-055603-169978048-5050-644-0000
Sep 11 09:46:27 mesos-slave-03 mesos-slave[733]: I0911 09:46:27.748561 888 status_update_manager.cpp:826] Checkpointing UPDATE for status update TASK_LOST (UUID: 69765d46-b587-47c3-b6d7-789e03081a4c) for task task.journalnode.journalnode.NodeExecutor.1441964696232 of framework 20150911-055603-169978048-5050-644-0000
Sep 11 09:46:27 mesos-slave-03 mesos-slave[733]: I0911 09:46:27.753988 888 slave.cpp:2926] Forwarding the update TASK_LOST (UUID: 69765d46-b587-47c3-b6d7-789e03081a4c) for task task.journalnode.journalnode.NodeExecutor.1441964696232 of framework 20150911-055603-169978048-5050-644-0000 to master@192.168.33.10:5050
Sep 11 09:46:27 mesos-slave-03 mesos-slave[733]: I0911 09:46:27.780983 888 status_update_manager.cpp:394] Received status update acknowledgement (UUID: 69765d46-b587-47c3-b6d7-789e03081a4c) for task task.journalnode.journalnode.NodeExecutor.1441964696232 of framework 20150911-055603-169978048-5050-644-0000
Sep 11 09:46:27 mesos-slave-03 mesos-slave[733]: I0911 09:46:27.781189 888 status_update_manager.cpp:826] Checkpointing ACK for status update TASK_LOST (UUID: 69765d46-b587-47c3-b6d7-789e03081a4c) for task task.journalnode.journalnode.NodeExecutor.1441964696232 of framework 20150911-055603-169978048-5050-644-0000
Sep 11 09:46:27 mesos-slave-03 mesos-slave[733]: I0911 09:46:27.783129 888 slave.cpp:3460] Cleaning up executor 'executor.journalnode.NodeExecutor.1441964696232' of framework 20150911-055603-169978048-5050-644-0000
Sep 11 09:46:27 mesos-slave-03 mesos-slave[733]: I0911 09:46:27.783387 883 gc.cpp:56] Scheduling '/tmp/mesos/slaves/20150911-055603-169978048-5050-644-S3/frameworks/20150911-055603-169978048-5050-644-0000/executors/executor.journalnode.NodeExecutor.1441964696232/runs/f583dcb6-5a78-4d86-9576-da09527a5f6c' for gc 6.99999093409778days in the future
Sep 11 09:46:27 mesos-slave-03 mesos-slave[733]: I0911 09:46:27.783429 888 slave.cpp:3549] Cleaning up framework 20150911-055603-169978048-5050-644-0000
Sep 11 09:46:27 mesos-slave-03 mesos-slave[733]: I0911 09:46:27.783458 883 gc.cpp:56] Scheduling '/tmp/mesos/slaves/20150911-055603-169978048-5050-644-S3/frameworks/20150911-055603-169978048-5050-644-0000/executors/executor.journalnode.NodeExecutor.1441964696232' for gc 6.99999093339259days in the future
Sep 11 09:46:27 mesos-slave-03 mesos-slave[733]: I0911 09:46:27.783504 883 gc.cpp:56] Scheduling '/tmp/mesos/meta/slaves/20150911-055603-169978048-5050-644-S3/frameworks/20150911-055603-169978048-5050-644-0000/executors/executor.journalnode.NodeExecutor.1441964696232/runs/f583dcb6-5a78-4d86-9576-da09527a5f6c' for gc 6.99999093307852days in the future
Sep 11 09:46:27 mesos-slave-03 mesos-slave[733]: I0911 09:46:27.783571 883 gc.cpp:56] Scheduling '/tmp/mesos/meta/slaves/20150911-055603-169978048-5050-644-S3/frameworks/20150911-055603-169978048-5050-644-0000/executors/executor.journalnode.NodeExecutor.1441964696232' for gc 6.99999093265185days in the future
Sep 11 09:46:27 mesos-slave-03 mesos-slave[733]: I0911 09:46:27.783618 883 gc.cpp:56] Scheduling '/tmp/mesos/slaves/20150911-055603-169978048-5050-644-S3/frameworks/20150911-055603-169978048-5050-644-0000' for gc 6.99999093189926days in the future
Sep 11 09:46:27 mesos-slave-03 mesos-slave[733]: I0911 09:46:27.783735 883 gc.cpp:56] Scheduling '/tmp/mesos/meta/slaves/20150911-055603-169978048-5050-644-S3/frameworks/20150911-055603-169978048-5050-644-0000' for gc 6.99999093167704days in the future
Sep 11 09:46:27 mesos-slave-03 mesos-slave[733]: I0911 09:46:27.783519 888 status_update_manager.cpp:284] Closing status update streams for framework 20150911-055603-169978048-5050-644-0000
Sep 11 09:46:28 mesos-slave-03 mesos-slave[733]: I0911 09:46:28.625985 883 slave.cpp:4179] Querying resource estimator for oversubscribable resources
Sep 11 09:46:28 mesos-slave-03 mesos-slave[733]: I0911 09:46:28.626142 883 slave.cpp:4193] Received oversubscribable resources from the resource estimator
Sep 11 09:46:29 mesos-slave-03 mesos-slave[733]: I0911 09:46:29.466192 887 slave.cpp:1244] Got assigned task task.journalnode.journalnode.NodeExecutor.1441964789479 for framework 20150911-055603-169978048-5050-644-0000
Sep 11 09:46:29 mesos-slave-03 mesos-slave[733]: I0911 09:46:29.468091 887 gc.cpp:84] Unscheduling '/tmp/mesos/slaves/20150911-055603-169978048-5050-644-S3/frameworks/20150911-055603-169978048-5050-644-0000' from gc
Sep 11 09:46:29 mesos-slave-03 mesos-slave[733]: I0911 09:46:29.468191 887 gc.cpp:84] Unscheduling '/tmp/mesos/meta/slaves/20150911-055603-169978048-5050-644-S3/frameworks/20150911-055603-169978048-5050-644-0000' from gc
Sep 11 09:46:29 mesos-slave-03 mesos-slave[733]: I0911 09:46:29.468298 887 slave.cpp:1355] Launching task task.journalnode.journalnode.NodeExecutor.1441964789479 for framework 20150911-055603-169978048-5050-644-0000
Sep 11 09:46:29 mesos-slave-03 mesos-slave[733]: I0911 09:46:29.476373 887 slave.cpp:4733] Launching executor executor.journalnode.NodeExecutor.1441964789479 of framework 20150911-055603-169978048-5050-644-0000 with resources cpus(*):0.5; mem(*):256 in work directory '/tmp/mesos/slaves/20150911-055603-169978048-5050-644-S3/frameworks/20150911-055603-169978048-5050-644-0000/executors/executor.journalnode.NodeExecutor.1441964789479/runs/d8109757-6f3b-46ca-90a9-eda87ce25553'
Sep 11 09:46:29 mesos-slave-03 mesos-slave[733]: I0911 09:46:29.477032 887 slave.cpp:1573] Queuing task 'task.journalnode.journalnode.NodeExecutor.1441964789479' for executor executor.journalnode.NodeExecutor.1441964789479 of framework '20150911-055603-169978048-5050-644-0000
Sep 11 09:46:29 mesos-slave-03 mesos-slave[733]: I0911 09:46:29.477267 887 docker.cpp:739] No container info found, skipping launch
Sep 11 09:46:29 mesos-slave-03 mesos-slave[733]: I0911 09:46:29.477401 887 containerizer.cpp:534] Starting container 'd8109757-6f3b-46ca-90a9-eda87ce25553' for executor 'executor.journalnode.NodeExecutor.1441964789479' of framework '20150911-055603-169978048-5050-644-0000'
Sep 11 09:46:29 mesos-slave-03 mesos-slave[733]: I0911 09:46:29.478739 884 launcher.cpp:131] Forked child with pid '3320' for container 'd8109757-6f3b-46ca-90a9-eda87ce25553'
Sep 11 09:46:29 mesos-slave-03 mesos-slave[733]: I0911 09:46:29.479094 884 containerizer.cpp:770] Checkpointing executor's forked pid 3320 to '/tmp/mesos/meta/slaves/20150911-055603-169978048-5050-644-S3/frameworks/20150911-055603-169978048-5050-644-0000/executors/executor.journalnode.NodeExecutor.1441964789479/runs/d8109757-6f3b-46ca-90a9-eda87ce25553/pids/forked.pid'
Sep 11 09:46:36 mesos-slave-03 mesos-slave[733]: 2015-09-11 09:46:36,349:638(0x7ff2e9dd6700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 12ms
Sep 11 09:46:43 mesos-slave-03 mesos-slave[733]: I0911 09:46:43.626682 888 slave.cpp:4179] Querying resource estimator for oversubscribable resources
Sep 11 09:46:43 mesos-slave-03 mesos-slave[733]: I0911 09:46:43.627041 888 slave.cpp:4193] Received oversubscribable resources from the resource estimator
Sep 11 09:46:46 mesos-slave-03 mesos-slave[733]: I0911 09:46:46.048861 887 slave.cpp:3842] Current disk usage 9.50%. Max allowed age: 5.634927629665150days
stdout
is empty and stderr
just contains some log entries showing the journal executor being downloaded and extracted:
I0911 09:43:15.941109 3249 fetcher.cpp:409] Fetcher Info: {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/20150911-055603-169978048-5050-644-S3\/root","items":[{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"http:\/\/192.168.33.12:8765\/hdfs-mesos-executor-0.1.3.tgz"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"http:\/\/192.168.33.12:8765\/hdfs-site.xml"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"https:\/\/downloads.mesosphere.io\/java\/jre-7u76-linux-x64.tar.gz"}}],"sandbox_directory":"\/tmp\/mesos\/slaves\/20150911-055603-169978048-5050-644-S3\/frameworks\/20150911-055603-169978048-5050-644-0000\/executors\/executor.journalnode.NodeExecutor.1441964595884\/runs\/fecea3fc-87f2-404b-943f-08230a1f7938","user":"root"}
I0911 09:43:15.943086 3249 fetcher.cpp:364] Fetching URI 'http://192.168.33.12:8765/hdfs-mesos-executor-0.1.3.tgz'
I0911 09:43:15.943120 3249 fetcher.cpp:238] Fetching directly into the sandbox directory
I0911 09:43:15.943151 3249 fetcher.cpp:176] Fetching URI 'http://192.168.33.12:8765/hdfs-mesos-executor-0.1.3.tgz'
I0911 09:43:15.943177 3249 fetcher.cpp:126] Downloading resource from 'http://192.168.33.12:8765/hdfs-mesos-executor-0.1.3.tgz' to '/tmp/mesos/slaves/20150911-055603-169978048-5050-644-S3/frameworks/20150911-055603-169978048-5050-644-0000/executors/executor.journalnode.NodeExecutor.1441964595884/runs/fecea3fc-87f2-404b-943f-08230a1f7938/hdfs-mesos-executor-0.1.3.tgz'
I0911 09:43:17.369046 3249 fetcher.cpp:76] Extracting with command: tar -C '/tmp/mesos/slaves/20150911-055603-169978048-5050-644-S3/frameworks/20150911-055603-169978048-5050-644-0000/executors/executor.journalnode.NodeExecutor.1441964595884/runs/fecea3fc-87f2-404b-943f-08230a1f7938' -xf '/tmp/mesos/slaves/20150911-055603-169978048-5050-644-S3/frameworks/20150911-055603-169978048-5050-644-0000/executors/executor.journalnode.NodeExecutor.1441964595884/runs/fecea3fc-87f2-404b-943f-08230a1f7938/hdfs-mesos-executor-0.1.3.tgz'
I0911 09:43:18.306526 3249 fetcher.cpp:84] Extracted '/tmp/mesos/slaves/20150911-055603-169978048-5050-644-S3/frameworks/20150911-055603-169978048-5050-644-0000/executors/executor.journalnode.NodeExecutor.1441964595884/runs/fecea3fc-87f2-404b-943f-08230a1f7938/hdfs-mesos-executor-0.1.3.tgz' into '/tmp/mesos/slaves/20150911-055603-169978048-5050-644-S3/frameworks/20150911-055603-169978048-5050-644-0000/executors/executor.journalnode.NodeExecutor.1441964595884/runs/fecea3fc-87f2-404b-943f-08230a1f7938'
W0911 09:43:18.306586 3249 fetcher.cpp:260] Copying instead of extracting resource from URI with 'extract' flag, because it does not seem to be an archive: http://192.168.33.12:8765/hdfs-mesos-executor-0.1.3.tgz
I0911 09:43:18.306607 3249 fetcher.cpp:441] Fetched 'http://192.168.33.12:8765/hdfs-mesos-executor-0.1.3.tgz' to '/tmp/mesos/slaves/20150911-055603-169978048-5050-644-S3/frameworks/20150911-055603-169978048-5050-644-0000/executors/executor.journalnode.NodeExecutor.1441964595884/runs/fecea3fc-87f2-404b-943f-08230a1f7938/hdfs-mesos-executor-0.1.3.tgz'
I0911 09:43:18.306622 3249 fetcher.cpp:364] Fetching URI 'http://192.168.33.12:8765/hdfs-site.xml'
I0911 09:43:18.306656 3249 fetcher.cpp:238] Fetching directly into the sandbox directory
I0911 09:43:18.306704 3249 fetcher.cpp:176] Fetching URI 'http://192.168.33.12:8765/hdfs-site.xml'
I0911 09:43:18.306733 3249 fetcher.cpp:126] Downloading resource from 'http://192.168.33.12:8765/hdfs-site.xml' to '/tmp/mesos/slaves/20150911-055603-169978048-5050-644-S3/frameworks/20150911-055603-169978048-5050-644-0000/executors/executor.journalnode.NodeExecutor.1441964595884/runs/fecea3fc-87f2-404b-943f-08230a1f7938/hdfs-site.xml'
W0911 09:43:18.316086 3249 fetcher.cpp:260] Copying instead of extracting resource from URI with 'extract' flag, because it does not seem to be an archive: http://192.168.33.12:8765/hdfs-site.xml
I0911 09:43:18.316125 3249 fetcher.cpp:441] Fetched 'http://192.168.33.12:8765/hdfs-site.xml' to '/tmp/mesos/slaves/20150911-055603-169978048-5050-644-S3/frameworks/20150911-055603-169978048-5050-644-0000/executors/executor.journalnode.NodeExecutor.1441964595884/runs/fecea3fc-87f2-404b-943f-08230a1f7938/hdfs-site.xml'
I0911 09:43:18.316148 3249 fetcher.cpp:364] Fetching URI 'https://downloads.mesosphere.io/java/jre-7u76-linux-x64.tar.gz'
I0911 09:43:18.316161 3249 fetcher.cpp:238] Fetching directly into the sandbox directory
I0911 09:43:18.316180 3249 fetcher.cpp:176] Fetching URI 'https://downloads.mesosphere.io/java/jre-7u76-linux-x64.tar.gz'
I0911 09:43:18.316197 3249 fetcher.cpp:126] Downloading resource from 'https://downloads.mesosphere.io/java/jre-7u76-linux-x64.tar.gz' to '/tmp/mesos/slaves/20150911-055603-169978048-5050-644-S3/frameworks/20150911-055603-169978048-5050-644-0000/executors/executor.journalnode.NodeExecutor.1441964595884/runs/fecea3fc-87f2-404b-943f-08230a1f7938/jre-7u76-linux-x64.tar.gz'
I0911 09:44:54.110146 3249 fetcher.cpp:76] Extracting with command: tar -C '/tmp/mesos/slaves/20150911-055603-169978048-5050-644-S3/frameworks/20150911-055603-169978048-5050-644-0000/executors/executor.journalnode.NodeExecutor.1441964595884/runs/fecea3fc-87f2-404b-943f-08230a1f7938' -xf '/tmp/mesos/slaves/20150911-055603-169978048-5050-644-S3/frameworks/20150911-055603-169978048-5050-644-0000/executors/executor.journalnode.NodeExecutor.1441964595884/runs/fecea3fc-87f2-404b-943f-08230a1f7938/jre-7u76-linux-x64.tar.gz'
I0911 09:44:55.855676 3249 fetcher.cpp:84] Extracted '/tmp/mesos/slaves/20150911-055603-169978048-5050-644-S3/frameworks/20150911-055603-169978048-5050-644-0000/executors/executor.journalnode.NodeExecutor.1441964595884/runs/fecea3fc-87f2-404b-943f-08230a1f7938/jre-7u76-linux-x64.tar.gz' into '/tmp/mesos/slaves/20150911-055603-169978048-5050-644-S3/frameworks/20150911-055603-169978048-5050-644-0000/executors/executor.journalnode.NodeExecutor.1441964595884/runs/fecea3fc-87f2-404b-943f-08230a1f7938'
W0911 09:44:55.855677 3249 fetcher.cpp:260] Copying instead of extracting resource from URI with 'extract' flag, because it does not seem to be an archive: https://downloads.mesosphere.io/java/jre-7u76-linux-x64.tar.gz
I0911 09:44:55.855731 3249 fetcher.cpp:441] Fetched 'https://downloads.mesosphere.io/java/jre-7u76-linux-x64.tar.gz' to '/tmp/mesos/slaves/20150911-055603-169978048-5050-644-S3/frameworks/20150911-055603-169978048-5050-644-0000/executors/executor.journalnode.NodeExecutor.1441964595884/runs/fecea3fc-87f2-404b-943f-08230a1f7938/jre-7u76-linux-x64.tar.gz'
Hm, I have an ubuntu cluster currently running correctly, but without oversubscription. Still appears to be some kind of containerizer issue related to oversubscription. I have seen issues with the oversubscription module interacting with the containerizer. The other possibility is a bug similar to https://issues.apache.org/jira/browse/MESOS-2601 or https://issues.apache.org/jira/browse/MESOS-2605, but those were fixed in Mesos 0.23.0
Sep 11 09:45:56 mesos-slave-03 mesos-slave[733]: I0911 09:45:56.242110 888 slave.cpp:3798] Terminating executor executor.journalnode.NodeExecutor.1441964696232 of framework 20150911-055603-169978048-5050-644-0000 because it did not register within 1mins
Which version of Ubuntu, mesos and marathon are you running? Also, which version of hdfs are you using? I've been building the latest HEAD
(692b8b732fa1f14db41441a2ed78cd4baf058e3b yesterday).
I will try and set up an identical cluster to yours and see if I can get it to work.
There are a couple of pretty easy set ups I usually run:
1) http://docs.mesosphere.com/ (DCOS). Very simple. Just dcos package install hdfs
.
2) google.mesosphere.com does a preinstalled Mesos and Marathon install (GCE with debian or ubuntu images - I recommend the images with at least 4 CPU's).
Uninstall the existing that is preinstalled HDFS with the following command, aptitude purge hadoop hadoop-yarn hadoop-hdfs hadoop-hdfs-namenode hadoop-hdfs-datanode hadoop-0.20-mapreduce hadoop-0.20-mapreduce-jobtracker hadoop-0.20-mapreduce-tasktracker hadoop-mapreduce rm -rf /etc/hadoop /mnt/hdfs /var/lib/hadoop* /var/log/hadoop*
dpkg -l hadoop* (to verify uninstall)
Before building HDFS-Mesos to run on GCE, I change one configuration value: The only value I typically change is in mesos-site.xml, mesos.hdfs.zkfc.ha.zookeeper.quorum. I change localhost to point to the list of zookeeper nodes (i.e. zookeeper is running on the master so I can use masterip:2181. Second step is to upload the HDFS tarball on the master and run it there.
3) Vagrant here: https://github.com/Banno/vagrant-mesos. We did development of HDFS using this vagrantfile.
I'm very curious about your issue to see if it's related to mesos modules or the containerizer or not. Let's get to the bottom of it and see if we can update the documentation/fix any additional issues. Thanks for your investigation, @F21!
Here's some more testing.
This cluster consists of 4 nodes: 1 master/slave and 3 slaves. Ubuntu is 14.04 64-bit, mesos is 0.22.1 and Marathon is 0.10.1.
OpenJDK said tools.jar
was missing, so I installed Oracle JDK 8 using these instructions: http://www.webupd8.org/2012/09/install-oracle-java-8-in-ubuntu-via-ppa.html
I also built the latest HEAD
of mesos-hdfs and ran it on one of the slaves.
After launching the framework, I can see it registered in the mesos web ui. However, no tasks are being launched.
This is the output of the hdfs framework:
2015-09-14 03:35:47,472:3872(0x7f741affd700):ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.5
2015-09-14 03:35:47,472:3872(0x7f741affd700):ZOO_INFO@log_env@716: Client environment:host.name=mesos-slave-02
2015-09-14 03:35:47,472:3872(0x7f741affd700):ZOO_INFO@log_env@723: Client environment:os.name=Linux
2015-09-14 03:35:47,472:3872(0x7f741affd700):ZOO_INFO@log_env@724: Client environment:os.arch=3.13.0-63-generic
2015-09-14 03:35:47,472:3872(0x7f741affd700):ZOO_INFO@log_env@725: Client environment:os.version=#103-Ubuntu SMP Fri Aug 14 21:42:59 UTC 2015
2015-09-14 03:35:47,472:3872(0x7f741affd700):ZOO_INFO@log_env@733: Client environment:user.name=vagrant
2015-09-14 03:35:47,472:3872(0x7f741affd700):ZOO_INFO@log_env@741: Client environment:user.home=/home/vagrant
2015-09-14 03:35:47,472:3872(0x7f741affd700):ZOO_INFO@log_env@753: Client environment:user.dir=/home/vagrant/hdfs/build/hdfs-mesos-0.1.4
2015-09-14 03:35:47,472:3872(0x7f741affd700):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=master.mesos:2181 sessionTimeout=20000 watcher=0x7f7422e35a60 sessionId=0 sessionPasswd=<null> context=0x7f7400000bf0 flags=0
2015-09-14 03:35:47,477:3872(0x7f7417ddc700):ZOO_INFO@check_events@1703: initiated connection to server [192.168.33.10:2181]
2015-09-14 03:35:47,485:3872(0x7f7417ddc700):ZOO_INFO@check_events@1750: session establishment complete on server [192.168.33.10:2181], sessionId=0x14fc962e269000a, negotiated timeout=20000
03:35:47.521 [HdfsScheduler] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Registering without authentication
2015-09-14 03:35:47,530:3872(0x7f741b7fe700):ZOO_INFO@log_env@712: Client environment:zookeeper.version=zookeeper C client 3.4.5
2015-09-14 03:35:47,530:3872(0x7f741b7fe700):ZOO_INFO@log_env@716: Client environment:host.name=mesos-slave-02
2015-09-14 03:35:47,530:3872(0x7f741b7fe700):ZOO_INFO@log_env@723: Client environment:os.name=Linux
2015-09-14 03:35:47,530:3872(0x7f741b7fe700):ZOO_INFO@log_env@724: Client environment:os.arch=3.13.0-63-generic
2015-09-14 03:35:47,530:3872(0x7f741b7fe700):ZOO_INFO@log_env@725: Client environment:os.version=#103-Ubuntu SMP Fri Aug 14 21:42:59 UTC 2015
I0914 03:35:47.530618 3902 sched.cpp:157] Version: 0.22.1
2015-09-14 03:35:47,531:3872(0x7f741b7fe700):ZOO_INFO@log_env@733: Client environment:user.name=vagrant
2015-09-14 03:35:47,531:3872(0x7f741b7fe700):ZOO_INFO@log_env@741: Client environment:user.home=/home/vagrant
2015-09-14 03:35:47,531:3872(0x7f741b7fe700):ZOO_INFO@log_env@753: Client environment:user.dir=/home/vagrant/hdfs/build/hdfs-mesos-0.1.4
2015-09-14 03:35:47,531:3872(0x7f741b7fe700):ZOO_INFO@zookeeper_init@786: Initiating client connection, host=master.mesos:2181 sessionTimeout=10000 watcher=0x7f7422e35a60 sessionId=0 sessionPasswd=<null> context=0x7f740c001230 flags=0
03:35:47.535 [main] INFO org.eclipse.jetty.util.log - Logging initialized @925ms
2015-09-14 03:35:47,535:3872(0x7f7416ad0700):ZOO_INFO@check_events@1703: initiated connection to server [192.168.33.10:2181]
2015-09-14 03:35:47,538:3872(0x7f7416ad0700):ZOO_INFO@check_events@1750: session establishment complete on server [192.168.33.10:2181], sessionId=0x14fc962e269000b, negotiated timeout=10000
I0914 03:35:47.538872 3892 group.cpp:313] Group process (group(1)@192.168.33.12:38168) connected to ZooKeeper
I0914 03:35:47.538956 3892 group.cpp:790] Syncing group operations: queue size (joins, cancels, datas) = (0, 0, 0)
I0914 03:35:47.538993 3892 group.cpp:385] Trying to create path '/mesos' in ZooKeeper
I0914 03:35:47.544342 3892 detector.cpp:138] Detected a new leader: (id='3')
I0914 03:35:47.544850 3894 group.cpp:659] Trying to get '/mesos/info_0000000003' in ZooKeeper
I0914 03:35:47.546298 3894 detector.cpp:452] A new leading master (UPID=master@192.168.33.10:5050) is detected
I0914 03:35:47.546525 3893 sched.cpp:254] New master detected at master@192.168.33.10:5050
I0914 03:35:47.546869 3893 sched.cpp:264] No credentials provided. Attempting to register without authentication
I0914 03:35:47.549916 3896 sched.cpp:448] Framework registered with 20150914-010557-169978048-5050-2050-0000
03:35:47.558 [Thread-2] INFO o.a.m.hdfs.scheduler.HdfsScheduler - Registered framework frameworkId=20150914-010557-169978048-5050-2050-0000
03:35:47.559 [Thread-2] INFO o.apache.mesos.hdfs.state.LiveState - Acquisition phase is already 'RECONCILING_TASKS'
03:35:47.589 [main] INFO org.eclipse.jetty.server.Server - jetty-9.2.z-SNAPSHOT
03:35:47.631 [main] INFO o.e.jetty.server.ServerConnector - Started ServerConnector@1d082e88{HTTP/1.1}{0.0.0.0:8765}
03:35:47.632 [main] INFO org.eclipse.jetty.server.Server - Started @1032ms
Noticed there was a bug in mesos 0.22.1 that didn't play well with docker 1.8.1:
Sep 14 04:33:55 mesos-slave-02 mesos-slave[2195]: Failed to create a containerizer: Could not create DockerContainerizer: Insufficient version of Docker! Please upgrade to >= 1.0.0
Will downgrade the docker version and report back.
Thanks @F21, we can also discuss by email or IRC to debug what the issue is with the Mesos cluster or the framework.
@elingg Here's my latesting findings:
Mesos is 0.22.1 Ubuntu is 14.04 64-bit and Marathon is 0.10.1. Docker is 1.7.1.
Still seeing the same issue where the journal node fails to launch:
Sep 15 00:46:53 mesos-master-01 mesos-master[2015]: I0915 00:46:53.326689 2080 master.cpp:2273] Processing ACCEPT call for offers: [ 20150914-234602-169978048-5050-2015-O2691 ] on slave 20150914-234602-169978048-5050-2015-S0 at slave(1)@192.168.33.10:5051 (mesos-master-01) for framework 20150914-234602-169978048-5050-2015-0000 (hdfs) at scheduler-32bcebe3-ec7c-4c6f-bed8-5a06a332624a@192.168.33.11:40392
Sep 15 00:46:53 mesos-master-01 mesos-master[2015]: I0915 00:46:53.326963 2080 hierarchical.hpp:648] Recovered cpus(*):2.4; mem(*):212; disk(*):35164; ports(*):[31000-31565, 31567-32000] (total allocatable: cpus(*):2.4; mem(*):212; disk(*):35164; ports(*):[31000-31565, 31567-32000]) on slave 20150914-234602-169978048-5050-2015-S0 from framework 20150914-234602-169978048-5050-2015-0000
Sep 15 00:46:53 mesos-master-01 marathon[1148]: [2015-09-15 00:46:53,325] INFO started processing 2 offers, launching at most 1 tasks per offer and 1000 tasks in total (mesosphere.marathon.tasks.IterativeOfferMatcher$:132)
Sep 15 00:46:53 mesos-master-01 mesos-master[2015]: I0915 00:46:53.327729 2077 master.cpp:2273] Processing ACCEPT call for offers: [ 20150914-234602-169978048-5050-2015-O2692 ] on slave 20150914-234602-169978048-5050-2015-S1 at slave(1)@192.168.33.11:5051 (mesos-slave-01) for framework 20150914-234602-169978048-5050-2015-0000 (hdfs) at scheduler-32bcebe3-ec7c-4c6f-bed8-5a06a332624a@192.168.33.11:40392
Sep 15 00:46:53 mesos-master-01 mesos-master[2015]: I0915 00:46:53.328044 2077 hierarchical.hpp:648] Recovered cpus(*):4; mem(*):1000; disk(*):35164; ports(*):[31000-32000] (total allocatable: cpus(*):4; mem(*):1000; disk(*):35164; ports(*):[31000-32000]) on slave 20150914-234602-169978048-5050-2015-S1 from framework 20150914-234602-169978048-5050-2015-0000
Sep 15 00:46:53 mesos-master-01 marathon[1148]: [2015-09-15 00:46:53,326] INFO Offer [20150914-234602-169978048-5050-2015-O2693]. Decline with default filter refuseSeconds (use --decline_offer_duration to configure) (mesosphere.marathon.tasks.IterativeOfferMatcher$:231)
Sep 15 00:46:53 mesos-master-01 marathon[1148]: [2015-09-15 00:46:53,327] INFO Offer [20150914-234602-169978048-5050-2015-O2694]. Decline with default filter refuseSeconds (use --decline_offer_duration to configure) (mesosphere.marathon.tasks.IterativeOfferMatcher$:231)
Sep 15 00:46:53 mesos-master-01 marathon[1148]: [2015-09-15 00:46:53,327] INFO Launched 0 tasks on 0 offers, declining 2 (mesosphere.marathon.tasks.IterativeOfferMatcher$:241)
Sep 15 00:46:53 mesos-master-01 mesos-master[2015]: I0915 00:46:53.329223 2074 master.cpp:2273] Processing ACCEPT call for offers: [ 20150914-234602-169978048-5050-2015-O2693 ] on slave 20150914-234602-169978048-5050-2015-S2 at slave(1)@192.168.33.12:5051 (mesos-slave-02) for framework 20150914-221832-169978048-5050-10929-0000 (marathon) at scheduler-8717a9c8-2e0e-43d4-82c5-4200d4d04cb1@192.168.33.10:52790
Sep 15 00:46:53 mesos-master-01 mesos-master[2015]: I0915 00:46:53.329744 2074 hierarchical.hpp:648] Recovered cpus(*):4; mem(*):1000; disk(*):35164; ports(*):[31000-32000] (total allocatable: cpus(*):4; mem(*):1000; disk(*):35164; ports(*):[31000-32000]) on slave 20150914-234602-169978048-5050-2015-S2 from framework 20150914-221832-169978048-5050-10929-0000
Sep 15 00:46:53 mesos-master-01 mesos-master[2015]: I0915 00:46:53.330298 2080 master.cpp:2273] Processing ACCEPT call for offers: [ 20150914-234602-169978048-5050-2015-O2694 ] on slave 20150914-234602-169978048-5050-2015-S3 at slave(1)@192.168.33.13:5051 (mesos-slave-03) for framework 20150914-221832-169978048-5050-10929-0000 (marathon) at scheduler-8717a9c8-2e0e-43d4-82c5-4200d4d04cb1@192.168.33.10:52790
Sep 15 00:46:53 mesos-master-01 mesos-master[2015]: I0915 00:46:53.330484 2080 hierarchical.hpp:648] Recovered cpus(*):4; mem(*):1000; disk(*):35164; ports(*):[31000-32000] (total allocatable: cpus(*):4; mem(*):1000; disk(*):35164; ports(*):[31000-32000]) on slave 20150914-234602-169978048-5050-2015-S3 from framework 20150914-221832-169978048-5050-10929-0000
Sep 15 00:46:54 mesos-master-01 mesos-master[2015]: I0915 00:46:54.317855 2077 master.cpp:3760] Sending 2 offers to framework 20150914-234602-169978048-5050-2015-0000 (hdfs) at scheduler-32bcebe3-ec7c-4c6f-bed8-5a06a332624a@192.168.33.11:40392
Sep 15 00:46:54 mesos-master-01 mesos-master[2015]: I0915 00:46:54.330700 2079 master.cpp:2273] Processing ACCEPT call for offers: [ 20150914-234602-169978048-5050-2015-O2695 ] on slave 20150914-234602-169978048-5050-2015-S2 at slave(1)@192.168.33.12:5051 (mesos-slave-02) for framework 20150914-234602-169978048-5050-2015-0000 (hdfs) at scheduler-32bcebe3-ec7c-4c6f-bed8-5a06a332624a@192.168.33.11:40392
Sep 15 00:46:54 mesos-master-01 mesos-master[2015]: I0915 00:46:54.331167 2079 hierarchical.hpp:648] Recovered cpus(*):4; mem(*):1000; disk(*):35164; ports(*):[31000-32000] (total allocatable: cpus(*):4; mem(*):1000; disk(*):35164; ports(*):[31000-32000]) on slave 20150914-234602-169978048-5050-2015-S2 from framework 20150914-234602-169978048-5050-2015-0000
Sep 15 00:46:54 mesos-master-01 mesos-master[2015]: I0915 00:46:54.331682 2066 master.cpp:2273] Processing ACCEPT call for offers: [ 20150914-234602-169978048-5050-2015-O2696 ] on slave 20150914-234602-169978048-5050-2015-S3 at slave(1)@192.168.33.13:5051 (mesos-slave-03) for framework 20150914-234602-169978048-5050-2015-0000 (hdfs) at scheduler-32bcebe3-ec7c-4c6f-bed8-5a06a332624a@192.168.33.11:40392
Sep 15 00:46:54 mesos-master-01 mesos-master[2015]: I0915 00:46:54.332104 2066 hierarchical.hpp:648] Recovered cpus(*):4; mem(*):1000; disk(*):35164; ports(*):[31000-32000] (total allocatable: cpus(*):4; mem(*):1000; disk(*):35164; ports(*):[31000-32000]) on slave 20150914-234602-169978048-5050-2015-S3 from framework 20150914-234602-169978048-5050-2015-0000
Sep 15 00:46:55 mesos-master-01 mesos-slave[2019]: W0915 00:46:55.861906 2075 containerizer.cpp:907] Ignoring destroy of unknown container: df865edb-c019-4f1a-8a70-1cee56a4f792
Sep 15 00:46:55 mesos-master-01 mesos-slave[2019]: E0915 00:46:55.862077 2068 slave.cpp:3112] Container 'df865edb-c019-4f1a-8a70-1cee56a4f792' for executor 'executor.journalnode.NodeExecutor.1442277857378' of framework '20150914-234602-169978048-5050-2015-0000' failed to start: Container destroyed during launch
Sep 15 00:46:55 mesos-master-01 mesos-slave[2019]: E0915 00:46:55.862442 2068 slave.cpp:3207] Termination of executor 'executor.journalnode.NodeExecutor.1442277857378' of framework '20150914-234602-169978048-5050-2015-0000' failed: Unknown container: df865edb-c019-4f1a-8a70-1cee56a4f792
Sep 15 00:46:55 mesos-master-01 mesos-slave[2019]: E0915 00:46:55.862818 2073 slave.cpp:3461] Failed to unmonitor container for executor executor.journalnode.NodeExecutor.1442277857378 of framework 20150914-234602-169978048-5050-2015-0000: Not monitored
Sep 15 00:46:55 mesos-master-01 mesos-slave[2019]: I0915 00:46:55.863688 2068 slave.cpp:2531] Handling status update TASK_LOST (UUID: 826824c1-f1f0-4a8d-a07f-e841c3b402d5) for task task.journalnode.journalnode.NodeExecutor.1442277857378 of framework 20150914-234602-169978048-5050-2015-0000 from @0.0.0.0:0
Sep 15 00:46:55 mesos-master-01 mesos-slave[2019]: W0915 00:46:55.863901 2068 containerizer.cpp:814] Ignoring update for unknown container: df865edb-c019-4f1a-8a70-1cee56a4f792
Sep 15 00:46:55 mesos-master-01 mesos-slave[2019]: I0915 00:46:55.863982 2068 status_update_manager.cpp:317] Received status update TASK_LOST (UUID: 826824c1-f1f0-4a8d-a07f-e841c3b402d5) for task task.journalnode.journalnode.NodeExecutor.1442277857378 of framework 20150914-234602-169978048-5050-2015-0000
Sep 15 00:46:55 mesos-master-01 mesos-slave[2019]: I0915 00:46:55.864219 2068 status_update_manager.hpp:346] Checkpointing UPDATE for status update TASK_LOST (UUID: 826824c1-f1f0-4a8d-a07f-e841c3b402d5) for task task.journalnode.journalnode.NodeExecutor.1442277857378 of framework 20150914-234602-169978048-5050-2015-0000
Sep 15 00:46:55 mesos-master-01 mesos-master[2015]: I0915 00:46:55.864292 2066 master.cpp:3393] Executor executor.journalnode.NodeExecutor.1442277857378 of framework 20150914-234602-169978048-5050-2015-0000 on slave 20150914-234602-169978048-5050-2015-S0 at slave(1)@192.168.33.10:5051 (mesos-master-01) terminated with signal Unknown signal 127
Sep 15 00:46:55 mesos-master-01 mesos-master[2015]: I0915 00:46:55.864419 2066 master.cpp:4719] Removing executor 'executor.journalnode.NodeExecutor.1442277857378' with resources cpus(*):0.5; mem(*):256 of framework 20150914-234602-169978048-5050-2015-0000 on slave 20150914-234602-169978048-5050-2015-S0 at slave(1)@192.168.33.10:5051 (mesos-master-01)
Sep 15 00:46:55 mesos-master-01 mesos-master[2015]: I0915 00:46:55.864542 2066 hierarchical.hpp:648] Recovered cpus(*):0.5; mem(*):256 (total allocatable: cpus(*):2.9; mem(*):468; disk(*):35164; ports(*):[31000-31565, 31567-32000]) on slave 20150914-234602-169978048-5050-2015-S0 from framework 20150914-234602-169978048-5050-2015-0000
Sep 15 00:46:55 mesos-master-01 mesos-slave[2019]: I0915 00:46:55.867033 2068 slave.cpp:2776] Forwarding the update TASK_LOST (UUID: 826824c1-f1f0-4a8d-a07f-e841c3b402d5) for task task.journalnode.journalnode.NodeExecutor.1442277857378 of framework 20150914-234602-169978048-5050-2015-0000 to master@192.168.33.10:5050
Sep 15 00:46:55 mesos-master-01 mesos-master[2015]: I0915 00:46:55.867727 2072 master.cpp:3300] Status update TASK_LOST (UUID: 826824c1-f1f0-4a8d-a07f-e841c3b402d5) for task task.journalnode.journalnode.NodeExecutor.1442277857378 of framework 20150914-234602-169978048-5050-2015-0000 from slave 20150914-234602-169978048-5050-2015-S0 at slave(1)@192.168.33.10:5051 (mesos-master-01)
Sep 15 00:46:55 mesos-master-01 mesos-master[2015]: I0915 00:46:55.867784 2072 master.cpp:3341] Forwarding status update TASK_LOST (UUID: 826824c1-f1f0-4a8d-a07f-e841c3b402d5) for task task.journalnode.journalnode.NodeExecutor.1442277857378 of framework 20150914-234602-169978048-5050-2015-0000
Sep 15 00:46:55 mesos-master-01 mesos-master[2015]: I0915 00:46:55.867913 2072 master.cpp:4623] Updating the latest state of task task.journalnode.journalnode.NodeExecutor.1442277857378 of framework 20150914-234602-169978048-5050-2015-0000 to TASK_LOST
Sep 15 00:46:55 mesos-master-01 mesos-master[2015]: I0915 00:46:55.868052 2072 hierarchical.hpp:648] Recovered cpus(*):1; mem(*):512 (total allocatable: cpus(*):3.9; mem(*):980; disk(*):35164; ports(*):[31000-31565, 31567-32000]) on slave 20150914-234602-169978048-5050-2015-S0 from framework 20150914-234602-169978048-5050-2015-0000
Sep 15 00:46:55 mesos-master-01 mesos-master[2015]: I0915 00:46:55.890699 2074 master.cpp:4690] Removing task task.journalnode.journalnode.NodeExecutor.1442277857378 with resources cpus(*):1; mem(*):512 of framework 20150914-234602-169978048-5050-2015-0000 on slave 20150914-234602-169978048-5050-2015-S0 at slave(1)@192.168.33.10:5051 (mesos-master-01)
Sep 15 00:46:55 mesos-master-01 mesos-master[2015]: I0915 00:46:55.890844 2074 master.cpp:2787] Forwarding status update acknowledgement 826824c1-f1f0-4a8d-a07f-e841c3b402d5 for task task.journalnode.journalnode.NodeExecutor.1442277857378 of framework 20150914-234602-169978048-5050-2015-0000 (hdfs) at scheduler-32bcebe3-ec7c-4c6f-bed8-5a06a332624a@192.168.33.11:40392 to slave 20150914-234602-169978048-5050-2015-S0 at slave(1)@192.168.33.10:5051 (mesos-master-01)
Sep 15 00:46:55 mesos-master-01 mesos-slave[2019]: I0915 00:46:55.891283 2073 status_update_manager.cpp:389] Received status update acknowledgement (UUID: 826824c1-f1f0-4a8d-a07f-e841c3b402d5) for task task.journalnode.journalnode.NodeExecutor.1442277857378 of framework 20150914-234602-169978048-5050-2015-0000
Sep 15 00:46:55 mesos-master-01 mesos-slave[2019]: I0915 00:46:55.891348 2073 status_update_manager.hpp:346] Checkpointing ACK for status update TASK_LOST (UUID: 826824c1-f1f0-4a8d-a07f-e841c3b402d5) for task task.journalnode.journalnode.NodeExecutor.1442277857378 of framework 20150914-234602-169978048-5050-2015-0000
What's the best way to debug this over IRC? My timezone is AEST (Australia).
Ah, timezones are tricky, but we can set up a time if needed. You can always check with other Mesos users on IRC as well as the mesos user mailing lists to see if you get a response in your timezone. At the same time, we can proceed through github issues. This seems to be a Mesos issue still with the containerizer. See the similar Marathon issue with unknown container, https://github.com/mesosphere/marathon/issues/734. Also, this is a custom executor. Does the launch of the executor exceed the default mesos-slave executor registration timeout? What do you see being downloaded in the executor sandbox?
I am currently removing marathon from the equation and am launching hdfs-mesos
manually via the commandline on one of the nodes.
In terms of the download, I can see the executor and java 7 being downloaded from downloads.mesosphere.io
and extracted into the sandbox.
The executor_registration_timeout
has always been set to a pretty high value, so it shouldn't be a problem.
What is strange is that I got the hdfs framework running back in June or July without too many problems. I will go and build 0.1.2
and see if it works.
If it's a Mesos containerizer issue, downgrading the HDFS-Mesos version probably won't help much, but you can try to help diagnose. Running without Marathon might if you are running marathon in docker or other apps in docker, because the recent containizer issues we've seen occur when the user runs docker containers alongside mesos containers. Since HDFS-Mesos doesn't require Docker, one thing to try is to remove docker from the slave containerizer flags.
I did have docker in my previous set up, but it's a good idea to setup a very minimal mesos cluster with just mesos, mesos-dns and hdfs-mesos to replicate. I think I will also not install marathon as well. All of these are installed from the Mesosphere apt repos.
I'll give that a go and see if it makes a difference.
Just reporting back.
I am still seeing the journal node task being lost.
My cluster is running Ubuntu 14.04 64-bit with mesos dns and the head of HDFS-mesos. Docker and marathon was not installed.
Very strange, not sure why you are seeing containerizer issues then. Still the same container error messages? Sep 15 00:46:55 mesos-master-01 mesos-slave[2019]: W0915 00:46:55.863901 2068 containerizer.cpp:814] Ignoring update for unknown container: df865edb-c019-4f1a-8a70-1cee56a4f792
One thought after double checking your settings is that your memory settings are very low.
<value>256</value>
Perhaps the JN task requires more memory by default to even launch. HDFS does have certain minimums. Could you possibly try bumping up your memory settings? Would you also consider launching on the environments I mentioned above (DCOS, GCE, Vagrant)? That might give us an idea of what is different. You can also certainly try with an older release, but I'm not sure that will help, especially since I'm unable to reproduce your issue. Again, thanks for your investigation!
@elingg I made some progress! Got the JN to launch on my stripped own setup with the HEAD of hdfs-mesos. I noticed that I neglected to set executor_registration_timeout
for it.
Currently, the clusters are launched using vagrant.
Since mesos 0.24 was just released, I am going to build the binaries and setup a CoreOS cluster again to see if it resolves the problem.
@F21, that's great news! executor_registration_timeout is a common issue (which is why I mentioned it above) as the container will be destroyed if the executor is not created in time.
That's awesome that Ubuntu is working. Core OS should be fine, but remember the comment I made about the sym link permissions issue. To get around that, we use the predistributed binaries option.
The strange thing is that I always had the MESOS_EXECUTOR_REGISTRATION_TIMEOUT
environment variable set on all my CoreOS slaves, so it really shouldn't be the cause on CoreOS. I will try adding that to --executor_environment_variables
to see if can get it going on CoreOS.
Is this safe to close @F21 with the MESOS_EXECUTOR_REGISTRATION_TIMEOUT environment variable fix?
@elingg Unfortunately, that didn't make any difference on CoreOS. I am still investigating to see why that's happening.
Understood, let us know if you have questions @F21 and as a product plug, with DCOS, installation on Core OS is as easy as one command :+1:
@elingg Is there anyway to increase the verbosity of the executor?
Sep 16 22:22:28 core-03 mesos-slave[829]: I0916 22:22:28.686725 841 slave.cpp:1249] Got assigned task task.journalnode.journalnode.NodeExecutor.1442442148388 for framework 20150916-220225-169978048-5050-600-0001
Sep 16 22:22:28 core-03 mesos-slave[829]: I0916 22:22:28.689013 841 slave.cpp:1365] Launching task task.journalnode.journalnode.NodeExecutor.1442442148388 for framework 20150916-220225-169978048-5050-600-0001
Sep 16 22:22:28 core-03 mesos-slave[829]: I0916 22:22:28.722630 841 slave.cpp:4799] Launching executor executor.journalnode.NodeExecutor.1442442148388 of framework 20150916-220225-169978048-5050-600-0001 with resources cpus(*):0.5; mem(*):256 in work directory '/tmp/mesos/slaves/20150916-220225-169978048-5050-600-S2/frameworks/20150916-220225-169978048-5050-600-0001/executors/executor.journalnode.NodeExecutor.1442442148388/runs/46f63025-41f1-4bf0-b634-9e46a2b8770e'
Sep 16 22:22:28 core-03 mesos-slave[829]: I0916 22:22:28.725831 846 docker.cpp:739] No container info found, skipping launch
Sep 16 22:22:28 core-03 mesos-slave[829]: I0916 22:22:28.727078 841 slave.cpp:1583] Queuing task 'task.journalnode.journalnode.NodeExecutor.1442442148388' for executor executor.journalnode.NodeExecutor.1442442148388 of framework '20150916-220225-169978048-5050-600-0001
Sep 16 22:22:28 core-03 mesos-slave[829]: I0916 22:22:28.732386 844 containerizer.cpp:633] Starting container '46f63025-41f1-4bf0-b634-9e46a2b8770e' for executor 'executor.journalnode.NodeExecutor.1442442148388' of framework '20150916-220225-169978048-5050-600-0001'
Sep 16 22:22:28 core-03 mesos-slave[829]: I0916 22:22:28.740202 844 launcher.cpp:131] Forked child with pid '1477' for container '46f63025-41f1-4bf0-b634-9e46a2b8770e'
Sep 16 22:22:28 core-03 mesos-slave[829]: I0916 22:22:28.741034 844 containerizer.cpp:855] Checkpointing executor's forked pid 1477 to '/tmp/mesos/meta/slaves/20150916-220225-169978048-5050-600-S2/frameworks/20150916-220225-169978048-5050-600-0001/executors/executor.journalnode.NodeExecutor.1442442148388/runs/46f63025-41f1-4bf0-b634-9e46a2b8770e/pids/forked.pid'
Sep 16 22:22:28 core-03 mesos-slave[829]: I0916 22:22:28.783072 843 containerizer.cpp:1266] Executor for container '46f63025-41f1-4bf0-b634-9e46a2b8770e' has exited
Sep 16 22:22:28 core-03 mesos-slave[829]: I0916 22:22:28.785030 843 containerizer.cpp:1079] Destroying container '46f63025-41f1-4bf0-b634-9e46a2b8770e'
Sep 16 22:22:32 core-03 mesos-slave[829]: 2015-09-16 22:22:32,621:829(0x7faf52c83700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 159ms
Sep 16 22:22:35 core-03 mesos-slave[829]: 2015-09-16 22:22:35,977:829(0x7faf52c83700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 22ms
Sep 16 22:22:39 core-03 mesos-slave[829]: 2015-09-16 22:22:39,322:829(0x7faf52c83700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 13ms
Sep 16 22:22:41 core-03 mesos-slave[829]: I0916 22:22:41.547842 844 http.cpp:174] HTTP GET for /slave(1)/state.json from 192.168.33.1:59577 with User-Agent='Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.93 Safari/537.36'
Sep 16 22:22:42 core-03 mesos-slave[829]: 2015-09-16 22:22:42,708:829(0x7faf52c83700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 51ms
Sep 16 22:22:45 core-03 mesos-slave[829]: I0916 22:22:45.218466 843 slave.cpp:3885] Current disk usage 11.00%. Max allowed age: 5.530020488612940days
Sep 16 22:23:45 core-03 mesos-slave[829]: I0916 22:23:45.228628 845 slave.cpp:3885] Current disk usage 12.02%. Max allowed age: 5.458615492878796days
Sep 16 22:24:12 core-03 mesos-slave[829]: 2015-09-16 22:24:12,989:829(0x7faf52c83700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 166ms
Sep 16 22:24:23 core-03 mesos-slave[829]: 2015-09-16 22:24:23,056:829(0x7faf52c83700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 52ms
Sep 16 22:24:45 core-03 mesos-slave[829]: I0916 22:24:45.232887 846 slave.cpp:3885] Current disk usage 13.88%. Max allowed age: 5.328164058364514days
Sep 16 22:24:49 core-03 mesos-slave[829]: 2015-09-16 22:24:49,816:829(0x7faf52c83700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 56ms
Sep 16 22:25:06 core-03 mesos-slave[829]: 2015-09-16 22:25:06,997:829(0x7faf52c83700):ZOO_WARN@zookeeper_interest@1557: Exceeded deadline by 495ms
Sep 16 22:25:20 core-03 mesos-slave[829]: E0916 22:25:20.980870 845 slave.cpp:3301] Container '46f63025-41f1-4bf0-b634-9e46a2b8770e' for executor 'executor.journalnode.NodeExecutor.1442442148388' of framework '20150916-220225-169978048-5050-600-0001' failed to start: Container destroyed during launch
Sep 16 22:25:20 core-03 mesos-slave[829]: W0916 22:25:20.981760 845 containerizer.cpp:1068] Ignoring destroy of unknown container: 46f63025-41f1-4bf0-b634-9e46a2b8770e
Sep 16 22:25:20 core-03 mesos-slave[829]: E0916 22:25:20.981994 841 slave.cpp:3383] Termination of executor 'executor.journalnode.NodeExecutor.1442442148388' of framework '20150916-220225-169978048-5050-600-0001' failed: Unknown container: 46f63025-41f1-4bf0-b634-9e46a2b8770e
Sep 16 22:25:20 core-03 mesos-slave[829]: I0916 22:25:20.983971 841 slave.cpp:2696] Handling status update TASK_LOST (UUID: c7e443a4-886c-4c97-af71-951247001a79) for task task.journalnode.journalnode.NodeExecutor.1442442148388 of framework 20150916-220225-169978048-5050-600-0001 from @0.0.0.0:0
Sep 16 22:25:20 core-03 mesos-slave[829]: W0916 22:25:20.985064 844 containerizer.cpp:970] Ignoring update for unknown container: 46f63025-41f1-4bf0-b634-9e46a2b8770e
Sep 16 22:25:20 core-03 mesos-slave[829]: I0916 22:25:20.985843 845 status_update_manager.cpp:322] Received status update TASK_LOST (UUID: c7e443a4-886c-4c97-af71-951247001a79) for task task.journalnode.journalnode.NodeExecutor.1442442148388 of framework 20150916-220225-169978048-5050-600-0001
Sep 16 22:25:20 core-03 mesos-slave[829]: I0916 22:25:20.986872 845 status_update_manager.cpp:826] Checkpointing UPDATE for status update TASK_LOST (UUID: c7e443a4-886c-4c97-af71-951247001a79) for task task.journalnode.journalnode.NodeExecutor.1442442148388 of framework 20150916-220225-169978048-5050-600-0001
Sep 16 22:25:20 core-03 mesos-slave[829]: I0916 22:25:20.987951 844 slave.cpp:2975] Forwarding the update TASK_LOST (UUID: c7e443a4-886c-4c97-af71-951247001a79) for task task.journalnode.journalnode.NodeExecutor.1442442148388 of framework 20150916-220225-169978048-5050-600-0001 to master@192.168.33.10:5050
Sep 16 22:25:21 core-03 mesos-slave[829]: I0916 22:25:21.061379 840 status_update_manager.cpp:394] Received status update acknowledgement (UUID: c7e443a4-886c-4c97-af71-951247001a79) for task task.journalnode.journalnode.NodeExecutor.1442442148388 of framework 20150916-220225-169978048-5050-600-0001
Sep 16 22:25:21 core-03 mesos-slave[829]: I0916 22:25:21.061650 840 status_update_manager.cpp:826] Checkpointing ACK for status update TASK_LOST (UUID: c7e443a4-886c-4c97-af71-951247001a79) for task task.journalnode.journalnode.NodeExecutor.1442442148388 of framework 20150916-220225-169978048-5050-600-0001
Sep 16 22:25:21 core-03 mesos-slave[829]: I0916 22:25:21.062326 840 slave.cpp:3503] Cleaning up executor 'executor.journalnode.NodeExecutor.1442442148388' of framework 20150916-220225-169978048-5050-600-0001
Is the executor meant to exit? I think this item in the mesos-slave log looks a bit suspicious:
Sep 16 22:22:28 core-03 mesos-slave[829]: I0916 22:22:28.783072 843 containerizer.cpp:1266] Executor for container '46f63025-41f1-4bf0-b634-9e46a2b8770e' has exited
Interestingly, mesos tries to relaunch the executor on the same node a few times and then the mesos-slave crashes and goes offline:
Sep 16 22:30:45 core-03 mesos-slave[829]: I0916 22:30:45.369068 842 slave.cpp:3885] Current disk usage 88.59%. Max allowed age: 2.363740759884445hrs
Sep 16 22:31:09 core-03 mesos-slave[829]: E0916 22:31:09.222362 843 fetcher.cpp:515] Failed to run mesos-fetcher: Failed to fetch all URIs for container '4788225d-fb42-44fe-b44d-d8bd67f144c9' with exit status: 256
Sep 16 22:31:09 core-03 mesos-slave[829]: E0916 22:31:09.223407 843 slave.cpp:3301] Container '4788225d-fb42-44fe-b44d-d8bd67f144c9' for executor 'executor.journalnode.NodeExecutor.1442442573784' of framework '20150916-220225-169978048-5050-600-0001' failed to start: Failed to fetch all URIs for container '4788225d-fb42-44fe-b44d-d8bd67f144c9' with exit status: 256
Sep 16 22:31:09 core-03 mesos-slave[829]: W0916 22:31:09.223945 843 containerizer.cpp:1068] Ignoring destroy of unknown container: 4788225d-fb42-44fe-b44d-d8bd67f144c9
Sep 16 22:31:09 core-03 mesos-slave[829]: E0916 22:31:09.224094 842 slave.cpp:3383] Termination of executor 'executor.journalnode.NodeExecutor.1442442573784' of framework '20150916-220225-169978048-5050-600-0001' failed: Unknown container: 4788225d-fb42-44fe-b44d-d8bd67f144c9
Sep 16 22:31:09 core-03 mesos-slave[829]: I0916 22:31:09.227615 842 slave.cpp:2696] Handling status update TASK_LOST (UUID: f5f8ce65-88c5-41f0-93ab-41e82022efc5) for task task.journalnode.journalnode.NodeExecutor.1442442573784 of framework 20150916-220225-169978048-5050-600-0001 from @0.0.0.0:0
Sep 16 22:31:09 core-03 mesos-slave[829]: W0916 22:31:09.228691 847 containerizer.cpp:970] Ignoring update for unknown container: 4788225d-fb42-44fe-b44d-d8bd67f144c9
Sep 16 22:31:09 core-03 mesos-slave[829]: I0916 22:31:09.230973 842 status_update_manager.cpp:322] Received status update TASK_LOST (UUID: f5f8ce65-88c5-41f0-93ab-41e82022efc5) for task task.journalnode.journalnode.NodeExecutor.1442442573784 of framework 20150916-220225-169978048-5050-600-0001
Sep 16 22:31:09 core-03 mesos-slave[829]: I0916 22:31:09.231657 842 status_update_manager.cpp:826] Checkpointing UPDATE for status update TASK_LOST (UUID: f5f8ce65-88c5-41f0-93ab-41e82022efc5) for task task.journalnode.journalnode.NodeExecutor.1442442573784 of framework 20150916-220225-169978048-5050-600-0001
Sep 16 22:31:09 core-03 mesos-slave[829]: F0916 22:31:09.232223 842 slave.cpp:2897] CHECK_READY(future): is FAILED: Failed to write status update TASK_LOST (UUID: f5f8ce65-88c5-41f0-93ab-41e82022efc5) for task task.journalnode.journalnode.NodeExecutor.1442442573784 of framework 20150916-220225-169978048-5050-600-0001 to '/tmp/mesos/meta/slaves/20150916-220225-169978048-5050-600-S2/frameworks/20150916-220225-169978048-5050-600-0001/executors/executor.journalnode.NodeExecutor.1442442573784/runs/4788225d-fb42-44fe-b44d-d8bd67f144c9/tasks/task.journalnode.journalnode.NodeExecutor.1442442573784/task.updates': Failed to write size: No space left on device Failed to handle status update TASK_LOST (UUID: f5f8ce65-88c5-41f0-93ab-41e82022efc5) for task task.journalnode.journalnode.NodeExecutor.1442442573784 of framework 20150916-220225-169978048-5050-600-0001
Sep 16 22:31:09 core-03 mesos-slave[829]: *** Check failure stack trace: ***
Sep 16 22:31:09 core-03 mesos-slave[829]: @ 0x7faf5e151ac0 google::LogMessage::Fail()
Sep 16 22:31:09 core-03 mesos-slave[829]: @ 0x7faf5e151a0c google::LogMessage::SendToLog()
Sep 16 22:31:09 core-03 mesos-slave[829]: @ 0x7faf5e15140e google::LogMessage::Flush()
Sep 16 22:31:09 core-03 mesos-slave[829]: @ 0x7faf5e154322 google::LogMessageFatal::~LogMessageFatal()
Sep 16 22:31:09 core-03 mesos-slave[829]: @ 0x7faf5d3ff2d8 _CheckFatal::~_CheckFatal()
Sep 16 22:31:09 core-03 mesos-slave[829]: @ 0x7faf5d865c24 mesos::internal::slave::Slave::__statusUpdate()
Sep 16 22:31:09 core-03 mesos-slave[829]: @ 0x7faf5d89e452 _ZZN7process8dispatchIN5mesos8internal5slave5SlaveERKNS_6FutureI7NothingEERKNS2_12StatusUpdateERKNS_4UPIDES7_SA_SD_EEvRKNS_3PIDIT_EEMSH_FvT0_T1_T2_ET3_T4_T5_ENKUlPNS_11ProcessBaseEE_clESU_
Sep 16 22:31:09 core-03 mesos-slave[829]: @ 0x7faf5d8d1dd3 _ZNSt17_Function_handlerIFvPN7process11ProcessBaseEEZNS0_8dispatchIN5mesos8internal5slave5SlaveERKNS0_6FutureI7NothingEERKNS6_12StatusUpdateERKNS0_4UPIDESB_SE_SH_EEvRKNS0_3PIDIT_EEMSL_FvT0_T1_T2_ET3_T4_T5_EUlS2_E_E9_M_invokeERKSt9_Any_dataS2_
Sep 16 22:31:09 core-03 mesos-slave[829]: @ 0x7faf5e0dbcf1 std::function<>::operator()()
Sep 16 22:31:09 core-03 mesos-slave[829]: @ 0x7faf5e0c5545 process::ProcessBase::visit()
Sep 16 22:31:09 core-03 mesos-slave[829]: @ 0x7faf5e0c8398 process::DispatchEvent::visit()
Sep 16 22:31:09 core-03 mesos-slave[829]: @ 0x7faf5d3de37a process::ProcessBase::serve()
Sep 16 22:31:09 core-03 mesos-slave[829]: @ 0x7faf5e0c1a4c process::ProcessManager::resume()
Sep 16 22:31:10 core-03 mesos-slave[829]: @ 0x7faf5e0b5f58 process::internal::schedule()
Sep 16 22:31:10 core-03 mesos-slave[829]: @ 0x7faf5e113be7 _ZNSt12_Bind_simpleIFPFvvEvEE9_M_invokeIJEEEvSt12_Index_tupleIJXspT_EEE
Sep 16 22:31:10 core-03 mesos-slave[829]: @ 0x7faf5e113b41 std::_Bind_simple<>::operator()()
Sep 16 22:31:10 core-03 mesos-slave[829]: @ 0x7faf5e113ada std::thread::_Impl<>::_M_run()
Sep 16 22:31:10 core-03 mesos-slave[829]: @ 0x7faf5a1c1d73 (unknown)
Sep 16 22:31:10 core-03 mesos-slave[829]: @ 0x7faf5a42566c (unknown)
Sep 16 22:31:10 core-03 mesos-slave[829]: @ 0x7faf599202ed (unknown)
Yes, that is a suspicious containerizer issue. It points to a possible Mesos containerizer issue related to systemd or not giving the executor enough time to register (i.e. executor registration time out).
@elingg I think I might be one step closer to finding the problem.
I noticed that the hdfs-site.xml
file downloaded to the executor's sandbox have some issues:
<property>
<name>dfs.namenode.rpc-address.hdfs.nn1</name>
<value>:50071</value>
</property>
<property>
<name>dfs.namenode.http-address.hdfs.nn1</name>
<value>:50070</value>
</property>
It looks like ${nn1Hostname}
is an empty value and gets expanded to :50070
How is the value of nn1Hostname
determined? My mesos-slaves are launched with --hostname=${COREOS_PRIVATE_IPV4} --ip=${COREOS_PRIVATE_IPV4}
so that everything is done with ip addresses and I don't have to add the hostnames of all my nodes to my /etc/hosts
.
I don't think that's the issue as the NN's have not yet launched so that would be expected to be blank until they launch. You should actually check for the *-site.xml files in hdfs-mesos-executor-0.1.4/etc/hadoop folder in your sandbox
Within the hdfs-mesos-executor-0.1.4/etc/hadoop/*
folder, things look fine.
mesos-site.xml
:
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>mesos.hdfs.data.dir</name>
<description>The primary data directory in HDFS</description>
<value>/var/lib/hdfs/data</value>
</property>
<property>
<name>mesos.hdfs.secondary.data.dir</name>
<description>The secondary data directory in HDFS</description>
<value>/var/run/hadoop-hdfs</value>
</property>
<property>
<name>mesos.hdfs.native-hadoop-binaries</name>
<description>Mark true if you have hadoop pre-installed on your host machines (otherwise it will be distributed by the scheduler)</description>
<value>false</value>
</property>
<property>
<name>mesos.hdfs.framework.mnt.path</name>
<description>Mount location (if mesos.hdfs.native-hadoop-binaries is marked false)</description>
<value>/opt/mesosphere</value>
</property>
<property>
<name>mesos.hdfs.state.zk</name>
<description>Comma-separated hostname-port pairs of zookeeper node locations for HDFS framework state information</description>
<value>master.mesos:2181</value>
</property>
<property>
<name>mesos.master.uri</name>
<description>Zookeeper entry for mesos master location</description>
<value>zk://master.mesos:2181/mesos</value>
</property>
<property>
<name>mesos.hdfs.zkfc.ha.zookeeper.quorum</name>
<description>Comma-separated list of zookeeper hostname-port pairs for HDFS HA features</description>
<value>master.mesos:2181</value>
</property>
<property>
<name>mesos.hdfs.framework.name</name>
<description>Your Mesos framework name and cluster name when accessing files (hdfs://YOUR_NAME)</description>
<value>hdfs</value>
</property>
<property>
<name>mesos.hdfs.mesosdns</name>
<description>Whether to use Mesos DNS for service discovery within HDFS</description>
<value>true</value>
</property>
<property>
<name>mesos.hdfs.mesosdns.domain</name>
<description>Root domain name of Mesos DNS (usually 'mesos')</description>
<value>mesos</value>
</property>
<property>
<name>mesos.native.library</name>
<description>Location of libmesos.so</description>
<value>/opt/test/packages/mesos/lib/libmesos.so</value>
</property>
<property>
<name>mesos.hdfs.journalnode.count</name>
<description>Number of journal nodes (must be odd)</description>
<value>1</value>
</property>
<!-- Additional settings for fine-tuning -->
<property>
<name>mesos.hdfs.jvm.overhead</name>
<description>Multiplier on resources reserved in order to account for JVM allocation</description>
<value>1</value>
</property>
<property>
<name>mesos.hdfs.hadoop.heap.size</name>
<value>512</value>
</property>
<property>
<name>mesos.hdfs.namenode.heap.size</name>
<value>512</value>
</property>
<property>
<name>mesos.hdfs.datanode.heap.size</name>
<value>512</value>
</property>
<property>
<name>mesos.hdfs.executor.heap.size</name>
<value>256</value>
</property>
<property>
<name>mesos.hdfs.executor.cpus</name>
<value>0.5</value>
</property>
<property>
<name>mesos.hdfs.namenode.cpus</name>
<value>0.5</value>
</property>
<property>
<name>mesos.hdfs.journalnode.cpus</name>
<value>0.5</value>
</property>
<property>
<name>mesos.hdfs.datanode.cpus</name>
<value>0.5</value>
</property>
<property>
<name>mesos.hdfs.user</name>
<value>root</value>
</property>
<property>
<name>mesos.hdfs.role</name>
<value>*</value>
</property>
</configuration>
hdfs-site.xml
:
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.nameservice.id</name>
<value>${frameworkName}</value>
</property>
<property>
<name>dfs.nameservices</name>
<value>${frameworkName}</value>
</property>
<property>
<name>dfs.ha.namenodes.${frameworkName}</name>
<value>nn1</value>
</property>
<property>
<name>dfs.namenode.rpc-address.${frameworkName}.nn1</name>
<value>${nn1Hostname}:50071</value>
</property>
<property>
<name>dfs.namenode.http-address.${frameworkName}.nn1</name>
<value>${nn1Hostname}:50070</value>
</property>
<property>
<name>dfs.client.failover.proxy.provider.${frameworkName}</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://${journalnodes}/${frameworkName}</value>
</property>
<property>
<name>ha.zookeeper.quorum</name>
<value>${haZookeeperQuorum}</value>
</property>
<property>
<name>dfs.journalnode.edits.dir</name>
<value>${dataDir}/jn</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file://${dataDir}/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file://${dataDir}/data</value>
</property>
<property>
<name>dfs.ha.fencing.methods</name>
<value>shell(/bin/true)</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.datanode.du.reserved</name>
<value>10485760</value>
</property>
<property>
<name>dfs.datanode.balance.bandwidthPerSec</name>
<value>41943040</value>
</property>
<property>
<name>dfs.namenode.safemode.threshold-pct</name>
<value>0.90</value>
</property>
<property>
<name>dfs.namenode.heartbeat.recheck-interval</name>
<!-- 60 seconds -->
<value>60000</value>
</property>
<property>
<name>dfs.datanode.handler.count</name>
<value>10</value>
</property>
<property>
<name>dfs.namenode.handler.count</name>
<value>20</value>
</property>
<property>
<name>dfs.image.compress</name>
<value>true</value>
</property>
<property>
<name>dfs.image.compression.codec</name>
<value>org.apache.hadoop.io.compress.SnappyCodec</value>
</property>
<property>
<name>dfs.namenode.invalidate.work.pct.per.iteration</name>
<value>0.35f</value>
</property>
<property>
<name>dfs.namenode.replication.work.multiplier.per.iteration</name>
<value>4</value>
</property>
<!-- This property allows us to use IP's directly for communication instead of hostnames. -->
<property>
<name>dfs.namenode.datanode.registration.ip-hostname-check</name>
<value>false</value>
</property>
<property>
<name>dfs.client.read.shortcircuit</name>
<value>true</value>
</property>
<property>
<name>dfs.client.read.shortcircuit.streams.cache.size</name>
<value>1000</value>
</property>
<property>
<name>dfs.client.read.shortcircuit.streams.cache.size.expiry.ms</name>
<value>1000</value>
</property>
<!-- This property needs to be consistent with mesos.hdfs.secondary.data.dir -->
<property>
<name>dfs.domain.socket.path</name>
<value>/var/run/hadoop-hdfs/dn._PORT</value>
</property>
</configuration>
I tried running ./bin/hdfs-mesos-journalnode
, but it wasn't able to expand the variables in the config files:
15/09/16 23:38:20 INFO server.JournalNode: registered UNIX signal handlers for [TERM, HUP, INT]
Exception in thread "main" java.lang.IllegalArgumentException: Journal dir '${dataDir}/jn' should be an absolute path
at org.apache.hadoop.hdfs.qjournal.server.JournalNode.validateAndCreateJournalDir(JournalNode.java:112)
at org.apache.hadoop.hdfs.qjournal.server.JournalNode.start(JournalNode.java:136)
at org.apache.hadoop.hdfs.qjournal.server.JournalNode.run(JournalNode.java:126)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.hdfs.qjournal.server.JournalNode.main(JournalNode.java:295)
Is there anyway to turn up the verbosity of the executor so that I can get some output in stderr
and stdout
? Or perhaps a way to run the executor for the journal node manually from the commandline to see if there's an error.
Copied the expanded hdfs-site.xml
to etc/haddop/hdfs-site.xml
and launched hdfs-mesos-journalnode
and it appears to launch properly:
STARTUP_MSG: build = http://github.com/cloudera/hadoop -r 4cda8416c73034b59cc8baafbe3666b074472846; compiled by 'jenkins' on 2015-01-28T00:37Z
STARTUP_MSG: java = 1.7.0_76
************************************************************/
15/09/17 00:02:26 INFO server.JournalNode: registered UNIX signal handlers for [TERM, HUP, INT]
15/09/17 00:02:26 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
15/09/17 00:02:27 INFO impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
15/09/17 00:02:27 INFO impl.MetricsSystemImpl: JournalNode metrics system started
15/09/17 00:02:29 INFO hdfs.DFSUtil: Starting web server as: null
15/09/17 00:02:29 INFO hdfs.DFSUtil: Starting Web-server for journal at: http://0.0.0.0:8480
15/09/17 00:02:29 INFO mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
15/09/17 00:02:29 INFO http.HttpRequestLog: Http request log for http.requests.journal is not defined
15/09/17 00:02:29 INFO http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
15/09/17 00:02:29 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context journal
15/09/17 00:02:29 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
15/09/17 00:02:29 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs
15/09/17 00:02:29 INFO http.HttpServer2: Jetty bound to port 8480
15/09/17 00:02:29 INFO mortbay.log: jetty-6.1.26.cloudera.4
15/09/17 00:02:30 INFO mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:8480
15/09/17 00:02:32 INFO ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue
15/09/17 00:02:32 INFO ipc.Server: Starting Socket Reader #1 for port 8485
15/09/17 00:02:32 INFO ipc.Server: IPC Server Responder: starting
15/09/17 00:02:32 INFO ipc.Server: IPC Server listener on 8485: starting
My best guess based on your findings would be a containerizer issue on coreos or executor registration timeout issue. Might be best to check with the Mesos core team.
@elingg I have made some progress!
Current stack is CoreOS 808.0.0 and Mesos 0.24.0 and marathon 0.10.0.
I compiled mesos by setting prefix
to /opt/test/mesos
and deployed mesos to /opt/test/mesos
. In addition, LD_LIBRARY_PATH
is set to /opt/test/mesos/lib
.
I am now able to get some useful information out of stderr
for the journal node executor.
The current mesos-site.xml
I am compiling with mesos-hdfs with is:
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>mesos.hdfs.data.dir</name>
<description>The primary data directory in HDFS</description>
<value>/var/lib/hdfs/data</value>
</property>
<property>
<name>mesos.hdfs.secondary.data.dir</name>
<description>The secondary data directory in HDFS</description>
<value>/var/run/hadoop-hdfs</value>
</property>
<property>
<name>mesos.hdfs.native-hadoop-binaries</name>
<description>Mark true if you have hadoop pre-installed on your host machines (otherwise it will be distributed by the scheduler)</description>
<value>false</value>
</property>
<property>
<name>mesos.hdfs.framework.mnt.path</name>
<description>Mount location (if mesos.hdfs.native-hadoop-binaries is marked false)</description>
<value>/opt/mesosphere</value>
</property>
<property>
<name>mesos.hdfs.state.zk</name>
<description>Comma-separated hostname-port pairs of zookeeper node locations for HDFS framework state information</description>
<value>master.mesos:2181</value>
</property>
<property>
<name>mesos.master.uri</name>
<description>Zookeeper entry for mesos master location</description>
<value>zk://master.mesos:2181/mesos</value>
</property>
<property>
<name>mesos.hdfs.zkfc.ha.zookeeper.quorum</name>
<description>Comma-separated list of zookeeper hostname-port pairs for HDFS HA features</description>
<value>master.mesos:2181</value>
</property>
<property>
<name>mesos.hdfs.framework.name</name>
<description>Your Mesos framework name and cluster name when accessing files (hdfs://YOUR_NAME)</description>
<value>hdfs</value>
</property>
<property>
<name>mesos.hdfs.mesosdns</name>
<description>Whether to use Mesos DNS for service discovery within HDFS</description>
<value>true</value>
</property>
<property>
<name>mesos.hdfs.mesosdns.domain</name>
<description>Root domain name of Mesos DNS (usually 'mesos')</description>
<value>mesos</value>
</property>
<property>
<name>mesos.native.library</name>
<description>Location of libmesos.so</description>
<value>/opt/test/mesos/lib/libmesos.so</value>
</property>
<property>
<name>mesos.hdfs.journalnode.count</name>
<description>Number of journal nodes (must be odd)</description>
<value>1</value>
</property>
<!-- Additional settings for fine-tuning -->
<property>
<name>mesos.hdfs.jvm.overhead</name>
<description>Multiplier on resources reserved in order to account for JVM allocation</description>
<value>1</value>
</property>
<property>
<name>mesos.hdfs.hadoop.heap.size</name>
<value>512</value>
</property>
<property>
<name>mesos.hdfs.namenode.heap.size</name>
<value>512</value>
</property>
<property>
<name>mesos.hdfs.datanode.heap.size</name>
<value>512</value>
</property>
<property>
<name>mesos.hdfs.executor.heap.size</name>
<value>256</value>
</property>
<property>
<name>mesos.hdfs.executor.cpus</name>
<value>0.5</value>
</property>
<property>
<name>mesos.hdfs.namenode.cpus</name>
<value>0.5</value>
</property>
<property>
<name>mesos.hdfs.journalnode.cpus</name>
<value>0.5</value>
</property>
<property>
<name>mesos.hdfs.datanode.cpus</name>
<value>0.5</value>
</property>
<property>
<name>mesos.hdfs.user</name>
<value>root</value>
</property>
<property>
<name>mesos.hdfs.role</name>
<value>*</value>
</property>
<property>
<name>mesos.hdfs.ld-library-path</name>
<value>/opt/test/mesos/lib</value>
</property>
</configuration>
This is the output from the journal node executor's stderr
and stdout
:
I0923 06:26:57.411906 1385 logging.cpp:172] INFO level logging started!
I0923 06:26:57.412338 1385 fetcher.cpp:414] Fetcher Info: {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/20150923-054500-169978048-5050-1212-S3\/root","items":[{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"http:\/\/192.168.33.13:10000\/hdfs-mesos-executor-0.1.4.tgz"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"http:\/\/192.168.33.13:10000\/hdfs-site.xml"}},{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"https:\/\/downloads.mesosphere.io\/java\/jre-7u76-linux-x64.tar.gz"}}],"sandbox_directory":"\/var\/lib\/mesos\/sandbox\/slaves\/20150923-054500-169978048-5050-1212-S3\/frameworks\/20150923-054500-169978048-5050-1212-0001\/executors\/executor.journalnode.NodeExecutor.1442989617044\/runs\/4d1ad220-b667-4d8a-b08a-83a48914e9d3","user":"root"}
I0923 06:26:57.418102 1385 fetcher.cpp:369] Fetching URI 'http://192.168.33.13:10000/hdfs-mesos-executor-0.1.4.tgz'
I0923 06:26:57.418159 1385 fetcher.cpp:243] Fetching directly into the sandbox directory
I0923 06:26:57.418212 1385 fetcher.cpp:180] Fetching URI 'http://192.168.33.13:10000/hdfs-mesos-executor-0.1.4.tgz'
I0923 06:26:57.418263 1385 fetcher.cpp:127] Downloading resource from 'http://192.168.33.13:10000/hdfs-mesos-executor-0.1.4.tgz' to '/var/lib/mesos/sandbox/slaves/20150923-054500-169978048-5050-1212-S3/frameworks/20150923-054500-169978048-5050-1212-0001/executors/executor.journalnode.NodeExecutor.1442989617044/runs/4d1ad220-b667-4d8a-b08a-83a48914e9d3/hdfs-mesos-executor-0.1.4.tgz'
I0923 06:26:57.867511 1385 fetcher.cpp:76] Extracting with command: tar -C '/var/lib/mesos/sandbox/slaves/20150923-054500-169978048-5050-1212-S3/frameworks/20150923-054500-169978048-5050-1212-0001/executors/executor.journalnode.NodeExecutor.1442989617044/runs/4d1ad220-b667-4d8a-b08a-83a48914e9d3' -xf '/var/lib/mesos/sandbox/slaves/20150923-054500-169978048-5050-1212-S3/frameworks/20150923-054500-169978048-5050-1212-0001/executors/executor.journalnode.NodeExecutor.1442989617044/runs/4d1ad220-b667-4d8a-b08a-83a48914e9d3/hdfs-mesos-executor-0.1.4.tgz'
I0923 06:26:59.066606 1385 fetcher.cpp:84] Extracted '/var/lib/mesos/sandbox/slaves/20150923-054500-169978048-5050-1212-S3/frameworks/20150923-054500-169978048-5050-1212-0001/executors/executor.journalnode.NodeExecutor.1442989617044/runs/4d1ad220-b667-4d8a-b08a-83a48914e9d3/hdfs-mesos-executor-0.1.4.tgz' into '/var/lib/mesos/sandbox/slaves/20150923-054500-169978048-5050-1212-S3/frameworks/20150923-054500-169978048-5050-1212-0001/executors/executor.journalnode.NodeExecutor.1442989617044/runs/4d1ad220-b667-4d8a-b08a-83a48914e9d3'
W0923 06:26:59.066730 1385 fetcher.cpp:265] Copying instead of extracting resource from URI with 'extract' flag, because it does not seem to be an archive: http://192.168.33.13:10000/hdfs-mesos-executor-0.1.4.tgz
I0923 06:26:59.066947 1385 fetcher.cpp:446] Fetched 'http://192.168.33.13:10000/hdfs-mesos-executor-0.1.4.tgz' to '/var/lib/mesos/sandbox/slaves/20150923-054500-169978048-5050-1212-S3/frameworks/20150923-054500-169978048-5050-1212-0001/executors/executor.journalnode.NodeExecutor.1442989617044/runs/4d1ad220-b667-4d8a-b08a-83a48914e9d3/hdfs-mesos-executor-0.1.4.tgz'
I0923 06:26:59.066982 1385 fetcher.cpp:369] Fetching URI 'http://192.168.33.13:10000/hdfs-site.xml'
I0923 06:26:59.067008 1385 fetcher.cpp:243] Fetching directly into the sandbox directory
I0923 06:26:59.067049 1385 fetcher.cpp:180] Fetching URI 'http://192.168.33.13:10000/hdfs-site.xml'
I0923 06:26:59.067090 1385 fetcher.cpp:127] Downloading resource from 'http://192.168.33.13:10000/hdfs-site.xml' to '/var/lib/mesos/sandbox/slaves/20150923-054500-169978048-5050-1212-S3/frameworks/20150923-054500-169978048-5050-1212-0001/executors/executor.journalnode.NodeExecutor.1442989617044/runs/4d1ad220-b667-4d8a-b08a-83a48914e9d3/hdfs-site.xml'
W0923 06:26:59.134197 1385 fetcher.cpp:265] Copying instead of extracting resource from URI with 'extract' flag, because it does not seem to be an archive: http://192.168.33.13:10000/hdfs-site.xml
I0923 06:26:59.135251 1385 fetcher.cpp:446] Fetched 'http://192.168.33.13:10000/hdfs-site.xml' to '/var/lib/mesos/sandbox/slaves/20150923-054500-169978048-5050-1212-S3/frameworks/20150923-054500-169978048-5050-1212-0001/executors/executor.journalnode.NodeExecutor.1442989617044/runs/4d1ad220-b667-4d8a-b08a-83a48914e9d3/hdfs-site.xml'
I0923 06:26:59.135306 1385 fetcher.cpp:369] Fetching URI 'https://downloads.mesosphere.io/java/jre-7u76-linux-x64.tar.gz'
I0923 06:26:59.135496 1385 fetcher.cpp:243] Fetching directly into the sandbox directory
I0923 06:26:59.135550 1385 fetcher.cpp:180] Fetching URI 'https://downloads.mesosphere.io/java/jre-7u76-linux-x64.tar.gz'
I0923 06:26:59.135594 1385 fetcher.cpp:127] Downloading resource from 'https://downloads.mesosphere.io/java/jre-7u76-linux-x64.tar.gz' to '/var/lib/mesos/sandbox/slaves/20150923-054500-169978048-5050-1212-S3/frameworks/20150923-054500-169978048-5050-1212-0001/executors/executor.journalnode.NodeExecutor.1442989617044/runs/4d1ad220-b667-4d8a-b08a-83a48914e9d3/jre-7u76-linux-x64.tar.gz'
I0923 06:30:25.936226 1385 fetcher.cpp:76] Extracting with command: tar -C '/var/lib/mesos/sandbox/slaves/20150923-054500-169978048-5050-1212-S3/frameworks/20150923-054500-169978048-5050-1212-0001/executors/executor.journalnode.NodeExecutor.1442989617044/runs/4d1ad220-b667-4d8a-b08a-83a48914e9d3' -xf '/var/lib/mesos/sandbox/slaves/20150923-054500-169978048-5050-1212-S3/frameworks/20150923-054500-169978048-5050-1212-0001/executors/executor.journalnode.NodeExecutor.1442989617044/runs/4d1ad220-b667-4d8a-b08a-83a48914e9d3/jre-7u76-linux-x64.tar.gz'
I0923 06:30:27.865962 1385 fetcher.cpp:84] Extracted '/var/lib/mesos/sandbox/slaves/20150923-054500-169978048-5050-1212-S3/frameworks/20150923-054500-169978048-5050-1212-0001/executors/executor.journalnode.NodeExecutor.1442989617044/runs/4d1ad220-b667-4d8a-b08a-83a48914e9d3/jre-7u76-linux-x64.tar.gz' into '/var/lib/mesos/sandbox/slaves/20150923-054500-169978048-5050-1212-S3/frameworks/20150923-054500-169978048-5050-1212-0001/executors/executor.journalnode.NodeExecutor.1442989617044/runs/4d1ad220-b667-4d8a-b08a-83a48914e9d3'
W0923 06:30:27.866242 1385 fetcher.cpp:265] Copying instead of extracting resource from URI with 'extract' flag, because it does not seem to be an archive: https://downloads.mesosphere.io/java/jre-7u76-linux-x64.tar.gz
I0923 06:30:27.866333 1385 fetcher.cpp:446] Fetched 'https://downloads.mesosphere.io/java/jre-7u76-linux-x64.tar.gz' to '/var/lib/mesos/sandbox/slaves/20150923-054500-169978048-5050-1212-S3/frameworks/20150923-054500-169978048-5050-1212-0001/executors/executor.journalnode.NodeExecutor.1442989617044/runs/4d1ad220-b667-4d8a-b08a-83a48914e9d3/jre-7u76-linux-x64.tar.gz'
I0923 06:30:30.748337 1419 exec.cpp:133] Version: 0.24.0
I0923 06:30:30.763150 1439 exec.cpp:207] Executor registered on slave 20150923-054500-169978048-5050-1212-S3
unlink: cannot unlink '/opt/mesosphere/hdfs': No such file or directory
This is the output of stdout
:
MESOS_DIRECTORY=/var/lib/mesos/sandbox/slaves/20150923-054500-169978048-5050-1212-S3/frameworks/20150923-054500-169978048-5050-1212-0001/executors/executor.journalnode.NodeExecutor.1442989617044/runs/4d1ad220-b667-4d8a-b08a-83a48914e9d3
LD_LIBRARY_PATH=/opt/test/mesos/lib
MESOS_EXECUTOR_ID=executor.journalnode.NodeExecutor.1442989617044
PATH=/opt/test/java/bin:/usr/sbin:/usr/bin
PWD=/var/lib/mesos/sandbox/slaves/20150923-054500-169978048-5050-1212-S3/frameworks/20150923-054500-169978048-5050-1212-0001/executors/executor.journalnode.NodeExecutor.1442989617044/runs/4d1ad220-b667-4d8a-b08a-83a48914e9d3
JAVA_HOME=/var/lib/mesos/sandbox/slaves/20150923-054500-169978048-5050-1212-S3/frameworks/20150923-054500-169978048-5050-1212-0001/executors/executor.journalnode.NodeExecutor.1442989617044/runs/4d1ad220-b667-4d8a-b08a-83a48914e9d3/jre1.7.0_76
MESOS_NATIVE_LIBRARY=/opt/test/mesos/lib/libmesos-0.24.0.so
MESOS_SLAVE_PID=slave(1)@192.168.33.13:5051
MESOS_FRAMEWORK_ID=20150923-054500-169978048-5050-1212-0001
MESOS_CHECKPOINT=1
EXECUTOR_OPTS=-Xmx256m -Xms256m
SHLVL=1
LIBPROCESS_PORT=0
MESOS_SLAVE_ID=20150923-054500-169978048-5050-1212-S3
MESOS_RECOVERY_TIMEOUT=15mins
_=/usr/bin/env
Warning: MESOS_NATIVE_LIBRARY is deprecated, use MESOS_NATIVE_JAVA_LIBRARY instead. Future releases will not support JNI bindings via MESOS_NATIVE_LIBRARY.
06:30:30.839 [Thread-3] INFO o.a.m.h.e.AbstractNodeExecutor - Creating a symbolic link for HDFS binary
06:30:30.843 [Thread-3] INFO org.apache.mesos.hdfs.file.FileUtils - data dir exits:/opt/mesosphere
06:30:31.643 [Thread-3] INFO o.a.m.h.e.AbstractNodeExecutor - Unable to unlink old sym link. Link may not exist. Exit code: 1
06:30:31.652 [Thread-3] INFO o.a.m.h.e.AbstractNodeExecutor - The linked HDFS binary path is: /var/lib/mesos/sandbox/slaves/20150923-054500-169978048-5050-1212-S3/frameworks/20150923-054500-169978048-5050-1212-0001/executors/executor.journalnode.NodeExecutor.1442989617044/runs/4d1ad220-b667-4d8a-b08a-83a48914e9d3/hdfs-mesos-executor-0.1.4
06:30:31.652 [Thread-3] INFO o.a.m.h.e.AbstractNodeExecutor - The symbolic link path is: /opt/mesosphere/hdfs
06:30:31.663 [Thread-3] ERROR o.a.m.h.e.AbstractNodeExecutor - Error creating the symbolic link to hdfs binary
java.io.FileNotFoundException: /usr/bin/hadoop (Read-only file system)
at java.io.FileOutputStream.open(Native Method) ~[na:1.7.0_76]
at java.io.FileOutputStream.<init>(Unknown Source) ~[na:1.7.0_76]
at java.io.FileOutputStream.<init>(Unknown Source) ~[na:1.7.0_76]
at org.apache.mesos.hdfs.executor.AbstractNodeExecutor.addBinaryToPath(AbstractNodeExecutor.java:168) [hdfs-executor-0.1.4-uber.jar:na]
at org.apache.mesos.hdfs.executor.AbstractNodeExecutor.createSymbolicLink(AbstractNodeExecutor.java:147) [hdfs-executor-0.1.4-uber.jar:na]
at org.apache.mesos.hdfs.executor.AbstractNodeExecutor.registered(AbstractNodeExecutor.java:82) [hdfs-executor-0.1.4-uber.jar:na]
06:30:31.665 [Thread-2] INFO o.a.m.hdfs.executor.TaskShutdownHook - shutdown hook shutting down tasks
06:30:31.666 [Thread-2] INFO o.a.mesos.hdfs.executor.NodeExecutor - Executor asked to shutdown
Any ideas what might be causing this?
I just remembered your comment regarding CoreOS being locked down. I then pre-distributed binaries for hadoop (CDH distribution) to all my CoreOS nodes and added it to the PATH environment variable available to the mesos slaves.
Now, the journal node is able to launch, but it keeps on complaining about the journal directory not being an absolute path:
************************************************************/
15/09/24 03:41:13 INFO server.JournalNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting JournalNode
STARTUP_MSG: host = core-01/192.168.33.10
STARTUP_MSG: args = []
STARTUP_MSG: version = 2.5.0-cdh5.3.1
STARTUP_MSG: classpath = /var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/etc/hadoop:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/commons-compress-1.4.1.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/paranamer-2.3.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/commons-configuration-1.6.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/commons-cli-1.2.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/gson-2.2.4.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/jersey-core-1.9.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/avro-1.7.6-cdh5.3.1.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/hadoop-auth-2.5.0-cdh5.3.1.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/commons-io-2.4.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/commons-lang-2.6.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/httpcore-4.2.5.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/junit-4.11.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/jsch-0.1.42.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/commons-codec-1.4.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/jettison-1.1.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/hamcrest-core-1.3.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/zookeeper-3.4.5-cdh5.3.1.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/commons-collections-3.2.1.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/log4j-1.2.17.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/jackson-xc-1.8.8.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/mockito-all-1.8.5.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/netty-3.6.2.Final.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/jsp-api-2.1.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/jetty-6.1.26.cloudera.4.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/commons-net-3.1.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/commons-digester-1.8.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/jetty-util-6.1.26.cloudera.4.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/commons-math3-3.1.1.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/curator-framework-2.6.0.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/jets3t-0.9.0.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/commons-httpclient-3.1.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/activation-1.1.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/jsr305-1.3.9.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/stax-api-1.0-2.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/httpclient-4.2.5.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/curator-recipes-2.6.0.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/commons-logging-1.1.3.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/servlet-api-2.5.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/jersey-json-1.9.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/jersey-server-1.9.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/xz-1.0.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/curator-client-2.6.0.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/hadoop-annotations-2.5.0-cdh5.3.1.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/asm-3.2.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/xmlenc-0.52.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/commons-el-1.0.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/lib/guava-11.0.2.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/common/hadoop-common-2.5.0-cdh5.3.1.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/hdfs:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/hdfs/lib/jackson-mapper-asl-1.8.8.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/hdfs/lib/commons-io-2.4.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/hdfs/lib/jackson-core-asl-1.8.8.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/hdfs/lib/jetty-6.1.26.cloudera.4.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/hdfs/lib/jetty-util-6.1.26.cloudera.4.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/hdfs/lib/asm-3.2.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/hdfs/lib/commons-el-1.0.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/hdfs/lib/guava-11.0.2.jar:/var/lib/mesos/sandbox/slaves/20150924-031430-169978048-5050-596-S0/frameworks/20150924-031430-169978048-5050-596-0000/executors/executor.journalnode.NodeExecutor.1443065388745/runs/2d390c59-d00f-46d3-afa9-347f94588fdf/hdfs-mesos-executor-0.1.4/share/hadoop/hdfs/hadoop-hdfs-2.5.0-cdh5.3.1.jar:/share/hadoop/yarn/*:/share/hadoop/mapreduce/*:/contrib/capacity-scheduler/*.jar
STARTUP_MSG: build = http://github.com/cloudera/hadoop -r 4cda8416c73034b59cc8baafbe3666b074472846; compiled by 'jenkins' on 2015-01-28T00:37Z
STARTUP_MSG: java = 1.7.0_76
************************************************************/
15/09/24 03:41:13 INFO server.JournalNode: registered UNIX signal handlers for [TERM, HUP, INT]
Exception in thread "main" java.lang.IllegalArgumentException: Journal dir '${dataDir}/jn' should be an absolute path
at org.apache.hadoop.hdfs.qjournal.server.JournalNode.validateAndCreateJournalDir(JournalNode.java:112)
at org.apache.hadoop.hdfs.qjournal.server.JournalNode.start(JournalNode.java:136)
at org.apache.hadoop.hdfs.qjournal.server.JournalNode.run(JournalNode.java:126)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.hdfs.qjournal.server.JournalNode.main(JournalNode.java:295)
15/09/24 03:41:13 INFO server.JournalNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down JournalNode at core-01/192.168.33.10
************************************************************/
if I check in the sandbox, the hdfs-site.xml
looks like so:
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.nameservice.id</name>
<value>hdfs</value>
</property>
<property>
<name>dfs.nameservices</name>
<value>hdfs</value>
</property>
<property>
<name>dfs.ha.namenodes.hdfs</name>
<value>nn1</value>
</property>
<property>
<name>dfs.namenode.rpc-address.hdfs.nn1</name>
<value>:50071</value>
</property>
<property>
<name>dfs.namenode.http-address.hdfs.nn1</name>
<value>:50070</value>
</property>
<property>
<name>dfs.client.failover.proxy.provider.hdfs</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://192.168.33.10:8485/hdfs</value>
</property>
<property>
<name>ha.zookeeper.quorum</name>
<value>master.mesos:2181</value>
</property>
<property>
<name>dfs.journalnode.edits.dir</name>
<value>/var/lib/hdfs/data/jn</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///var/lib/hdfs/data/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:///var/lib/hdfs/data/data</value>
</property>
<property>
<name>dfs.ha.fencing.methods</name>
<value>shell(/bin/true)</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.datanode.du.reserved</name>
<value>10485760</value>
</property>
<property>
<name>dfs.datanode.balance.bandwidthPerSec</name>
<value>41943040</value>
</property>
<property>
<name>dfs.namenode.safemode.threshold-pct</name>
<value>0.90</value>
</property>
<property>
<name>dfs.namenode.heartbeat.recheck-interval</name>
<!-- 60 seconds -->
<value>60000</value>
</property>
<property>
<name>dfs.datanode.handler.count</name>
<value>10</value>
</property>
<property>
<name>dfs.namenode.handler.count</name>
<value>20</value>
</property>
<property>
<name>dfs.image.compress</name>
<value>true</value>
</property>
<property>
<name>dfs.image.compression.codec</name>
<value>org.apache.hadoop.io.compress.SnappyCodec</value>
</property>
<property>
<name>dfs.namenode.invalidate.work.pct.per.iteration</name>
<value>0.35f</value>
</property>
<property>
<name>dfs.namenode.replication.work.multiplier.per.iteration</name>
<value>4</value>
</property>
<!-- This property allows us to use IP's directly for communication instead of hostnames. -->
<property>
<name>dfs.namenode.datanode.registration.ip-hostname-check</name>
<value>false</value>
</property>
<property>
<name>dfs.client.read.shortcircuit</name>
<value>true</value>
</property>
<property>
<name>dfs.client.read.shortcircuit.streams.cache.size</name>
<value>1000</value>
</property>
<property>
<name>dfs.client.read.shortcircuit.streams.cache.size.expiry.ms</name>
<value>1000</value>
</property>
<!-- This property needs to be consistent with mesos.hdfs.secondary.data.dir -->
<property>
<name>dfs.domain.socket.path</name>
<value>/var/run/hadoop-hdfs/dn._PORT</value>
</property>
</configuration>
In hdfs-mesos-executor-0.1.4/etc/hadoop/hdfs-site.xml
, I see a copy with templated variables:
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.nameservice.id</name>
<value>${frameworkName}</value>
</property>
<property>
<name>dfs.nameservices</name>
<value>${frameworkName}</value>
</property>
<property>
<name>dfs.ha.namenodes.${frameworkName}</name>
<value>nn1</value>
</property>
<property>
<name>dfs.namenode.rpc-address.${frameworkName}.nn1</name>
<value>${nn1Hostname}:50071</value>
</property>
<property>
<name>dfs.namenode.http-address.${frameworkName}.nn1</name>
<value>${nn1Hostname}:50070</value>
</property>
<property>
<name>dfs.client.failover.proxy.provider.${frameworkName}</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://${journalnodes}/${frameworkName}</value>
</property>
<property>
<name>ha.zookeeper.quorum</name>
<value>${haZookeeperQuorum}</value>
</property>
<property>
<name>dfs.journalnode.edits.dir</name>
<value>${dataDir}/jn</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file://${dataDir}/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file://${dataDir}/data</value>
</property>
<property>
<name>dfs.ha.fencing.methods</name>
<value>shell(/bin/true)</value>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
<property>
<name>dfs.datanode.du.reserved</name>
<value>10485760</value>
</property>
<property>
<name>dfs.datanode.balance.bandwidthPerSec</name>
<value>41943040</value>
</property>
<property>
<name>dfs.namenode.safemode.threshold-pct</name>
<value>0.90</value>
</property>
<property>
<name>dfs.namenode.heartbeat.recheck-interval</name>
<!-- 60 seconds -->
<value>60000</value>
</property>
<property>
<name>dfs.datanode.handler.count</name>
<value>10</value>
</property>
<property>
<name>dfs.namenode.handler.count</name>
<value>20</value>
</property>
<property>
<name>dfs.image.compress</name>
<value>true</value>
</property>
<property>
<name>dfs.image.compression.codec</name>
<value>org.apache.hadoop.io.compress.SnappyCodec</value>
</property>
<property>
<name>dfs.namenode.invalidate.work.pct.per.iteration</name>
<value>0.35f</value>
</property>
<property>
<name>dfs.namenode.replication.work.multiplier.per.iteration</name>
<value>4</value>
</property>
<!-- This property allows us to use IP's directly for communication instead of hostnames. -->
<property>
<name>dfs.namenode.datanode.registration.ip-hostname-check</name>
<value>false</value>
</property>
<property>
<name>dfs.client.read.shortcircuit</name>
<value>true</value>
</property>
<property>
<name>dfs.client.read.shortcircuit.streams.cache.size</name>
<value>1000</value>
</property>
<property>
<name>dfs.client.read.shortcircuit.streams.cache.size.expiry.ms</name>
<value>1000</value>
</property>
<!-- This property needs to be consistent with mesos.hdfs.secondary.data.dir -->
<property>
<name>dfs.domain.socket.path</name>
<value>/var/run/hadoop-hdfs/dn._PORT</value>
</property>
</configuration>
Is there any reason why it's trying to load the template version of hdfs-site.xml
rather than the final version of hdfs-site.xml
?
Glad to hear of your progress! If you use the Predistributed binaries option (which does make sense as CoreOS is locked down like we discussed) that means you need to fill out hdfs-site.xml yourself to make sure it's configured properly. My recommendation would be if you are using predistributed binaries and also use Mesos DNS with the example configs for Mesos DNS as a part of your binaries, i.e. https://github.com/mesosphere/hdfs/tree/master/example-conf/mesosphere-dcos
Closing this as I was able to successfully launch a POC cluster with 3 slaves and 1 master/slave all running CoreOS!
I have 4 mesos 0.23 nodes with 1 master/slave and 3 slaves running via vagrant. I am using Java 8 and running everything natively on CoreOS 794.0.0 (not inside docker containers).
I have build the
HEAD
of the hdfs framework and launched it using marathon 0.10.1.I noticed that after deploying the scheduler via marathon, it attempts to launch a JournalNode on one of my slaves. However, the JournalNode task will become lost and another attempt to launch the JournalNode starts again, until I kill the scheduler.
This is my
mesos-site.xml
:This is the stderr of the scheduler:
Inspecting the
journal
of the slave node where the journal node is launched didn't yield anything interesting: