search

mesosphere-backup / hdfs-deprecated

[DEPRECATED] This project is deprecated. It will be archived on December 1, 2017.
Apache License 2.0
147 stars 52 forks source link

only 2/4 datanodes available #159

Closed AndriiOmelianenko closed 9 years ago

AndriiOmelianenko commented 9 years ago

I installed hdfs package with DCOS, in Mesos 4 datanode tasks are running, but available only two of them.

This is stderr of failing task:

15/07/23 09:18:26 INFO datanode.DataNode: STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting DataNode
STARTUP_MSG:   host = dcos-slave-03.novalocal/<slave-03 IP>
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 2.5.0-cdh5.3.1
STARTUP_MSG:   classpath = /var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/etc/hadoop:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/commons-io-2.4.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/jetty-6.1.26.cloudera.4.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/junit-4.11.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/httpcore-4.2.5.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/jackson-xc-1.8.8.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/jersey-core-1.9.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/guava-11.0.2.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/log4j-1.2.17.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/httpclient-4.2.5.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/jsp-api-2.1.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/zookeeper-3.4.5-cdh5.3.1.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/commons-cli-1.2.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/jackson-mapper-asl-1.8.8.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/jersey-server-1.9.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/avro-1.7.6-cdh5.3.1.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/commons-collections-3.2.1.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/stax-api-1.0-2.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/commons-compress-1.4.1.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/commons-math3-3.1.1.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/jets3t-0.9.0.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/commons-lang-2.6.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/xz-1.0.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/commons-logging-1.1.3.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/commons-digester-1.8.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/activation-1.1.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/xmlenc-0.52.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/hadoop-annotations-2.5.0-cdh5.3.1.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/commons-el-1.0.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/jettison-1.1.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/commons-net-3.1.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/gson-2.2.4.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/curator-framework-2.6.0.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/commons-codec-1.4.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/commons-httpclient-3.1.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/jsch-0.1.42.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/paranamer-2.3.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/hamcrest-core-1.3.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/asm-3.2.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/hadoop-auth-2.5.0-cdh5.3.1.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/curator-client-2.6.0.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/servlet-api-2.5.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/jasper-compiler-5.5.23.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/mockito-all-1.8.5.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/jsr305-1.3.9.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/jersey-json-1.9.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/commons-configuration-1.6.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/curator-recipes-2.6.0.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/jetty-util-6.1.26.cloudera.4.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/lib/netty-3.6.2.Final.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/common/hadoop-common-2.5.0-cdh5.3.1.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/hdfs:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/hdfs/lib/commons-io-2.4.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/hdfs/lib/jetty-6.1.26.cloudera.4.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/hdfs/lib/guava-11.0.2.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/hdfs/lib/jackson-core-asl-1.8.8.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/hdfs/lib/jsp-api-2.1.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/hdfs/lib/jasper-runtime-5.5.23.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/hdfs/lib/jackson-mapper-asl-1.8.8.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/hdfs/lib/commons-el-1.0.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/hdfs/lib/asm-3.2.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/hdfs/lib/jsr305-1.3.9.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/hdfs/lib/jetty-util-6.1.26.cloudera.4.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/var/lib/mesos/slave/slaves/20150721-161743-4245187456-5050-13986-S2/frameworks/20150721-161743-4245187456-5050-13986-0001/executors/executor.datanode.NodeExecutor.1437642938301/runs/4c13e504-ccc4-4c2e-8539-47c013b0ae44/hdfs-mesos-0.1.1/share/hadoop/hdfs/hadoop-hdfs-2.5.0-cdh5.3.1.jar:/share/hadoop/yarn/*:/share/hadoop/mapreduce/*:/contrib/capacity-scheduler/*.jar
STARTUP_MSG:   build = http://github.com/cloudera/hadoop -r 4cda8416c73034b59cc8baafbe3666b074472846; compiled by 'jenkins' on 2015-01-28T00:37Z
STARTUP_MSG:   java = 1.7.0_75
************************************************************/
15/07/23 09:18:26 INFO datanode.DataNode: registered UNIX signal handlers for [TERM, HUP, INT]
15/07/23 09:18:26 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
15/07/23 09:18:26 INFO impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
15/07/23 09:18:26 INFO impl.MetricsSystemImpl: DataNode metrics system started
15/07/23 09:18:26 INFO datanode.DataNode: File descriptor passing is enabled.
15/07/23 09:18:26 INFO datanode.DataNode: Configured hostname is dcos-slave-03
15/07/23 09:18:26 INFO datanode.DataNode: Starting DataNode with maxLockedMemory = 0
15/07/23 09:18:26 INFO datanode.DataNode: Opened streaming server at /0.0.0.0:50010
15/07/23 09:18:26 INFO datanode.DataNode: Balancing bandwith is 41943040 bytes/s
15/07/23 09:18:26 INFO datanode.DataNode: Number threads for balancing is 5
15/07/23 09:18:26 INFO datanode.DataNode: Balancing bandwith is 41943040 bytes/s
15/07/23 09:18:26 INFO datanode.DataNode: Number threads for balancing is 5
15/07/23 09:18:26 INFO datanode.DataNode: Listening on UNIX domain socket: /var/run/hadoop-hdfs/dn.50010
15/07/23 09:18:27 INFO mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
15/07/23 09:18:27 INFO http.HttpRequestLog: Http request log for http.requests.datanode is not defined
15/07/23 09:18:27 INFO http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
15/07/23 09:18:27 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context datanode
15/07/23 09:18:27 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs
15/07/23 09:18:27 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
15/07/23 09:18:27 INFO http.HttpServer2: addJerseyResourcePackage: packageName=org.apache.hadoop.hdfs.server.datanode.web.resources;org.apache.hadoop.hdfs.web.resources, pathSpec=/webhdfs/v1/*
15/07/23 09:18:27 INFO http.HttpServer2: Jetty bound to port 50075
15/07/23 09:18:27 INFO mortbay.log: jetty-6.1.26.cloudera.4
15/07/23 09:18:27 INFO mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:50075
15/07/23 09:18:27 INFO datanode.DataNode: dnUserName = root
15/07/23 09:18:27 INFO datanode.DataNode: supergroup = supergroup
15/07/23 09:18:27 INFO ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue
15/07/23 09:18:27 INFO ipc.Server: Starting Socket Reader #1 for port 50020
15/07/23 09:18:27 INFO datanode.DataNode: Opened IPC server at /0.0.0.0:50020
15/07/23 09:18:27 INFO datanode.DataNode: Refresh request received for nameservices: hdfs
15/07/23 09:18:27 INFO datanode.DataNode: Starting BPOfferServices for nameservices: hdfs
15/07/23 09:18:27 INFO datanode.DataNode: Block pool <registering> (Datanode Uuid unassigned) service to namenode1.hdfs.mesos/<namenode1 IP>:50071 starting to offer service
15/07/23 09:18:27 INFO datanode.DataNode: Block pool <registering> (Datanode Uuid unassigned) service to namenode2.hdfs.mesos/<namenode2 IP>:50071 starting to offer service
15/07/23 09:18:27 INFO ipc.Server: IPC Server listener on 50020: starting
15/07/23 09:18:27 INFO ipc.Server: IPC Server Responder: starting
15/07/23 09:18:27 INFO common.Storage: Lock on /var/lib/hdfs/data/data/in_use.lock acquired by nodename 4130@dcos-slave-03.novalocal
15/07/23 09:18:27 WARN common.Storage: java.io.IOException: Incompatible clusterIDs in /var/lib/hdfs/data/data: namenode clusterID = CID-42e3c6c7-*6c6817; datanode clusterID = CID-c156b955-*ce1
15/07/23 09:18:27 INFO common.Storage: Lock on /var/lib/hdfs/data/data/in_use.lock acquired by nodename 4130@dcos-slave-03.novalocal
15/07/23 09:18:27 WARN common.Storage: java.io.IOException: Incompatible clusterIDs in /var/lib/hdfs/data/data: namenode clusterID = CID-42e3c6c7-*6c6817; datanode clusterID = CID-c156b955-*ce1
15/07/23 09:18:27 FATAL datanode.DataNode: Initialization failed for Block pool <registering> (Datanode Uuid unassigned) service to namenode2.hdfs.mesos/<namenode2 IP>:50071. Exiting. 
java.io.IOException: All specified directories are failed to load.
    at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:463)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1318)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1288)
    at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:320)
    at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:221)
    at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:829)
    at java.lang.Thread.run(Thread.java:745)
15/07/23 09:18:27 WARN datanode.DataNode: Ending block pool service for: Block pool <registering> (Datanode Uuid unassigned) service to namenode2.hdfs.mesos/<namenode2 IP>:50071
15/07/23 09:18:27 FATAL datanode.DataNode: Initialization failed for Block pool <registering> (Datanode Uuid unassigned) service to namenode1.hdfs.mesos/<namenode1 IP>:50071. Exiting. 
java.io.IOException: All specified directories are failed to load.
    at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage.java:463)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.initStorage(DataNode.java:1318)
    at org.apache.hadoop.hdfs.server.datanode.DataNode.initBlockPool(DataNode.java:1288)
    at org.apache.hadoop.hdfs.server.datanode.BPOfferService.verifyAndSetNamespaceInfo(BPOfferService.java:320)
    at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:221)
    at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:829)
    at java.lang.Thread.run(Thread.java:745)
15/07/23 09:18:27 WARN datanode.DataNode: Ending block pool service for: Block pool <registering> (Datanode Uuid unassigned) service to namenode1.hdfs.mesos/<namenode1 IP>:50071
15/07/23 09:18:28 INFO datanode.DataNode: Removed Block pool <registering> (Datanode Uuid unassigned)
15/07/23 09:18:30 WARN datanode.DataNode: Exiting Datanode
15/07/23 09:18:30 INFO util.ExitUtil: Exiting with status 0
15/07/23 09:18:30 INFO datanode.DataNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down DataNode at dcos-slave-03.novalocal/<slave-03 IP>
************************************************************/

Can any one help to find root cause of this?

elingg commented 9 years ago

Did you reinstall multiple times? It looks like it is possibly finding an old cluster id and that the datanode lock is acquired by another instance. One idea here is to uninstall and reinstall and let me know if you have the same issue.

AndriiOmelianenko commented 9 years ago

yes, I did reinstall it. I also removed hdfs-mesos record in Exhibitor for Zookeeper as shown in docs here: https://docs.mesosphere.com/services/hdfs/ I'll try to reproduce issue tomorrow.

AndriiOmelianenko commented 9 years ago

didn't manage to reproduce

elingg commented 9 years ago

Thanks @AndriiOmelianenko . Ok, I'll close this issue for now. If you can reproduce it, we can reopen. That error typically means that the uninstall steps were not done completely so there is data there from an old cluster and it can't overwrite it.