viiyu / memo

0 stars 0 forks source link

hadoop/hdfs/pipes/mapreduce #7

Open viiyu opened 5 years ago

viiyu commented 5 years ago

Hadoop2.7.x is incompatible with springboot1.5.11

exclude slf4j/servlet-api in all hadoop dependency, for example:

    compile ("org.apache.hadoop:hadoop-client:$hadoopVersion") {
        exclude module: 'slf4j-log4j12'
        exclude module: 'servlet-api'
    }

viiyu commented 5 years ago

Tips of submit a mapreduce job with DCE by Hadoop2.7.x

yarn prepare

all yarn machine should install docker,
add user yarn/hdfs into docker user group
edit yarn config, make yarn.nodemanager.container-executor.class= org.apache.hadoop.yarn.server.nodemanager.DockerContainerExecutor

submit

viiyu commented 5 years ago

Diagnostics: Image:xxxxxx is not a proper docker image

in hadoop2.7~2.9, DockerContainerExecutor do validation of container image name, the pattern is ^(([\\w\\.-]+)(:\\d+)*\\/)?[\\w\\.:-]+$in DockerContainerExecutor.java