big-data-europe / docker-hadoop

Apache Hadoop docker image
2.18k stars 1.28k forks source link

Wrong hadoop version on tag 3.1.3 #66

Open jirojo2 opened 4 years ago

jirojo2 commented 4 years ago

Using the tag 2.0.0-hadoop3.1.3-java8 bash, the actual installed version is 3.2.1 and not 3.1.3.

~/docker-hive$ docker run -it bde2020/hadoop-namenode:2.0.0-hadoop3.1.3-java8 bash
Configuring core
 - Setting fs.defaultFS=hdfs://bc96e1de2c94:8020
Configuring hdfs
 - Setting dfs.namenode.name.dir=file:///hadoop/dfs/name
Configuring yarn
Configuring httpfs
Configuring kms
Configuring mapred
Configuring for multihomed network
root@bc96e1de2c94:/# echo $PATH
/opt/hadoop-3.2.1/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
root@784684f7199c:/bin# which hdfs
/opt/hadoop-3.2.1/bin//hdfs

Also, run.sh fails to run as hdfs cannot be found, presumably related to this different version being installed.

$ docker logs 765
Configuring core
 - Setting hadoop.proxyuser.hue.hosts=*
 - Setting fs.defaultFS=hdfs://namenode:8020
 - Setting hadoop.http.staticuser.user=root
 - Setting hadoop.proxyuser.hue.groups=*
Configuring hdfs
 - Setting dfs.namenode.datanode.registration.ip-hostname-check=false
 - Setting dfs.webhdfs.enabled=true
 - Setting dfs.permissions.enabled=false
 - Setting dfs.namenode.name.dir=file:///hadoop/dfs/name
Configuring yarn
 - Setting yarn.timeline-service.enabled=true
 - Setting yarn.resourcemanager.system-metrics-publisher.enabled=true
 - Setting yarn.resourcemanager.store.class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore
 - Setting yarn.log.server.url=http://historyserver:8188/applicationhistory/logs/
 - Setting yarn.resourcemanager.fs.state-store.uri=/rmstate
 - Setting yarn.timeline-service.generic-application-history.enabled=true
 - Setting yarn.log-aggregation-enable=true
 - Setting yarn.resourcemanager.hostname=resourcemanager
 - Setting yarn.resourcemanager.resource_tracker.address=resourcemanager:8031
 - Setting yarn.timeline-service.hostname=historyserver
 - Setting yarn.resourcemanager.scheduler.address=resourcemanager:8030
 - Setting yarn.resourcemanager.address=resourcemanager:8032
 - Setting yarn.nodemanager.remote-app-log-dir=/app-logs
 - Setting yarn.resourcemanager.recovery.enabled=true
Configuring httpfs
Configuring kms
Configuring mapred
Configuring for multihomed network
Formatting namenode name directory: /hadoop/dfs/name
/run.sh: line 16: /bin/hdfs: No such file or directory
/run.sh: line 19: /bin/hdfs: No such file or directory
Data-drone commented 4 years ago

you can download the code, load the 3.1.3 branch then run make and build the image yourself locally then launch it as a workaround

gmouchakis commented 4 years ago

This should be already fixed. @Data-drone did you have the same issue?

Data-drone commented 4 years ago

was just suggesting a workaround. I haven't used the 2.0.0-hadoop3.1.3-java8 version