search

big-data-europe / docker-hadoop-spark-workbench

[EXPERIMENTAL] This repo includes deployment instructions for running HDFS/Spark inside docker containers. Also includes spark-notebook and HDFS FileBrowser.
688 stars 373 forks source link

org.apache.hadoop.hdfs.server.common.IncorrectVersionException #46

Closed Vzzarr closed 6 years ago

Vzzarr commented 6 years ago

Thanks for the reply to the previous issue

I tried to execute the new setup as you suggested me in the previous issue; according to the README.md, after executing make hadoop I'm facing with this problem:

after the command sudo docker service ls I notice that hadoop_datanode doesn't start, so investigating with sudo docker logs -f hadoop_datanode.yrqg6y0x6i9r6fzumsmas6w2d.6gmr55bf2tc68cahnyjuu5xkw I found this exception: org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /hadoop/dfs/data. Reported: -57. Expecting = -56.

Is there anything wrong that I could have done? Or there is any incompatibility between the docker containers?

I am waiting for a kind reply, ask me if you need for more info from the stack-trace

earthquakesan commented 6 years ago

Hi @Vzzarr !

Incompatible versions of HDFS means that you have started HDFS previously with different version. Now, you are trying to start HDFS with superior version and it fails. I.e. it happens when you do:

  1. start docker-compose with Hadoop 1.0.0 (for example)
  2. change docker-compose to Hadoop 2.7.4 Here is the point: the volumes created on step 1 will remain on your machine. Thus:
  3. IncorrectVersionException

To avoid it simply wipe the data folder (if you are using the local disk mount) or remove corresponding docker volumes.

Vzzarr commented 6 years ago

OK thanks for your answer, indeed I removed the images using "different approaches", but I think that this approach worked:

  1. sudo systemctl stop docker
  2. sudo rm -rf /var/lib/docker (or at least the folder specified in Docker Root Dir:, obtainable with the command sudo docker info)
  3. sudo systemctl start docker
  4. re-pull the images from the Docker-Hub

Maybe this is an approach too hard but at least it worked for me; probably was necessary just deleting docker rmi bde2020/hadoop-datanode:1.1.0-hadoop2.8-java8 and sudo docker rmi bde2020/hadoop-namenode:1.1.0-hadoop2.8-java8 and re-pulling the images from your Docker-Hub