search

bigdatafoundation / docker-hadoop

Dockerfile for running Hadoop on Ubuntu
Apache License 2.0
84 stars 37 forks source link

Continuous integration: Automatize the testing of the docker image #17

Open davidonlaptop opened 9 years ago

davidonlaptop commented 9 years ago

Use something like Travis to automatize the testing of contributions. It would be great if Travis (or some other tool) tries to build the docker image upon a pull request and comment on the issue, whether or not the pull request broke the build.

It would be great if it could also test the commands in the README.

codingtony commented 9 years ago

It's a good idea! I don't know Travis a lot, but I found an example that runs docker with Travis : https://github.com/lukecyca/travis-docker-example.

It might be complex to determine the criteria of success

davidonlaptop commented 9 years ago

I guess, at the minimum we should to check if the docker build succeeded or not?

Then, I guess it would similar to how you test configuration management.. can we format HDFS. Can we start a namenode? Does it accept writing / reading data?

davidonlaptop commented 9 years ago

I haven't used Travis CI either, but looks like one of the most popular of its kind, especially for opensource.

davidonlaptop commented 9 years ago

For reference, here's the current script to test the building of an image:

echo "Rebuilding image ..."
docker build --no-cache -t hadoop-test .

echo "Remove previous containers (if they exists)"
docker stop hdfs-secondarynamenode hdfs-datanode1 hdfs-namenode
docker rm hdfs-secondarynamenode hdfs-datanode1 hdfs-namenode

echo "Create data directory and format HDFS"
rm -fr ~/data/hdfs-test
mkdir -p ~/data/hdfs-test

docker run --rm -i \
    -v $HOME/data/hdfs-test:/data \
    hadoop-test hdfs namenode -format

echo "Starting the NameNode"
docker run -d --name hdfs-namenode \
    -h hdfs-namenode -p 50070:50070 \
    -v $HOME/data/hdfs-test:/data \
    hadoop-test hdfs namenode && \
docker logs -f hdfs-namenode
# Hit CTRL+C manually if there is NO error and you see this message:
# 15/10/01 22:54:00 INFO namenode.NameNode: NameNode RPC up at: hdfs-namenode/172.17.0.31:9000

echo "Starting the DataNode"
docker run -d --name hdfs-datanode1 \
    -h hdfs-datanode1 -p 50075:50075 \
    -v $HOME/data/hdfs-test:/data \
    hadoop-test hdfs datanode && \
docker logs -f hdfs-datanode1
# Hit CTRL+C manually if there is NO error and you see this message:
# 15/10/01 22:54:11 INFO datanode.DataNode: Sent 1 blockreports 0 blocks total. Took 1 msec to generate and 35 msecs for RPC and NN processing.  Got back commands org.apache.hadoop.hdfs.server.protocol.FinalizeCommand@5403fa9a

echo "Starting the Secondary NameNode"
docker run -d --name hdfs-secondarynamenode \
    -h hdfs-secondarynamenode -p 50090:50090 \
    -v $HOME/data/hdfs-test:/data \
    hadoop-test hdfs secondarynamenode && \
docker logs -f hdfs-secondarynamenode
# Hit CTRL+C manually if there is NO error and you see this message:
# 15/10/01 22:56:57 INFO namenode.SecondaryNameNode: Web server init done

echo "Putting some data in HDFS"
docker run --rm -i hadoop-test hadoop checknative
docker run --rm -i hadoop-test hadoop fs -ls /
docker run --rm -i hadoop-test hadoop fs -mkdir /test
docker run --rm -i hadoop-test hadoop fs -put /bin/grep /test/
docker run --rm -i hadoop-test hadoop fs -ls -R /

echo "Cleaning up..."
#docker stop hdfs-secondarynamenode hdfs-datanode1 hdfs-namenode
#docker rm hdfs-secondarynamenode hdfs-datanode1 hdfs-namenode
#rm -fr ~/data/hdfs-test