search

yahoo / CaffeOnSpark

Distributed deep learning on Hadoop and Spark clusters.
Apache License 2.0
1.27k stars 358 forks source link

Adding Docker support for CaffeOnSpark #208

Closed arundasan91 closed 7 years ago

arundasan91 commented 7 years ago

Hello,

Please test the docker file locally. Steps are documented in docker/README.md.

The docker image is for CPU version of CaffeOnSpark. I will work on a GPU version soon.

I checked the working twice on my local machine.

@anfeng , @mriduljain , Could you guys please let me know if this is okay ? It was a bit tricky getting Hadoop to work on Docker. I had to refer "https://hub.docker.com/r/sequenceiq/hadoop-docker/" to get it working finally.

Also, I found that PATH variables were not set using Dockerfile. I had to set them explicitly by calling a bootstrap.sh script when we start the container for the first time (No need to use it while attaching to an already existing container). bootstrap.sh also makes sure that hadoop dfs and yarn starts with the container.

yahoocla commented 7 years ago

CLA is valid!

mriduljain commented 7 years ago

Docker support is a much needed feature which I was looking fwd to. I will test it early next week. Thanks!

On Thu, Dec 8, 2016 at 11:01 AM, Arun Das notifications@github.com wrote:

Hello,

Please test the docker file locally. Steps are documented in docker/README.md.

The docker image is for CPU version of CaffeOnSpark. I will work on a GPU version soon.

I checked the working twice on my local machine.

@anfeng https://github.com/anfeng , @mriduljain https://github.com/mriduljain , Could you guys please let me know if this is okay ? It was a bit tricky getting Hadoop to work on Docker. I had to refer "https://hub.docker.com/r/sequenceiq/hadoop-docker/" to get it working finally.

Also, I found that PATH variables were not set using Dockerfile. I had to set them explicitly by calling a bootstrap.sh script when we start the container for the first time (No need to use it while attaching to an already existing container). bootstrap.sh also makes sure that hadoop dfs and yarn starts with the container.

You can view, comment on, or merge this pull request online at:

https://github.com/yahoo/CaffeOnSpark/pull/208 Commit Summary

  • Create Dockerfile
  • Create ssh_config
  • Rename docker/standalone/cpu/ssh_config to docker/standalone/cpu/config/ssh_config
  • Create bootstrap.sh
  • Create README.md

File Changes

Patch Links:

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/yahoo/CaffeOnSpark/pull/208, or mute the thread https://github.com/notifications/unsubscribe-auth/ACCTVQQULYumBzvWq6ELjJ5AdiIj24Blks5rGFP1gaJpZM4LIKoR .

javadba commented 7 years ago

This is great idea! Thanks for contributing this!

arundasan91 commented 7 years ago

@javadba , were you able to get it working ? Could you please give it a try ?

anfeng commented 7 years ago

@arundasan91 Can you expand this PR with GPU support?

arundasan91 commented 7 years ago

@anfeng , sure can. I was waiting for a reply to make sure that this CPU version is enough. I will add a folder named gpu and a Dockerfile to support NVIDIA GPU's.

I have a doubt on installing the correct version of CUDA based on the GPU's architecture and cuda compute. Will refer caffe's own way of doing it.

arundasan91 commented 7 years ago

@anfeng , Please see the changes. A GPU version of the Dockerfile is added. I tested the same in an NVIDIA K80 environment. Works great.

The only change to note here is to use nvidia-docker instead of docker for GPU builds. I will go ahead and update the README. Also, the host machine should have the proper NVIDIA drivers installed (not cuda).

Please let me know if there is any changes/additions required.

anfeng commented 7 years ago

@arundasan91 Good work. Can you add copyright notices to all new source files? Let's avoid mention your name in the files, to be consistent with all other source codes.

@javadba Have you tried out this PR?

arundasan91 commented 7 years ago

@anfeng , Yes I will. I totally forgot to remove my name from the maintainer tag. Will do.

arundasan91 commented 7 years ago

@anfeng , Just to make it clear, these are the copyright lines right ? Shall I copy the same into every file in the PR ?

# Copyright 2016 Yahoo Inc.
# Licensed under the terms of the Apache 2.0 license.
# Please see LICENSE file in the project root for terms.
#
# This file.......

Thanks.

javadba commented 7 years ago

@anfeng I have not had a chance to try out the PR. Probably about two weeks out due to some other fires. But definitely of interest - thanks.

2017-01-19 14:04 GMT-08:00 Arun Das notifications@github.com:

@anfeng https://github.com/anfeng , Just to make it clear, these are the copyright lines right ? Shall I copy the same into every file in the PR ?

Copyright 2016 Yahoo Inc.

Licensed under the terms of the Apache 2.0 license.

Please see LICENSE file in the project root for terms.

#

This file.......

Thanks.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/yahoo/CaffeOnSpark/pull/208#issuecomment-273913312, or mute the thread https://github.com/notifications/unsubscribe-auth/AAZEkdyxI6C_-Y0iSzQxv5AEYjvwl279ks5rT933gaJpZM4LIKoR .

anfeng commented 7 years ago

@arundasan91 Yes, for all new files.

arundasan91 commented 7 years ago

@anfeng , I added the copyright to every file.

I added them to the config/ssh_config file too. This file will be copied to the containers' .ssh/config. Since # will be treated as comments, this should be okay.

Could you please take a look at the files and let me know. Thanks.

anfeng commented 7 years ago

+1 @arundasan91 Excellent work.