search

yahoo / CaffeOnSpark

Distributed deep learning on Hadoop and Spark clusters.
Apache License 2.0
1.27k stars 357 forks source link

Installation on NVIDIA Jetson TK1 #221

Open Cogito2012 opened 7 years ago

Cogito2012 commented 7 years ago

Is there any successful installation cases on TK1 board? I doubt whether CaffeOnSpark supports for mobile devices such as NVIDIA Jetson TK1.

arundasan91 commented 7 years ago

@Cogito2012 , I have not done it myself but can give you some pointers. One of my friends have a board which I can borrow, let me see if I can get my hands on one. I am positive that it will work. I was able to run Caffe in a Raspberry Pi that too with limited HW and SW support. TK1 boards support different operating systems as mentioned here. It should work ! Try these:

  1. (Hard way) If you are running Linux4Tegra or a similar ubuntu based OS, please check if you can run hadoop and spark on it and if they work properly install CaffeOnSpark manually. --OR--
  2. (Easy way) Docker will work on Ubuntu. Try running a CaffeOnSpark docker container and see if everything works out. Docker support will soon come to COS main branch. But for now, please go to https://github.com/arundasan91/CaffeOnSpark_old/tree/patch-3/docker and follow the instructions there.
junshi15 commented 7 years ago

BVLC caffe runs on Jetson TK1. I don't know about Spark. if Spark works on Jetson as well, then it is very likely you can run CaffeOnSpark on it.

Cogito2012 commented 7 years ago

@arundasan91 @junshi15 I'm grateful for your guidance, I'll have a try~~

arundasan91 commented 7 years ago

@Cogito2012 Hi, any updates ?

Cogito2012 commented 7 years ago

Not solved yet. The three main softwares (Caffe/Hadoop/Spark) can be seperately installed on TK1 board, but after they were all installed, the operation system became too slow to operate and even crushed for the limitted memory. Probably because of the improperly allocating the swap memory size, I'v been trying to solve it.

hyunjong commented 7 years ago

@Cogito2012 Did you have any issues when building CaffeOnSpark? I'm also now trying to install CaffeOnSpark on Jetson TX1, but it stops becuase of the protobuf version issue. I found out while googling that protobuf 2.5 does not support arm64, and unfortunately CaffeOnSpark only uses protobuf 2.5 Can I ask how did you solve this problem?

arundasan91 commented 7 years ago

@hyunjong , Try running CaffeOnSpark docker container on your TK1 and see if it works.

hyunjong commented 7 years ago

@arundasan91 I'm going to utilize the Jetson's GPU, so nvidia-docker needs to be installed. Unfortunately nvidia-docker is not supporting arm platform yet. https://github.com/NVIDIA/nvidia-docker/issues/214 I solved the protobuf issue, but got stuck at lmdbjni library building. I'll post an update when I make it run.

arundasan91 commented 7 years ago

@hyunjong , Thanks for the info. I didn't knew that. My friend is actually using his TK1 for some dev. No luck in getting it soon. Good job on getting the protobuf fixed. Please share the steps once you are done ! It will help many. Thanks a lot.