search

CUFCTL / face-recognition

A GPU-accelerated real-time face recognition system based on classical machine learning algorithms
MIT License
23 stars 11 forks source link

Prepare the C code for Palmetto and CUDA #14

Closed bentsherman closed 7 years ago

bentsherman commented 8 years ago

Palmetto is Clemson University's super-computing cluster, you can access it via SSH if you register an account with CITI. One long-term goal of this project is to make our code run on Palmetto so that any performance measurements that we make will come from state-of-the-art hardware (people won't take our papers seriously if we just say that we ran our code on our laptops). Palmetto also provides access to GPUs, so if we get our code to run on Palmetto then we'll have a good platform to test CUDA code. With that in mind, here's the plan:

  1. Get our C code to run on Palmetto
  2. Get our C code to compile with CUDA
  3. Incorporate CUDA into our C code (matrix library, run LDA/ICA in parallel, etc.)

Jesse and I have been exploring this thread on and off for a few weeks now, so I will share the steps we have taken as they yield successful results. We will eventually incorporate these steps into the repository in the form of documentation and any necessary code changes.

Get our C code to run on Palmetto

The Intel Math Kernel Library includes Intel's implementation of BLAS and LAPACK, optimized for Intel hardware. There is currently a Palmetto module called intel which includes MKL and Intel's compiler suite (icc, ifortran, etc). This software is proprietary, so I don't expect that any of us will have this software on our own machines, but I think this will become our software of choice for our Palmetto build.

That being said, I have modified our Makefile to use Intel MKL. To build with MKL on Palmetto:

module load intel/16.0
make MATLIB=mkl

Get our C code to compile with CUDA

Incorporate CUDA into our C code

CUDA is a framework for writing code that can be run on a GPU. cuBLAS is NVIDIA's implementation of BLAS in CUDA. MAGMA is an implementation of LAPACK in CUDA, developed by the creators of LAPACK, and it is a wrapper for cuBLAS.

With that in mind, I am currently trying to incorporate MAGMA into our matrix library. For future reference, this is how you will be able to run our code on a GPU on Palmetto:

  1. Load the CUDA module 
    module load cuda-toolkit/7.5.18
  2. Build the project with CUDA: 
    make MATLIB=cuda
  3. Start an interactive job with a GPU on Palmetto (I recommend that you create an alias for this command): 
    qsub -I -l select=1:ncpus=8:mpiprocs=1:ngpus=1:mem=16gb:gpu_model=k40;walltime=02:00:00
  4. Run the executable file in the GPU shell 
    cd face-rec
./test-matrix
./face-rec ...
bentsherman commented 7 years ago

I haven't found a package for Magma yet, so we'll have to build from source:

# the Makefile looks in ../magma-2.2.0
cd face-rec/../

# download MAGMA
wget http://icl.cs.utk.edu/projectsfiles/magma/downloads/magma-2.2.0.tar.gz
tar -xvf magma-2.2.0.tar.gz magma-2.2.0

# pick make.inc (on Palmetto you can use MKL/icc, I haven't tried any others yet)
cp make.inc-examples/make.inc.mkl-icc

# build (set num-jobs to the number of cores in your CPU)
make -j [num-jobs]
bentsherman commented 7 years ago

I have added MAGMA to our matrix library, and it is working with a few exceptions, so I will move this discussion to a new, more specific issue.