Current state of this fork

[lopho]

Hello dtmoodie,

I'm currently looking for a caffe version that works with both windows and linux. The Microsoft port apparently still has the limitation that any projects trying to use libcaffe have to be set up as a subproject of the caffe sln. So I stumbled on your fork which seems to try to mitigate this by making the cmake build usable for windows.

Is your fork already usable, using cmake? What about the merge branch? Are any special prerequisites required, like certain dependencies, vc12, etc... ?

Thank you for your time. lopho

[dtmoodie]

Hello lopho,

This fork of caffe is usable on windows and linux, I use it in a production application on windows, and I have it built and training networks on linux. The merge branch is where I've been staging some changes and updates for vs2015. It is not completely up to date, as I have one bug fix that I will commit once I get back to my work computer on monday, which is a pretty important one. That branch is also where I have replaced glog and gflags with boost log and boost program_options, so it has less dependencies than the bvlc branch.
I've only tested it on msvc 2013 and msvc 2015, so I can't be certain about older versions. For most of the caffe dependencies you can get binaries from whttps://github.com/dtmoodie/caffe_deps/ which has msvc2013 and msvc2015 binaries (less opencv because most people have their own build of that).

[lopho]

Ok. I'll try building it today with my local libraries. I'll keep this ticket open for a bit. If everything works out (or not) I'll try to give a bit of feedback. Thanks for your effort!

[dtmoodie]

Merge branch has been updated with bug fix.

[lopho]

I built the master branch on linux (gcc 4.8.4) and windows (vc12/vs2013), both using cmake. On linux it was necessary to pass the flags -DPROTO_EXPORTS="" and -DCAFFE_EXPORTS="" to the compiler, and I had to remove the usage of some C++11 functions. On windows it went more or less smoothly.

I went ahead and cherry picked only the modifications to BVLC caffe necessary to get the exports to work. Updated to latest BVLC master and modified new classes & layers to use caffe_pb.h and common.hpp and added missing DLL_EXPORTs. Smooth sailing.

I'm hesitant to use the merge branch as I want to keep the modifications to BVLC caffe to a minimum, so future merges might just work without much manual intervention. How did you handle the gflags and glog replacement? Is it a simple drop in, or do all calls to LOG(...) and gflags have to be replaced?

[dtmoodie]

Thanks for checking this out, I'll have to test my build against Linux again once I have a Linux box running again. Replacing glog was done by creating a caffe/logging.hpp file which had all the same logging macros, and then replacing the glog include with caffe/logging.hpp. All warning levels had to also be swapped to lower case, ie WARNING -> warning. gflags was only replaced in caffe.bin which involved re-writing the main function.

[lopho]

Ok, thanks for the info. Another thing that stood out to me, was that unit testing was disabled altogether on windows. I enabled the inclusion of the src/caffe/test/CMakelists.txt just to try it out, but when linking against libcaffe.lib it seems to have all symbols defined already and fails. Don't know how to get it to work yet. Maybe you have an idea?

Also, the RelWithDebInfo build configuration is always being overridden in cmake/Misc.cmake (first 3 lines)

# ---[ Configuration types
set(CMAKE_CONFIGURATION_TYPES "Debug;Release" CACHE STRING "Possible configurations" FORCE)
mark_as_advanced(CMAKE_CONFIGURATION_TYPES)

Adding RelWithDebInfo to this list fixes that problem.

[dtmoodie]

Thanks for the feedback. If I remember correctly, I was not able to get unit testing to work on windows while building shared libraries due to how linking works with cuda objects. If I remember correctly, the unit testing depends on being able to link to the actual device kernels which are not exported in a windows dll. I'm not familiar enough with linking cuda device code to be able to fix this, maybe it's just a matter of exporting the kernels. I believe opencv solves this by unit testing the C++ calls, not the kernel calls.

[lopho]

Back again with a new set of challenges. CPU_ONLY builds now work, including unittests all tools and examples. For the tests I just compiled all libcaffe sources directly into test.testbin. Tests also work with CUDA enabled.

But linking anything to libcaffe without CPU_ONLY (e.g. with CUDA enabled) fails with unresolved external symbols. caffe.bin for example cannot find caffe::caffe_gpu_dot<float>(...) when linking. Although __declspec(dllexport) is defined for that function in math_functions.hpp.

Has linking libcaffe with CUDA enabled worked for you? Or is it still WIP? Seems like just the same problem you mentioned regarding the unit tests.

[dtmoodie]

That's very odd, I've never actually compiled with CPU_ONLY. All my work has been with CUDA enabled and it links fine into external applications. Which branch are you using and can you paste the output from cmake config?

[lopho]

I'm currently using a custom branch. But basically it's your master with latest bvlc + ssd merged in. I did some modifications to the cmake files, nothing fancy, just to get the tests running. No changes to the math_functions.hpp/.cpp/.cu though.

I will try again using just your master branch with no modifications and check any differences.

Regardless, here's my CMake configure output:

Boost version: 1.59.0
Found the following Boost libraries:
  system
  thread
  date_time
  chrono
  filesystem
  atomic
Found gflags  (include: E:/work/svn_sdk/gflags/2.1.2/WIN64VC120/Debug/Include, library: optimized;E:/work/svn_sdk/gflags/2.1.2/WIN64VC120/Debug/Lib/gflags.lib;debug;E:/work/svn_sdk/gflags/2.1.2/WIN64VC120/Debug/Lib/gflags.lib)
Found glog    (include: E:/work/svn_sdk/glog/0.3.4/WIN64VC120/Release/include, library: optimized;E:/work/svn_sdk/caffe/1.0.0-rc3-lopho1/WIN64/third_party/lib/libglog.lib;debug;E:/work/svn_sdk/caffe/1.0.0-rc3-lopho1/WIN64/third_party/lib/libglog.lib)
Found PROTOBUF Compiler: E:/work/svn_sdk/protobuf/2.6.1/WIN64VC120/lib/x64/v120/Release/protoc.exe
Found lmdb    (include: E:/work/svn_sdk/lmdb/0.9.18/WIN64VC120/Release/include, library: E:/work/svn_sdk/lmdb/0.9.18/WIN64VC120/Release/lib/lmdb.lib)
CUDA detected: 7.5
Added CUDA NVCC flags for: sm_30 compute_30
OpenCV found (E:/work/svn_sdk/opencv/3.1.0/WIN64VC120)
Found OpenBLAS libraries: E:/work/svn_sdk/openblas/0.2.18/WIN64VC120_actually_0.2.14/lib/x64/libopenblas.dll.a
Found OpenBLAS include: E:/work/svn_sdk/openblas/0.2.18/WIN64VC120_actually_0.2.14/include
Python interface is disabled or not all required dependencies found. Building without it...
******************* Caffe Configuration Summary *******************
General:
  Version           :   1.0.0-rc3
  Git               :   f50cff2-dirty
  System            :   Windows
  C++ compiler      :   C:/Program Files (x86)/Microsoft Visual Studio 12.0/VC/bin/x86_amd64/cl.exe
  RelWithDebInfo    :   /MD /Zi /O2 /Ob1 /DNDEBUG /DWIN32 /D_WINDOWS /W3 /GR /EHsc /MP
  Release CXX flags :   /MD /O2 /Ob2 /DNDEBUG /Zo /Oy- /DWIN32 /D_WINDOWS /W3 /GR /EHsc /MP
  Debug CXX flags   :   /D_DEBUG /MDd /Zi /Ob0 /Od /RTC1 /DWIN32 /D_WINDOWS /W3 /GR /EHsc /MP
  Build type        :   Release
  BUILD_SHARED_LIBS :   ON
  BUILD_python      :   OFF
  BUILD_matlab      :   
  BUILD_docs        :   
  CPU_ONLY          :   OFF
  USE_OPENCV        :   ON
  USE_LEVELDB       :   OFF
  USE_LMDB          :   ON
  ALLOW_LMDB_NOLOCK :   OFF
Dependencies:
  BLAS              :   Yes (Open)
  Boost             :   Yes (ver. 1.59)
  glog              :   Yes
  gflags            :   Yes
  protobuf          :   Yes (ver. 2.6.1)
  lmdb              :   Yes (ver. 0.9.14)
  OpenCV            :   Yes (ver. 3.1.0)
  CUDA              :   Yes (ver. 7.5)
NVIDIA CUDA:
  Target GPU(s)     :   Manual
  GPU arch(s)       :   sm_30 compute_30
  cuDNN             :   Disabled
Install:
  Install path      :   E:/work/caffe/clean_build/install
Configuring done

[lopho]

The branch I'm currently working on: https://github.com/lopho/caffe/tree/lopho

[lopho]

AAaaah. Well. WHOOPS. I guess some template instancing got lost when merging ssd and bvlc. In math_functions.cu. Putting them back in, should work after that. Hopefully :)

[lopho]

Yeah... no... Still not linking. I really have no idea where the problem lies. I think I'll sleep over it, new ideas will come tomorrow I hope.

[lopho]

I found the culprit. Using your master branch. In caffe.bin e.g. caffe.cpp I reverted boost usage back to gflags. In the time(...) function the call on line 358: layers[i]->Forward(bottom_vecs[i], top_vecs[i]); This method of Layer is an inline method in the header and calls caffe::caffe_gpu_dot<float>(...)

include/caffe/layer.hpp (line 450)

// Forward and backward wrappers. You should implement the cpu and
// gpu specific implementations instead, and should not change these
// functions.
template <typename Dtype>
inline Dtype Layer<Dtype>::Forward(const vector<Blob<Dtype>*>& bottom,
    const vector<Blob<Dtype>*>& top) {
  // Lock during forward to ensure sequential forward
  Lock();
  Dtype loss = 0;
  Reshape(bottom, top);
  switch (Caffe::mode()) {
  case Caffe::CPU:
    Forward_cpu(bottom, top);
    for (int top_id = 0; top_id < top.size(); ++top_id) {
      if (!this->loss(top_id)) { continue; }
      const int count = top[top_id]->count();
      const Dtype* data = top[top_id]->cpu_data();
      const Dtype* loss_weights = top[top_id]->cpu_diff();
      loss += caffe_cpu_dot(count, data, loss_weights);
    }
    break;
  case Caffe::GPU:
    Forward_gpu(bottom, top);
#ifndef CPU_ONLY
    for (int top_id = 0; top_id < top.size(); ++top_id) {
      if (!this->loss(top_id)) { continue; }
      const int count = top[top_id]->count();
      const Dtype* data = top[top_id]->gpu_data();
      const Dtype* loss_weights = top[top_id]->gpu_diff();
      Dtype blob_loss = 0;
      caffe_gpu_dot(count, data, loss_weights, &blob_loss);
      loss += blob_loss;
    }
#endif
    break;
  default:
    LOG(FATAL) << "Unknown caffe mode.";
  }
  Unlock();
  return loss;
}

As such, when linking one has to link against a gpu math function, which does not seem to work. Why that is, I do not know yet. The first thing I will try is to put the method body into the implementation file instead of keeping it inline.

I also tried linking a simple binary with the following source:

#include "caffe/caffe.hpp"
int main(int argc, char** argv)
{
    float f1 = 0.1f;
    float f2 = 0.2f;
    float f3 = 0.3f;
    int i1 = 1;
    caffe::caffe_gpu_dot(i1, &f1, &f2, &f3);
    return 0;
}

Which fails the linker step with this error:

Error   3   error LNK2019: unresolved external symbol "void __cdecl caffe::caffe_gpu_dot<float>(int,float const *,float const *,float *)" (??$caffe_gpu_dot@M@caffe@@YAXHPEBM0PEAM@Z) referenced in function main   E:\work\caffe\clean_build\build_dtmoodie\tools\caffe.obj    caffe.bin
Error   4   error LNK1120: 1 unresolved externals   E:\work\caffe\clean_build\build_dtmoodie\tools\Debug\caffe-d.exe    1   1   caffe.bin

(I replaced the contents of caffe.cpp with my above example for testing purposes, just to simplify the setup)

This also happens with any other caffe::caffe_gpu_* function in math_functions.cu

[dtmoodie]

This does seem to be a mystery. I'm so sorry that I didn't run into these issues when initially developing the fork. I got it to the point where I could link it into an application and use it, and then didn't come back to some of the other details. I do believe I was able to avoid these linking problems by moving the layer implementation into a cpp file, since externally I never need to call a caffe math function.

[lopho]

No problem. I've gotten everything sorted out now, as long as nobody calls any cuda math functions directly. I'll clean everything up and push the branch to my repo. If you are interested you can take a look then. Thanks again. Your fork was a very big help to me!

[dtmoodie]

Glad this was useful, I'll look at your branch to try to clean up my master. My plan is to soon merge the "merge" branch into my master and try to keep it up to date with bvlc. Sans glog and gflags.

[lopho]

I pushed my branch to https://github.com/lopho/caffe/tree/lopho Be aware that this has https://github.com/weiliu89/caffe/tree/ssd merged as well.

My branch is not super clean but I think its "good enough".