Build error on Fedora - Githubissues

aiguofer commented 7 years ago

I'm getting the following error when I try to build... I installed all boost related rpms to no avail. I tried googling everywhere and I don't see much related to server.hpp

In file included from <path>/deepdetect/src/deepdetect.cc:25:0:
<path>l/deepdetect/src/httpjsonapi.h:26:50: fatal error: boost/network/protocol/http/server.hpp: No such file or directory
 #include <boost/network/protocol/http/server.hpp>
                                                  ^
compilation terminated.
src/CMakeFiles/ddetect.dir/build.make:62: recipe for target 'src/CMakeFiles/ddetect.dir/deepdetect.cc.o' failed
make[2]: *** [src/CMakeFiles/ddetect.dir/deepdetect.cc.o] Error 1
CMakeFiles/Makefile2:122: recipe for target 'src/CMakeFiles/ddetect.dir/all' failed
make[1]: *** [src/CMakeFiles/ddetect.dir/all] Error 2
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2

beniz commented 7 years ago

Hmm, this is used by cppnetlib. First I'd recommend to compile cppnetlib by hand as packaged versions are most often pretty old. Look ate the Ubuntu 15.10 and 16.04 deepdetect build issues in case this could be useful. Another handle is to look where the server.hpp file lies on your system. I'm not familiar with Fedora but boost is often broken into small subpackages and you just miss one here. Let me know how this goes, and if you can share your steps here it would be useful to other. Also start without GPU / Cuda, it'd be easier for a first build.

Note: to get up and running quickly, I'd suggest the docker builds.

aiguofer commented 7 years ago

Thanks for the quick response!

I had originally tried the docker image but was getting errors when trying to train a text classifier.. I'm not very familiar with docker so I thought it'd be easier to debug if I ran the server locally.

I'm trying with docker again and attaching to the container

I'm running this to train:

python dd_train.py --model-repo ./sbren --training-repo ./sbren --sname sbren --tsplit 0.1 --base-lr 0.01 --sequence 1014 --iterations 150000 --test-interval 1000 --nclasses 38

and I can see the following error in the container

E1104 19:10:11.987526    20 caffemodel.cc:69] error reading or listing caffe models in repository ./sbren

and I've tried various combinations for the model-repo and training-repo and none of them have worked, I'm thinking because the image can't see my filesystem?

Also, to verify... the model-repo will be the output directory and the training-repo is the "input" directory, correct?

beniz commented 7 years ago

I think the server tells you what the problem is: the server is looking locally within the container for your model repo but my guess is that it is fact on your host. Look at the DD +docker doc, it tells you how to share a directory with the container. Let me know if it is not clear.

beniz commented 7 years ago

Also I'd recommend to keep the training and model repositories different

aiguofer commented 7 years ago

ahh great thanks! I got it training now.

I'll go back to trying to get the build to work though, I'll keep this open until I figure out the build issue

aiguofer commented 7 years ago

Well, I compiled cpp-netlib manually and copied it, and asio, into src. This got me past my previous error, but now I'm getting:

[ 33%] Built target caffe_dd
[ 37%] Building CXX object src/CMakeFiles/ddetect.dir/deepdetect.cc.o
In file included from /home/rdu/difernan/Devel/deepdetect/src/services.h:29:0,
                 from /home/rdu/difernan/Devel/deepdetect/src/apistrategy.h:26,
                 from /home/rdu/difernan/Devel/deepdetect/src/deepdetect.h:25,
                 from /home/rdu/difernan/Devel/deepdetect/src/deepdetect.cc:22:
/home/rdu/difernan/Devel/deepdetect/src/imginputfileconn.h: In member function ‘void dd::ImgInputFileConn::transform(const dd::APIData&)’:
/home/rdu/difernan/Devel/deepdetect/src/imginputfileconn.h:320:4: error: ‘mt19937’ is not a member of ‘std’
    std::mt19937 g;
    ^~~
/home/rdu/difernan/Devel/deepdetect/src/imginputfileconn.h:322:6: error: ‘g’ was not declared in this scope
      g = std::mt19937(_seed);
      ^
/home/rdu/difernan/Devel/deepdetect/src/imginputfileconn.h:322:10: error: ‘mt19937’ is not a member of ‘std’
      g = std::mt19937(_seed);
          ^~~
/home/rdu/difernan/Devel/deepdetect/src/imginputfileconn.h:325:8: error: ‘random_device’ is not a member of ‘std’
        std::random_device rd;
        ^~~
/home/rdu/difernan/Devel/deepdetect/src/imginputfileconn.h:326:8: error: ‘g’ was not declared in this scope
        g = std::mt19937(rd());
        ^
/home/rdu/difernan/Devel/deepdetect/src/imginputfileconn.h:326:12: error: ‘mt19937’ is not a member of ‘std’
        g = std::mt19937(rd());
            ^~~
/home/rdu/difernan/Devel/deepdetect/src/imginputfileconn.h:326:28: error: ‘rd’ was not declared in this scope
        g = std::mt19937(rd());
                            ^
/home/rdu/difernan/Devel/deepdetect/src/imginputfileconn.h:328:47: error: ‘g’ was not declared in this scope
    std::shuffle(_images.begin(),_images.end(),g); //XXX beware: labels are not shuffled, i.e. let's not shuffle while testing
                                               ^
In file included from /home/rdu/difernan/Devel/deepdetect/src/caffeinputconns.h:28:0,
                 from /home/rdu/difernan/Devel/deepdetect/src/imginputfileconn.h:370,
                 from /home/rdu/difernan/Devel/deepdetect/src/services.h:29,
                 from /home/rdu/difernan/Devel/deepdetect/src/apistrategy.h:26,
                 from /home/rdu/difernan/Devel/deepdetect/src/deepdetect.h:25,
                 from /home/rdu/difernan/Devel/deepdetect/src/deepdetect.cc:22:

looking at some stuff on SO I think I need -std=c++0x, but I tried CXXFLAGS="-std=c++0x" make and that didn't work.

beniz commented 7 years ago

You need at least gcc4.8. You shouldn't need to tweak the app flags.

aiguofer commented 7 years ago

gcc --version
gcc (GCC) 6.2.1 20160916 (Red Hat 6.2.1-2)
Copyright (C) 2016 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

beniz commented 7 years ago

OK, two things:

try adding #include <random> to the headers of imginputfileconn.h
if that doesn't work, do you have any way to test with gcc5.

I haven't tested gcc6 yet, but I'll do it soon.

aiguofer commented 7 years ago

#include <random> in imginputfileconn.h got my past that point! also had to add it to txtinputfileconn.h but now I'm getting:

[ 50%] Building CXX object src/CMakeFiles/ddetect.dir/txtinputfileconn.cc.o
/home/rdu/difernan/Devel/deepdetect/src/txtinputfileconn.cc: In member function ‘void dd::TxtInputFileConn::parse_content(const string&, const float&)’:
/home/rdu/difernan/Devel/deepdetect/src/txtinputfileconn.cc:266:13: error: ‘utf8’ has not been declared
         c = utf8::next(str_i,end);
             ^~~~
/home/rdu/difernan/Devel/deepdetect/src/txtinputfileconn.cc: In member function ‘void dd::TxtInputFileConn::build_alphabet()’:
/home/rdu/difernan/Devel/deepdetect/src/txtinputfileconn.cc:350:15: error: ‘utf8’ has not been declared
  uint32_t c = utf8::next(str_i,end);

I had originally had an error about utf8.h missing so I had manually downloaded this into src: wget https://raw.githubusercontent.com/sheredom/utf8.h/master/utf8.h

beniz commented 7 years ago

OK, see PR #211 for the header fixes. This does not fix your utfcpp issue.

aiguofer commented 7 years ago

ahhh duh, I needed utf8cpp-devel... google failed me :)

Now I'm getting:

/home/rdu/difernan/Devel/deepdetect/src/httpjsonapi.cc:138:37: error: ‘boost::network::http::http_server<APIHandler>::response’ has not been declared
   void fillup_response(http_server::response &response,
                                     ^~~~~~~~
/home/rdu/difernan/Devel/deepdetect/src/httpjsonapi.cc:221:18: error: ‘boost::network::http::http_server<APIHandler>::response’ has not been declared
     http_server::response &response)
                  ^~~~~~~~
/home/rdu/difernan/Devel/deepdetect/src/httpjsonapi.cc: In member function ‘void APIHandler::fillup_response(int&, const JDoc&, std::__cxx11::string&, int&, const string&)’:
/home/rdu/difernan/Devel/deepdetect/src/httpjsonapi.cc:209:29: error: ‘boost::network::http::http_server<APIHandler>::response’ has not been declared
     response = http_server::response::stock_reply(http_server::response::status_type(outcode),stranswer);
                             ^~~~~~~~
/home/rdu/difernan/Devel/deepdetect/src/httpjsonapi.cc:209:64: error: ‘boost::network::http::http_server<APIHandler>::response’ has not been declared
     response = http_server::response::stock_reply(http_server::response::status_type(outcode),stranswer);
                                                                ^~~~~~~~
/home/rdu/difernan/Devel/deepdetect/src/httpjsonapi.cc:210:14: error: request for member ‘headers’ in ‘response’, which is of non-class type ‘int’
     response.headers[1].value = "application/json";
              ^~~~~~~
/home/rdu/difernan/Devel/deepdetect/src/httpjsonapi.cc:213:11: error: request for member ‘headers’ in ‘response’, which is of non-class type ‘int’
  response.headers.resize(3);
           ^~~~~~~
/home/rdu/difernan/Devel/deepdetect/src/httpjsonapi.cc:214:11: error: request for member ‘headers’ in ‘response’, which is of non-class type ‘int’
  response.headers[2].name = "Content-Encoding";
           ^~~~~~~
/home/rdu/difernan/Devel/deepdetect/src/httpjsonapi.cc:215:11: error: request for member ‘headers’ in ‘response’, which is of non-class type ‘int’
  response.headers[2].value = "gzip";
           ^~~~~~~
/home/rdu/difernan/Devel/deepdetect/src/httpjsonapi.cc:217:14: error: request for member ‘status’ in ‘response’, which is of non-class type ‘int’
     response.status = static_cast<http_server::response::status_type>(code);
              ^~~~~~
/home/rdu/difernan/Devel/deepdetect/src/httpjsonapi.cc:217:48: error: ‘response’ in ‘http_server {aka struct boost::network::http::server<APIHandler>}’ does not name a type
     response.status = static_cast<http_server::response::status_type>(code);
                                                ^~~~~~~~
/home/rdu/difernan/Devel/deepdetect/src/httpjsonapi.cc:217:56: error: expected ‘>’ before ‘::’ token
     response.status = static_cast<http_server::response::status_type>(code);
                                                        ^~
/home/rdu/difernan/Devel/deepdetect/src/httpjsonapi.cc:217:56: error: expected ‘(’ before ‘::’ token
/home/rdu/difernan/Devel/deepdetect/src/httpjsonapi.cc:217:56: error: ‘::status_type’ has not been declared
/home/rdu/difernan/Devel/deepdetect/src/httpjsonapi.cc:217:76: error: expected ‘)’ before ‘;’ token
     response.status = static_cast<http_server::response::status_type>(code);
                                                                            ^
/home/rdu/difernan/Devel/deepdetect/src/httpjsonapi.cc: In member function ‘void APIHandler::operator()(const request&, int&)’:
/home/rdu/difernan/Devel/deepdetect/src/httpjsonapi.cc:250:26: error: ‘boost::network::http::http_server<APIHandler>::response’ has not been declared
  response = http_server::response::stock_reply(http_server::response::not_found,_hja->jrender(_hja->dd_not_found_404()));
                          ^~~~~~~~
/home/rdu/difernan/Devel/deepdetect/src/httpjsonapi.cc:250:61: error: ‘boost::network::http::http_server<APIHandler>::response’ has not been declared
  response = http_server::response::stock_reply(http_server::response::not_found,_hja->jrender(_hja->dd_not_found_404()));
                                                             ^~~~~~~~
/home/rdu/difernan/Devel/deepdetect/src/httpjsonapi.cc:350:26: error: ‘boost::network::http::http_server<APIHandler>::response’ has not been declared
  response = http_server::response::stock_reply(http_server::response::not_found,_hja->jrender(_hja->dd_not_found_404()));
                          ^~~~~~~~
/home/rdu/difernan/Devel/deepdetect/src/httpjsonapi.cc:350:61: error: ‘boost::network::http::http_server<APIHandler>::response’ has not been declared
  response = http_server::response::stock_reply(http_server::response::not_found,_hja->jrender(_hja->dd_not_found_404()));
                                                             ^~~~~~~~

aiguofer commented 7 years ago

btw, the version I manually compiled was: cpp-netlib-0.12.0-final.tar.gz

beniz commented 7 years ago

Looks like https://github.com/performous/performous/issues/201.

Could you try against 0.11.2 at the moment ?

Due to #207 users are testing the forthcoming 0.13 and they successfully built againt it on Ubuntu 16.04. I'll try 0.12 + gcc6.

aiguofer commented 7 years ago

I tried 0.11.2 and I think it got me past that (I did a make clean at some point), but now I'm getting:

/usr/bin/ld: cannot find -lcppnetlib-uri
collect2: error: ld returned 1 exit status
main/CMakeFiles/dede.dir/build.make:121: recipe for target 'main/dede' failed
make[2]: *** [main/dede] Error 1
CMakeFiles/Makefile2:177: recipe for target 'main/CMakeFiles/dede.dir/all' failed
make[1]: *** [main/CMakeFiles/dede.dir/all] Error 2
Makefile:83: recipe for target 'all' failed

btw, what I've been doing is symlinking the boost dir from the cpp-netlib dir into the src dir so it builds it all together.

beniz commented 7 years ago

have you tried locating this lib in the cppnetlib directory ? It could be found in `build/libs/network/src. One you have it, adding its path to LD_LIBRARY_PATH should do the trick.

aiguofer commented 7 years ago

All I see in there is are static libs:

ls ~/cpp-netlib/build/libs/network/src
CMakeFiles  cmake_install.cmake  CTestTestfile.cmake  libcppnetlib-client-connections.a  libcppnetlib-server-parsers.a  libcppnetlib-uri.a  Makefile

I tried with LD_LIBRARY_PATH pointing there but nothing

beniz commented 7 years ago

hmm, one way could be to directly add the lib to https://github.com/beniz/deepdetect/blob/master/main/CMakeLists.txt. Look how its done for static libs in tests/cmakelists.txt

aiguofer commented 7 years ago

WOOOOO THAT WORKED!!! You are amazing, thanks for all the prompt feedback!

I changed the last line to

target_link_libraries (dede ddetect ${CUDA_LIB_DEPS} glog gflags ${OpenCV_LIBS} curlpp curl crypto ssl ${Boost_LIBRARIES} ${CAFFE_LIB_DEPS} ${XGBOOST_LIB_DEPS} ${TF_LIB_DEPS} $HOME/cpp-netlib/build/libs/network/src/libcppnetlib-uri.a)

I'll try to write up how I built it on Fedora 24 in case it's helpful for you or others

aiguofer commented 7 years ago

So I just re-built it with tests enabled...

When I run make the first time, I get:

/usr/bin/ld: cannot find -lcaffe
collect2: error: ld returned 1 exit status
main/CMakeFiles/dede.dir/build.make:122: recipe for target 'main/dede' failed
make[2]: *** [main/dede] Error 1
CMakeFiles/Makefile2:177: recipe for target 'main/CMakeFiles/dede.dir/all' failed
make[1]: *** [main/CMakeFiles/dede.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....

but when I run make again, it finishes fine... thinking it might be an issue with order of LD deps?

However, when I run ctests I get:

Test project /home/aiguofer/deepdetect/build/tests
No tests were found!!!

beniz commented 7 years ago

Maybe it is the make -j that fails compiling the Caffe proto file before the C++ files. This can happen depending on your hardware configuration.

Check that you have something in build/tests.

aiguofer commented 7 years ago

Ahh, yeah running without -j4 makes the compilation run fine.

And here's what's in build/tests:

ls tests
CMakeFiles  Testing  cmake_install.cmake  CTestTestfile.cmake  Makefile

beniz commented 7 years ago

Looks like the tests are not activated. Most likely this is due to missing gtest package.

jolibrain / deepdetect

Build error on Fedora #210