search

patrikhuber / eos

A lightweight 3D Morphable Face Model library in modern C++
Apache License 2.0
1.89k stars 596 forks source link

Bad wireframe projection using dlib landmarks #241

Open cfoch opened 5 years ago

cfoch commented 5 years ago

Hello. I will start the description of the issue by showing an image

output

I am using dlib to generate the landmarks, and taking as a base the example located at examples/fit-model.cpp. Unfortunately the result is obviously unexpected. I really have no idea what's going on. I do not think that is a problem of index ordering. I wonder if you may have any idea about this.

This is the code and the only main thing that I change is the way I read the landmarks that is from lines 187 to 193.

I also have to say that the original example with the landmarks located at image_0010.pts works well.

cfoch commented 5 years ago

I confirm that the problem is caused by building it in release mode with

set(CMAKE_BUILD_TYPE Release)

I at least need the Release mode to use it with dlib, otherwise dlib is too slow without the release mode. Why may this occur when I build this on release mode?

cfoch commented 5 years ago

A guy from the ##c++ IRC channel suggest to compile with -DCMAKE_C_FLAGS="-Wall -Wextra" and fix all the warnings, since there may be an undefined behavior.

patrikhuber commented 5 years ago

Hi,

So you're saying it works correctly in Debug mode, and in Release mode it produces the picture above? How about in RelWithDebInfo mode? Your code how you use the dlib landmarks looks okay on a very quick glance, given that you're using the 68-point dlib detector and they're in the ibug-68 order? It actually looks like you may have some mistake with the landmark ids/order, if you look at this figure https://ibug.doc.ic.ac.uk/resources/facial-point-annotations/, and compare with the red numbers on your screenshot, they're off-by-one, for example the left contour top point is 16 in your pic but 17 in the ibug scheme. This is very likely a user error on your side, I suggest you check everything and debug/step through your code.

cfoch commented 5 years ago

I will double check. But this is not the problem in the index order. I just compile the same code from the examples with "set(CMAKE_BUILD_TYPE Release)" and I get this problem. I remove the "set(CMAKE_BUILD_TYPE Release)" and it renders as expected. Just my Makefile is a bit different from yours.

Here is my CMakeLists.txt somewhat hard-coded because I don't know the best way to include eos yet.

cfoch commented 5 years ago

The problem is because I am compiling with

set(CMAKE_CXX_STANDARD 14)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
set(CMAKE_BUILD_TYPE Release)

So the combination of using -std=c++14 and Release mode is what leads to this problem. Just add those lines to the CMakeLists.txt from the examples and you should be able to reproduce the error.

cfoch commented 5 years ago

The reason why I compile with C++14 standard is because otherwise I get this error when I compile the code with eos+dlib: https://gist.github.com/cfoch/64b116cc1a2449051051ff14a9702daf

patrikhuber commented 5 years ago

Can you post your platform, exact compiler version and command-line output of the compiler call when compiling & linking the fit-model target? If you use a Makefile generator you should be able to do the last with make VERBOSE=1.

cfoch commented 5 years ago

Sorry for the delay...

[cfoch@localhost bin]$ uname -rio
4.18.16-300.fc29.x86_64 x86_64 GNU/Linux
[cfoch@localhost bin]$ cat /etc/redhat-release 
Fedora release 29 (Twenty Nine)

gcc/g++ version

[cfoch@localhost bin]$ g++ --version
g++ (GCC) 8.2.1 20180801 (Red Hat 8.2.1-2)
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[cfoch@localhost bin]$ gcc --version
gcc (GCC) 8.2.1 20180801 (Red Hat 8.2.1-2)
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Command line output when compiling fit-model with cmake (with set(CMAKE_CXX_STANDARD 14), set(CMAKE_CXX_STANDARD_REQUIRED ON) and set(CMAKE_BUILD_TYPE Release)):

[ 12%] Building CXX object examples/CMakeFiles/fit-model.dir/fit-model.cpp.o
cd /home/cfoch/dev/checkout/misc/eos/build/examples && /usr/bin/c++   -I/home/cfoch/dev/checkout/misc/eos/include -I/home/cfoch/dev/checkout/misc/eos/3rdparty/cereal/include -I/home/cfoch/dev/checkout/misc/eos/3rdparty/eigen -I/home/cfoch/dev/checkout/misc/eos/3rdparty/glm -I/home/cfoch/dev/checkout/misc/eos/3rdparty/nanoflann/include -I/home/cfoch/dev/checkout/misc/eos/3rdparty/eigen3-nnls/src -I/home/cfoch/dev/checkout/misc/eos/3rdparty/toml11 -isystem /usr/include/opencv  -O2 -DNDEBUG   -std=c++14 -o CMakeFiles/fit-model.dir/fit-model.cpp.o -c /home/cfoch/dev/checkout/misc/eos/examples/fit-model.cpp
[ 25%] Linking CXX executable fit-model
cd /home/cfoch/dev/checkout/misc/eos/build/examples && /usr/bin/cmake -E cmake_link_script CMakeFiles/fit-model.dir/link.txt --verbose=1
/usr/bin/c++  -O2 -DNDEBUG   CMakeFiles/fit-model.dir/fit-model.cpp.o  -o fit-model /usr/lib64/libopencv_imgcodecs.so.3.4.1 /usr/lib64/libboost_system.so /usr/lib64/libboost_filesystem.so /usr/lib64/libboost_program_options.so -pthread /usr/lib64/libopencv_imgproc.so.3.4.1 /usr/lib64/libopencv_core.so.3.4.1 
make[2]: Leaving directory '/home/cfoch/dev/checkout/misc/eos/build'
[ 25%] Built target fit-model
make -f examples/CMakeFiles/fit-model-multi.dir/build.make examples/CMakeFiles/fit-model-multi.dir/depend
make[2]: Entering directory '/home/cfoch/dev/checkout/misc/eos/build'
cd /home/cfoch/dev/checkout/misc/eos/build && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /home/cfoch/dev/checkout/misc/eos /home/cfoch/dev/checkout/misc/eos/examples /home/cfoch/dev/checkout/misc/eos/build /home/cfoch/dev/checkout/misc/eos/build/examples /home/cfoch/dev/checkout/misc/eos/build/examples/CMakeFiles/fit-model-multi.dir/DependInfo.cmake --color=
make[2]: Leaving directory '/home/cfoch/dev/checkout/misc/eos/build'
make -f examples/CMakeFiles/fit-model-multi.dir/build.make examples/CMakeFiles/fit-model-multi.dir/build
make[2]: Entering directory '/home/cfoch/dev/checkout/misc/eos/build'

Compiling as the the project is by default:

[ 12%] Building CXX object examples/CMakeFiles/fit-model.dir/fit-model.cpp.o
cd /home/cfoch/dev/checkout/misc/eos/build/examples && /usr/bin/c++   -I/home/cfoch/dev/checkout/misc/eos/include -I/home/cfoch/dev/checkout/misc/eos/3rdparty/cereal/include -I/home/cfoch/dev/checkout/misc/eos/3rdparty/eigen -I/home/cfoch/dev/checkout/misc/eos/3rdparty/glm -I/home/cfoch/dev/checkout/misc/eos/3rdparty/nanoflann/include -I/home/cfoch/dev/checkout/misc/eos/3rdparty/eigen3-nnls/src -I/home/cfoch/dev/checkout/misc/eos/3rdparty/toml11 -isystem /usr/include/opencv  -std=c++17 -o CMakeFiles/fit-model.dir/fit-model.cpp.o -c /home/cfoch/dev/checkout/misc/eos/examples/fit-model.cpp
[ 25%] Linking CXX executable fit-model
cd /home/cfoch/dev/checkout/misc/eos/build/examples && /usr/bin/cmake -E cmake_link_script CMakeFiles/fit-model.dir/link.txt --verbose=1
/usr/bin/c++     CMakeFiles/fit-model.dir/fit-model.cpp.o  -o fit-model /usr/lib64/libopencv_imgcodecs.so.3.4.1 /usr/lib64/libboost_system.so /usr/lib64/libboost_filesystem.so /usr/lib64/libboost_program_options.so -pthread /usr/lib64/libopencv_imgproc.so.3.4.1 /usr/lib64/libopencv_core.so.3.4.1 
make[2]: Leaving directory '/home/cfoch/dev/checkout/misc/eos/build'
[ 25%] Built target fit-model
make -f examples/CMakeFiles/fit-model-multi.dir/build.make examples/CMakeFiles/fit-model-multi.dir/depend
make[2]: Entering directory '/home/cfoch/dev/checkout/misc/eos/build'
cd /home/cfoch/dev/checkout/misc/eos/build && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /home/cfoch/dev/checkout/misc/eos /home/cfoch/dev/checkout/misc/eos/examples /home/cfoch/dev/checkout/misc/eos/build /home/cfoch/dev/checkout/misc/eos/build/examples /home/cfoch/dev/checkout/misc/eos/build/examples/CMakeFiles/fit-model-multi.dir/DependInfo.cmake --color=
make[2]: Leaving directory '/home/cfoch/dev/checkout/misc/eos/build'
make -f examples/CMakeFiles/fit-model-multi.dir/build.make examples/CMakeFiles/fit-model-multi.dir/build
make[2]: Entering directory '/home/cfoch/dev/checkout/misc/eos/build'
cfoch commented 5 years ago

Thank you. I solved it basically after checking the command line output using VERBOSE=1 as you suggested.

set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
set(CMAKE_BUILD_TYPE Release)

made the trick! :100:

patrikhuber commented 5 years ago

So what was the problem? Does it work in all configurations now?

cfoch commented 5 years ago

It is enough to me having this "just working". However, it does not work when building it on the C++14 standard. But since the description of this project states "A lightweight 3D Morphable Face Model fitting library in modern C++11/14", I think that this should work compiling it in the C++14 standard.

patrikhuber commented 5 years ago

I see. It should definitely work in all configurations and on all compilers, and it does so on all the ones that I've tested. If I'll ever find some time, I might test on your system/compiler configuration. I still think it's most likely that this is a problem on your system, and also this is the first report of such kind. I'll leave the issue open for now.