Closed PeterSprague closed 6 years ago
You're right. Checkout the changes in my 16.10 pull request #677 those errors look familiar and I think I fixed them in that.
Thanks, included your commits into my cuda branch. Performing a docker build now. Will report on the outcome one way or another.
Fails at the same point with same errors. That would have been too easy.
Time for a beer, and leave this for a bit
Listening to Tragically Hip in Gord Downie's honour
Oh, try deleting the build commands for odm_texturing. That's way outdated and deprecated code.
Hmm, but I'm working off an upstream merge of the master?
Yeah, it hasnt been removed in master yet... whoops
If I'm understanding, remove the statement: RUN cd ../.. && mkdir build && cd build && cmake .. && make -j$(nproc)
aka don't do the second build outside SuperBuild?
add_subdirectory(odm_texturing)?
yeah
Thanks, doing docker build on revised code.
modules/odm_meshing/CMakeFiles/odm_meshing.dir/build.make:62: recipe for target 'modules/odm_meshing/CMakeFiles/odm_meshing.dir/src/OdmMeshing.cpp.o' failed
make[2]: [modules/odm_meshing/CMakeFiles/odm_meshing.dir/src/OdmMeshing.cpp.o] Error 1
make[2]: Waiting for unfinished jobs....
CMakeFiles/Makefile2:225: recipe for target 'modules/odm_meshing/CMakeFiles/odm_meshing.dir/all' failed
make[1]: [modules/odm_meshing/CMakeFiles/odm_meshing.dir/all] Error 2
make[1]: Waiting for unfinished jobs....
[ 87%] Linking CXX executable ../../bin/odm_orthophoto
CMakeFiles/odm_orthophoto.dir/src/OdmOrthoPhoto.cpp.o: In function OdmOrthoPhoto::createOrthoPhoto()': OdmOrthoPhoto.cpp:(.text+0x916c): undefined reference to
cv::imread(cv::String const&, int)'
OdmOrthoPhoto.cpp:(.text+0x96bb): undefined reference to cv::imwrite(cv::String const&, cv::_InputArray const&, std::vector<int, std::allocator<int> > const&)' collect2: error: ld returned 1 exit status modules/odm_orthophoto/CMakeFiles/odm_orthophoto.dir/build.make:453: recipe for target 'bin/odm_orthophoto' failed make[2]: *** [bin/odm_orthophoto] Error 1 CMakeFiles/Makefile2:280: recipe for target 'modules/odm_orthophoto/CMakeFiles/odm_orthophoto.dir/all' failed make[1]: *** [modules/odm_orthophoto/CMakeFiles/odm_orthophoto.dir/all] Error 2 [ 91%] Linking CXX executable ../../bin/odm_25dmeshing [ 91%] Built target odm_25dmeshing [ 95%] Linking CXX executable ../../bin/odm_georef CMakeFiles/odm_georef.dir/src/Georef.cpp.o: In function
Georef::performGeoreferencingWithGCP()':
Georef.cpp:(.text+0xaec2): undefined reference to `cv::imread(cv::String const&, int)'
collect2: error: ld returned 1 exit status
modules/odm_georef/CMakeFiles/odm_georef.dir/build.make:506: recipe for target 'bin/odm_georef' failed
make[2]: [bin/odm_georef] Error 1
CMakeFiles/Makefile2:170: recipe for target 'modules/odm_georef/CMakeFiles/odm_georef.dir/all' failed
make[1]: [modules/odm_georef/CMakeFiles/odm_georef.dir/all] Error 2
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2
It's either not finding OpenCV, or not finding the specific function imwrite.
In odm_orthophoto for example, cmake looks for OpenCV here: https://github.com/OpenDroneMap/OpenDroneMap/blob/master/modules/odm_orthophoto/CMakeLists.txt#L15
I'm at a conference right now or I'd be able to diagnose the issue further. Are you installing OpenCV inside the SuperBuild?
You may also want to look at: https://docs.opencv.org/3.3.0/d4/da8/group__imgcodecs.html#gabbc7ef1aa2edfaa87772f1202d67e0ce
Weird, pretty sure its building OpenCV without errors. Using docker build to build the image up to the point of failure at ../.. cmake .. && make. Will poke about after another fresh build.
Building a gist of my steps and the cuda-8.0-opencv-3.3.0 branch is pushed to my fork @https://github.com/GeoVisionEnvironmental/OpenDroneMap.git
Enjoy your conference, heading out for a trail run. Back at this later
Updated all the modules/xx/CMakeLists.txt PCL 1.8 --> 1.8.1
SuperBuild cmake and make builds without errors. /code/build cmake shows finds OpenCV & PCL. /code/build make -j20 errors still:
modules/odm_meshing/CMakeFiles/odm_meshing.dir/build.make:62: recipe for target 'modules/odm_meshing/CMakeFiles/odm_meshing.dir/src/OdmMeshing.cpp.o' failed
make[2]: [modules/odm_meshing/CMakeFiles/odm_meshing.dir/src/OdmMeshing.cpp.o] Error 1
make[2]: Waiting for unfinished jobs....
CMakeFiles/Makefile2:225: recipe for target 'modules/odm_meshing/CMakeFiles/odm_meshing.dir/all' failed
make[1]: [modules/odm_meshing/CMakeFiles/odm_meshing.dir/all] Error 2
make[1]: Waiting for unfinished jobs....
[ 87%] Linking CXX executable ../../bin/odm_orthophoto
CMakeFiles/odm_orthophoto.dir/src/OdmOrthoPhoto.cpp.o: In function OdmOrthoPhoto::createOrthoPhoto()': OdmOrthoPhoto.cpp:(.text+0x916c): undefined reference to
cv::imread(cv::String const&, int)'
OdmOrthoPhoto.cpp:(.text+0x96bb): undefined reference to cv::imwrite(cv::String const&, cv::_InputArray const&, std::vector<int, std::allocator<int> > const&)' collect2: error: ld returned 1 exit status modules/odm_orthophoto/CMakeFiles/odm_orthophoto.dir/build.make:453: recipe for target 'bin/odm_orthophoto' failed make[2]: *** [bin/odm_orthophoto] Error 1 CMakeFiles/Makefile2:280: recipe for target 'modules/odm_orthophoto/CMakeFiles/odm_orthophoto.dir/all' failed make[1]: *** [modules/odm_orthophoto/CMakeFiles/odm_orthophoto.dir/all] Error 2 [ 91%] Linking CXX executable ../../bin/odm_25dmeshing [ 91%] Built target odm_25dmeshing [ 95%] Linking CXX executable ../../bin/odm_georef CMakeFiles/odm_georef.dir/src/Georef.cpp.o: In function
Georef::performGeoreferencingWithGCP()':
Georef.cpp:(.text+0xaec2): undefined reference to `cv::imread(cv::String const&, int)'
collect2: error: ld returned 1 exit status
modules/odm_georef/CMakeFiles/odm_georef.dir/build.make:506: recipe for target 'bin/odm_georef' failed
make[2]: [bin/odm_georef] Error 1
CMakeFiles/Makefile2:170: recipe for target 'modules/odm_georef/CMakeFiles/odm_georef.dir/all' failed
make[1]: [modules/odm_georef/CMakeFiles/odm_georef.dir/all] Error 2
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2
Would this be caused by the #include <opencv2/imgproc/imgproc.hpp> in Georef.cpp -->L#6&7 ?
Update: Just posted a Gist with my build steps and usage: https://gist.github.com/PeterSprague/98cc0ae744e8285111978167e5626678
Pushed latest to: https://github.com/GeoVisionEnvironmental/OpenDroneMap.git --> branch gei-cuda-8.0-opencv-3.3.1
The branch to date is only an experimental build to be run under nvidia-docker. Using configure.sh will not produce a working install yet.
Builds w/o errors, but when run against a known set of images throws a python error:
$ nvidia-docker run -it --name odm-cuda-8.0-opencv-3.3.0 -v $(pwd)/images/images_jpg:/code/images:Z -v $(pwd)/images_resize:/code/images_resize:Z -v $(pwd)/opensfm:/code/opensfm:Z -v $(pwd)/odm_meshinging:/code/odm_meshing:Z -v $(pwd)/odm_texturing:/code/odm_texturing:Z -v $(pwd)/odm_georeferencing:/code/odm_georeferencing:Z -v $(pwd)/odm_orthophoto:/code/odm_orthophoto:Z -v $(pwd)/settings.yaml:/code/settings.yaml:Z --rm gei/opendronemap:cuda-8.0-opencv-3.3.0 --force-focal 3.97 --force-ccd 4.63104
Error:
~
2017-10-26 04:04:25,915 Found 18564 points in 11.1263718605s
2017-10-26 04:04:26,052 Found 18469 points in 11.6344058514s
2017-10-26 04:04:27,398 Extracting ROOT_HAHOG features for image 2017_0908_135243_192.JPG
2017-10-26 04:04:27,398 Extracting ROOT_HAHOG features for image 2017_0908_134607_010.JPG
2017-10-26 04:04:27,399 Extracting ROOT_HAHOG features for image 2017_0908_135410_234.JPG
Traceback (most recent call last):
File "/code/SuperBuild/src/opensfm/bin/opensfm", line 34, in
You're not the only one: https://github.com/mapillary/OpenSfM/issues/228
Interesting, will have to let sit for now, see what comes back for solutions.
ODM master hasn't updated its OpenSFM version recently?
I pulled in latest upstream master to then branch and build ODM-Cuda.
Ok try it now, we just updated OpenSfM.
From: PeterSprague notifications@github.com Sent: Thursday, October 26, 2017 2:43:32 PM To: OpenDroneMap/OpenDroneMap Cc: Dakota M. Benjamin; Comment Subject: Re: [OpenDroneMap/OpenDroneMap] OpenCV-3.3.1 w/ nvidia/cuda:8.0-devel-ubuntu16.04 image for GPU processing tests (#694)
I pulled in latest upstream master to then branch and build ODM-Cuda.
— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/OpenDroneMap/OpenDroneMap/issues/694#issuecomment-339761134, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABwyCgHjlKohA6QWguQ-GSys8iD8rL2bks5swNLTgaJpZM4QBRRb.
Merged #699 loky fix, built Docker image, ran on same set of images:
Got farther, then error with similar message.
Error:
~~
2017-10-26 19:53:36,154 Found 18284 points in 20.0877490044s
2017-10-26 19:53:37,112 Found 18673 points in 20.6305661201s
2017-10-26 19:53:37,822 Found 18316 points in 16.269990921s
2017-10-26 19:53:39,152 Found 18529 points in 31.7408931255s
Traceback (most recent call last):
File "/code/SuperBuild/src/opensfm/bin/opensfm", line 34, in
Side question: what is core2.Dockerfile for? Vs Dockerfile?
The branch to date is only an experimental build to be run under nvidia-docker. Using configure.sh will not produce a working install yet.
Is this required for ODM? I had installed for testing, but not for last build of ODM. Install @ /code/SuperBuild/src ? Or /code/SuperBuild/build ?
OpenSfM Install note:
When running OpenSfM on top of OpenCV 3.0 the OpenCV Contrib modules are required for extracting SIFT or SURF features.
No, we use HAHOG not surf or sift
Does this dataset work with standard ODM? Have you tried on a different dataset that is known to work othewise?
Yes, the dataset works with my September102017 ODM docker build. Why I choose it, but can try another to test if difference in errors.
Rerunning with sept102017 ODM docker build to test
Yeah I don't know what else to do at this point
Get Outlook for Androidhttps://aka.ms/ghei36
From: PeterSprague notifications@github.com Sent: Thursday, October 26, 2017 5:02:52 PM To: OpenDroneMap/OpenDroneMap Cc: Dakota M. Benjamin; Comment Subject: Re: [OpenDroneMap/OpenDroneMap] OpenCV-3.3.1 w/ nvidia/cuda:8.0-devel-ubuntu16.04 image for GPU processing tests (#694)
Yes, the dataset works with my September ODM docker build. Why I choose it, but can try another to test if difference in errors
— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/OpenDroneMap/OpenDroneMap/issues/694#issuecomment-339800097, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABwyCq2hNCStnsinOCPmvphor0XMRaKrks5swPN8gaJpZM4QBRRb.
I will pull and build a new docker image from the latest master. See if that runs.
Would someone be able to check the diffs? https://github.com/OpenDroneMap/OpenDroneMap/compare/master...GeoVisionEnvironmental:cuda-8.0-opencv-3.3.1
I can't see anything that should/could cause this. Someone with better eyes/more knowledge may be able to find a solution.
So odd that you get the same error with two different parallelization methods, and it's not a data issue.
The diff looks fine to me. Can you try running with --opensfm-processes 1
The two errors posted above are different. Last one refers to "loky", earlier one refers to "multiprocessing". Last run sustained much deeper into the process before erroring.
Ran with --opensfm=1:
2017-10-27 17:50:22,457 Extracting focal lengths for image 2017_0908_134743_054.JPG
2017-10-27 17:50:22,815 Extracting focal lengths for image 2017_0908_135448_254.JPG
[DEBUG] running PYTHONPATH=/code/SuperBuild/install/lib/python2.7/dist-packages /code/SuperBuild/src/opensfm/bin/opensfm detect_features /code/opensfm
2017-10-27 17:50:23,533 Extracting ROOT_HAHOG features for image 2017_0908_134747_056.JPG
2017-10-27 17:50:31,038 Found 18515 points in 7.05036902428s
Traceback (most recent call last):
File "/code/SuperBuild/src/opensfm/bin/opensfm", line 34, in
Different error than either above.
Could it be something with OpenCV-3.3.1 --> "flann_Index = cv2.flann.Index if context.OPENCV3 else cv2.flann_Index"
Yeah I think you're on to something, i'm proing over the cv2 docs right now...
Maybe flann wasn't installed correctly?
Would it make any sense to build a non-cuda version with opencv-3.3.1 and test?
Wonder if its something to do with the build directives for OpenCV-3.3.1?
gei-nocuda-opencv-3.3.1 borks. Only thing changed from master was to revise the OpenCV version in SuperBuild/cmake/Ex...txt and CMakeLists.txt file
Ran with and w/o --opensfm-processes 1 - gives same errors as above respectively
Just saw this: https://github.com/mapillary/OpenSfM/commit/b05ccbe75f74b16e1ad05eb1ae32f38188843a62
Can you pull latest opensfm and try again?
@PeterSprague did the above suggestion help?
Will try when I get back from Toronto. Thanks for the pointer
Run with OpenSfM master with #231 commit that fixes flann opencv issue:
Successful docker build, and nvidia-docker run with known set of images.
Pushed latest code to repo. Only run in nvidia-docker, install.sh has not been updated or tested. Likely a bunch of clean-up/consolidation of the dockerfile should take place.
Not sure if any of this is of use to the larger community. Its only a step along the path to get GPU integration for ODM.
OpenSfM is not utilizing the GPU yet though OpenCV is built against Cuda:
$ nvidia-smi
Tue Nov 7 09:50:17 2017
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 384.90 Driver Version: 384.90 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 1080 Off | 00000000:05:00.0 On | N/A |
| 27% 34C P8 8W / 180W | 451MiB / 8112MiB | 4% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | 0 1784 G /usr/libexec/Xorg 414MiB | | 0 2415 G /usr/lib64/firefox/firefox 3MiB | | 0 20753 G /proc/self/exe 29MiB | +-----------------------------------------------------------------------------+
I thought OpenSfM would pick up the GPU if OpenCV was built against Cuda? Since this is not directly an ODM issue, I will continue this GPU issue over at OpenSfM.
OpenCV-2.4.xx w/ Cuda does not recognize newer classes of NVidia GPU when building, aka only up to Kepler. I need Pascal, therefore need to build with OpenCV-3.3.0
Updated SuperBuild/CMakeLists.txt & Extenal-xxxx.cmake to newest versions: Ecto --> 0.6.12 OpenCV-2.4.11 --> 3.3.0 PCL 1.8.0 -->1.8.1
On host: $ docker build --tag gei/opendronemap:cuda-8.0-opencv-3.3.0 Builds successfully through [rest of Dockerfile is commented out]: $ cd SuperBuild/build && cmake .. && make -j20 no errors
$ docker run -it --name odm-cuda-8.0-opencv-3.3.0 gei/opendronemap:cuda-8.0-opencv-3.3.0 bash In container -> moving up to next Dockerfile step: $ cd ../.. && mkdir build && cd build && cmake .. a few warnings, but -- Configuring done -- Generating done -- Build files have been written to: /code/build
$ make -j20 throws errors at odm_texturing: [ 86%] Linking CXX executable ../../bin/odm_meshing [ 86%] Built target odm_meshing [ 89%] Linking CXX executable ../../bin/odm_texturing CMakeFiles/odm_texturing.dir/src/OdmTexturing.cpp.o: In function > const&)'
collect2: error: ld returned 1 exit status
modules/odm_texturing/CMakeFiles/odm_texturing.dir/build.make:479: recipe for target 'bin/odm_texturing' failed
make[2]: [bin/odm_texturing] Error 1
CMakeFiles/Makefile2:338: recipe for target 'modules/odm_texturing/CMakeFiles/odm_texturing.dir/all' failed
make[1]: [modules/odm_texturing/CMakeFiles/odm_texturing.dir/all] Error 2
make[1]: Waiting for unfinished jobs....
[ 93%] Linking CXX executable ../../bin/odm_orthophoto
CMakeFiles/odm_orthophoto.dir/src/OdmOrthoPhoto.cpp.o: In function > const&)'
collect2: error: ld returned 1 exit status
modules/odm_orthophoto/CMakeFiles/odm_orthophoto.dir/build.make:453: recipe for target 'bin/odm_orthophoto' failed
make[2]: [bin/odm_orthophoto] Error 1
CMakeFiles/Makefile2:283: recipe for target 'modules/odm_orthophoto/CMakeFiles/odm_orthophoto.dir/all' failed
make[1]: [modules/odm_orthophoto/CMakeFiles/odm_orthophoto.dir/all] Error 2
[ 96%] Linking CXX executable ../../bin/odm_25dmeshing
[ 96%] Built target odm_25dmeshing
[100%] Linking CXX executable ../../bin/odm_georef
CMakeFiles/odm_georef.dir/src/Georef.cpp.o: In function
OdmTexturing::loadCameras()': OdmTexturing.cpp:(.text+0x2a3b): undefined reference to
cv::imread(cv::String const&, int)' CMakeFiles/odm_texturing.dir/src/OdmTexturing.cpp.o: In functionOdmTexturing::createTextures()': OdmTexturing.cpp:(.text+0x84ef): undefined reference to
cv::imread(cv::String const&, int)' OdmTexturing.cpp:(.text+0x8a43): undefined reference tocv::imwrite(cv::String const&, cv::_InputArray const&, std::vector<int, std::allocator<int> > const&)' OdmTexturing.cpp:(.text+0x8d35): undefined reference to
cv::imwrite(cv::String const&, cv::_InputArray const&, std::vector<int, std::allocatorOdmOrthoPhoto::createOrthoPhoto()': OdmOrthoPhoto.cpp:(.text+0x916c): undefined reference to
cv::imread(cv::String const&, int)' OdmOrthoPhoto.cpp:(.text+0x96bb): undefined reference to `cv::imwrite(cv::String const&, cv::_InputArray const&, std::vector<int, std::allocatorGeoref::performGeoreferencingWithGCP()': Georef.cpp:(.text+0xaec2): undefined reference to
cv::imread(cv::String const&, int)' collect2: error: ld returned 1 exit status modules/odm_georef/CMakeFiles/odm_georef.dir/build.make:506: recipe for target 'bin/odm_georef' failed make[2]: [bin/odm_georef] Error 1 CMakeFiles/Makefile2:173: recipe for target 'modules/odm_georef/CMakeFiles/odm_georef.dir/all' failed make[1]: [modules/odm_georef/CMakeFiles/odm_georef.dir/all] Error 2 Makefile:83: recipe for target 'all' failed make: [all] Error 2Ideas? I'm thinking that possibly the ODM scripts calls need to be tuned for the newer versions of software?