Opencv 3.0.0 module and merge contrib

tatarsky commented 8 years ago

Per Git #368 there is a desire for the Opencv 3.0.0 tree that was built to support the need by Caffe for a newer version to also include the contrib code.

This will be done more likely next week but noting it as separate. The module will also be reviewed for any other needed environment.

tatarsky commented 8 years ago

Reminder of tree. And to match version of source release. https://github.com/itseez/opencv_contrib

aday00 commented 8 years ago

Thanks!

tatarsky commented 8 years ago

This hasn't been done yet. Still busy with other things.

tatarsky commented 8 years ago

So before I look at this I need to decide the level of impact to caffe modules as opencv as it stands now is used by said rather complex module.

I basically no longer wish to touch the caffe software as I believe the main consumers of it have very precise needs and I don't want working on this to force another round of builds/configs for it.

So basically does this contrib addition impact the generic opencv libraries? Or should it be a separate module?

aday00 commented 8 years ago

For what it's worth, I think the opencv contrib stuff is separate only for patent reasons (namely, SIFT is patented). The opencv and opencv contrib stuff is actually one thing, so no separate module, in my mind. The contrib build just places more binaries in the opencv build dir, as far as i know.

Here's what i did with opencv 3.1, though similar works for opencv 3.0 (just be sure to use opencv contrib 3.0 with opencv 3.0, or use opencv contrib 3.1 with opencv 3.1):

username@computer:~/bin/opencv$ wget https://github.com/Itseez/opencv/archive/3.1.0.zip

username@computer:~/bin/opencv/opencv-3.1.0/install$ cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=pwd -D WITH_TBB=ON -D WITH_V4L=ON -D WITH_QT=ON -D WITH_OPENGL=ON -DOPENCV_EXTRA_MODULES_PATH=/home/username/bin/opencv_contrib/opencv_contrib-3.1.0/modules ..

…

-- Performing Test HAVE_CXX_WNO_DEPRECATED_DECLARATIONS

-- Performing Test HAVE_CXX_WNO_DEPRECATED_DECLARATIONS - Success

-- General configuration for OpenCV 3.1.0 =====================================

-- Version control: unknown

-- Platform: -- Host: Linux 3.16.0-4-amd64 x86_64 -- CMake: 3.4.1 -- CMake generator: Unix Makefiles -- CMake build tool: /usr/bin/make

-- Configuration: RELEASE

-- C/C++: -- Built as dynamic libs?: YES -- C++ Compiler: /usr/bin/c++ (ver 5.2.1) -- C++ flags (Release): -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wno-narrowing -Wno-delete-non-virtual-dtor -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -msse -msse2 -mno-avx -msse3 -mno-ssse3 -mno-sse4.1 -mno-sse4.2 -ffunction-sections -fvisibility=hidden -fvisibility-inlines-hidden -O3 -DNDEBUG -DNDEBUG -- C++ flags (Debug): -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wno-narrowing -Wno-delete-non-virtual-dtor -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -msse -msse2 -mno-avx -msse3 -mno-ssse3 -mno-sse4.1 -mno-sse4.2 -ffunction-sections -fvisibility=hidden -fvisibility-inlines-hidden -g -O0 -DDEBUG -D_DEBUG -- C Compiler: /usr/bin/cc -- C flags (Release): -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wno-narrowing -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -msse -msse2 -mno-avx -msse3 -mno-ssse3 -mno-sse4.1 -mno-sse4.2 -ffunction-sections -fvisibility=hidden -O3 -DNDEBUG -DNDEBUG -- C flags (Debug): -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wno-narrowing -fdiagnostics-show-option -Wno-long-long -pthread -fomit-frame-pointer -msse -msse2 -mno-avx -msse3 -mno-ssse3 -mno-sse4.1 -mno-sse4.2 -ffunction-sections -fvisibility=hidden -g -O0 -DDEBUG -D_DEBUG -- Linker flags (Release): -- Linker flags (Debug): -- Precompiled headers: YES -- Extra dependencies: Qt5::Core Qt5::Gui Qt5::Widgets Qt5::Test Qt5::Concurrent Qt5::OpenGL /usr/lib/x86_64-linux-gnu/libpng.so /usr/lib/x86_64-linux-gnu/libz.so /usr/lib/x86_64-linux-gnu/libtiff.so /usr/lib/x86_64-linux-gnu/libjasper.so /usr/lib/x86_64-linux-gnu/libjpeg.so /usr/lib/x86_64-linux-gnu/libImath.so /usr/lib/x86_64-linux-gnu/libIlmImf.so /usr/lib/x86_64-linux-gnu/libIex.so /usr/lib/x86_64-linux-gnu/libHalf.so /usr/lib/x86_64-linux-gnu/libIlmThread.so dc1394 avcodec-ffmpeg avformat-ffmpeg avutil-ffmpeg swscale-ffmpeg avresample-ffmpeg /usr/lib/x86_64-linux-gnu/libbz2.so dl m pthread rt /usr/lib/x86_64-linux-gnu/libGLU.so /usr/lib/x86_64-linux-gnu/libGL.so

-- 3rdparty dependencies: libwebp

-- OpenCV modules: -- To be built: core flann imgproc ml photo video imgcodecs shape videoio highgui objdetect superres ts features2d calib3d java videostab xfeatures2d stitching python2 -- Disabled: world -- Disabled by dependency: datasets optflow tracking

-- Unavailable: cudaarithm cudabgsegm cudacodec cudafeatures2d cudafilters cudaimgproc cudalegacy cudaobjdetect cudaoptflow cudastereo cudawarping cudev python3 viz

-- GUI: -- QT 5.x: YES (ver 5.4.2) -- QT OpenGL support: YES (Qt5::OpenGL 5.4.2) -- OpenGL support: YES (/usr/lib/x86_64-linux-gnu/libGLU.so /usr/lib/x86_64-linux-gnu/libGL.so)

-- VTK support: NO

-- Media I/O: -- ZLib: /usr/lib/x86_64-linux-gnu/libz.so (ver 1.2.8) -- JPEG: /usr/lib/x86_64-linux-gnu/libjpeg.so (ver ) -- WEBP: build (ver 0.3.1) -- PNG: /usr/lib/x86_64-linux-gnu/libpng.so (ver 1.2.50) -- TIFF: /usr/lib/x86_64-linux-gnu/libtiff.so (ver 42 - 4.0.6) -- JPEG 2000: /usr/lib/x86_64-linux-gnu/libjasper.so (ver 1.900.1) -- OpenEXR: /usr/lib/x86_64-linux-gnu/libImath.so /usr/lib/x86_64-linux-gnu/libIlmImf.so /usr/lib/x86_64-linux-gnu/libIex.so /usr/lib/x86_64-linux-gnu/libHalf.so /usr/lib/x86_64-linux-gnu/libIlmThread.so (ver 1.6.1)

-- GDAL: NO

-- Video I/O: -- DC1394 1.x: NO -- DC1394 2.x: YES (ver 2.2.3) -- FFMPEG: YES -- codec: YES (ver 56.60.100) -- format: YES (ver 56.40.101) -- util: YES (ver 54.31.100) -- swscale: YES (ver 3.1.101) -- resample: YES (ver 2.1.0) -- gentoo-style: YES -- GStreamer: NO -- OpenNI: NO -- OpenNI PrimeSensor Modules: NO -- OpenNI2: NO -- PvAPI: NO -- GigEVisionSDK: NO -- UniCap: NO -- UniCap ucil: NO -- V4L/V4L2: NO/YES -- XIMEA: NO -- Xine: NO

-- gPhoto2: NO

-- Parallel framework: pthreads

-- Other third-party libraries: -- Use IPP: 9.0.1 [9.0.1] -- at: /home/username/bin/opencv/opencv-3.1.0/3rdparty/ippicv/unpack/ippicv_lnx -- Use IPP Async: NO -- Use VA: NO -- Use Intel VA-API/OpenCL: NO -- Use Eigen: YES (ver 3.2.5) -- Use Cuda: NO -- Use OpenCL: YES

-- Use custom HAL: NO

-- OpenCL: -- Version: dynamic -- Include path: /home/username/bin/opencv/opencv-3.1.0/3rdparty/include/opencl/1.2 -- Use AMDFFT: NO

-- Use AMDBLAS: NO

-- Python 2: -- Interpreter: /usr/bin/python2.7 (ver 2.7.10) -- Libraries: /usr/lib/x86_64-linux-gnu/libpython2.7.so (ver 2.7.10rc1) -- numpy: /usr/lib/python2.7/dist-packages/numpy/core/include (ver 1.8.2)

-- packages path: lib/python2.7/dist-packages

-- Python 3:

-- Interpreter: /usr/bin/python3.4 (ver 3.4.3)

-- Python (for build): /usr/bin/python2.7

-- Java: -- ant: /usr/bin/ant (ver 1.9.4) -- JNI: /usr/lib/jvm/default-java/include /usr/lib/jvm/default-java/include /usr/lib/jvm/default-java/include -- Java wrappers: YES

-- Java tests: YES

-- Matlab: Matlab not found or implicitly disabled

-- Documentation: -- Doxygen: NO

-- PlantUML: NO

-- Tests and samples: -- Tests: YES -- Performance tests: YES

-- C/C++ Examples: NO

-- Install path: /home/username/bin/opencv/opencv-3.1.0/install

-- cvconfig.h is in: /home/username/bin/opencv/opencv-3.1.0/install

-- Configuring done -- Generating done -- Build files have been written to: /home/username/bin/opencv/opencv-3.1.0/install username@computer:~/bin/opencv/opencv-3.1.0/install$

username@computer:~/bin/opencv/opencv-3.1.0/install$ make ... [100%] Built target opencv_createsamples Scanning dependencies of target opencv_annotation [100%] Building CXX object apps/annotation/CMakeFiles/opencv_annotation.dir/opencv_annotation.cpp.o [100%] Linking CXX executable ../../bin/opencv_annotation [100%] Built target opencv_annotation username@computer:~/bin/opencv/opencv-3.1.0/install$

aday00 commented 8 years ago

Wow, apologies for the output being valid markup, above.

aday00 commented 8 years ago

A few links I bumped in to during the course of the contrib build, which I'm sure you're aware: http://docs.opencv.org/master/db/dfa/tutorial_transition_guide.html#gsc.tab=0 http://rodrigoberriel.com/2014/10/installing-opencv-3-0-0-on-ubuntu-14-04/ http://stackoverflow.com/questions/30155073/error-while-building-opencv-contrib-opencv-3-0

tatarsky commented 8 years ago

So if this is added to the current OpenCV module what level of notification of the Caffe user base is required? Are they all monitoring Git?

tatarsky commented 8 years ago

Alternately I can build a 3.1 module for opencv for you to test separately. But Caffe modules aren't linked to it then. I don't know if you care as you seem to have your own version of it.

aday00 commented 8 years ago

3.0's good so everything's linked. My 3.1 install is on a local machine, not integrated with Caffe, mostly just testing on my own.

Notifying the people participating in #368 is probably more than good enough. I'm not sure if I'm the only opencv user, if gunpowdergreentea uses opencv also, or who else does, sorry!

I'm not empowered to set policy or make wide-ranging decisions (though I think John Chodera can), but I'd propose advertising the opencv contrib change in /etc/motd and/or the opencv module itself via Tcl puts statements http://modules.sourceforge.net/man/modulefile.html as you see fit.

tatarsky commented 8 years ago

@gunpowdergreentea do you have any concerns about this item being added to OpenCV (which caffe is then linked against)?

@juanperin BTW sets my priorities and time allocation. I am a contractor. If he sees this and has an issue with that time spent he can comment as well.

I will prep the build of the 3.0.0 OpenCV contrib items tomorrow and see if any comments are made here as to concerns. I will backup the current opencv module so that if adding these items causes issues we can roll back quickly. Caffe I feel is fragile and I am time constrained right now to deal with it breaking.

tatarsky commented 8 years ago

Also, we don't have QT 5. If you need that deployed it will require more time. So advise on the importance of that QT support flag.

aday00 commented 8 years ago

It might be best to follow the same protocol as we did in #368: if something's of general utility, we'll have a shared install etc... otherwise we're left to our own home/local directories.

I'd like to wait until my PI Thomas comes back from Europe in a week or few before moving forward with discussion etc.

I suspect opencv contrib may be useful only to me (or Fuchs lab), in which case it would be closed as "not a bug", "good luck", etc. That's fine too for now.

tatarsky commented 8 years ago

Lets just mark it held. We are prepping for the expansion of the cluster and once done with that I'll have more cycles.

aday00 commented 8 years ago

Keep up the good work, and thanks for being so responsive! :)

tatarsky commented 8 years ago

I'm going to close this for now in favor of the migration to FogBugz. If you still need this please just open a case there for proper routing to the software maintainers!

jchodera commented 8 years ago

Where?

jchodera commented 8 years ago

What is FogBugz?

juanperin commented 8 years ago

just email hpc-request@cbio etc... Fogbugz handles distribution and tracking. That is for 'support'. Community based chatter and questions are a little different.

jchodera commented 8 years ago

I may just be a user now, but if the support mechanism is changing, I imagine we should

Update the hal documentation
Update the login message with information on how to get support, since it currently says

mski1776:latex choderaj$ cbio
Last login: Thu May  5 10:42:33 2016 from 140.163.53.245

--------------------------------------------------------------------------

Welcome to the HAL cluster now located in the MSKCC Datacenter!

Please report any issues to:
        https://github.com/cBio/cbio-cluster/issues

--------------------------------------------------------------------------

Let people know the support mechanism is changing by email

I, for one, am surprised by this.

juanperin commented 8 years ago

Thanks for reminding us about those. We will certainly update motd right away and the documentation.

The transition is a gradual one as we also decide what to do with the more ‘social’ component that git provides. To start we’ve created a slack channel where we’re inviting all cluster users. You’ll see an invitation to that soon. I’ve asked Paul to redirect most requests to hpc-request so that our whole team can have insight and ability to support.

On May 5, 2016, at 3:46 PM, John Chodera notifications@github.com wrote:

I may just be a user now, but if the support mechanism is changing, I imagine we should

Update the hal documentation https://github.com/cBio/cbio-cluster/wiki/MSKCC-cBio-Cluster-User-Guide#getting-help Update the login message with information on how to get support, since it currently says mski1776:latex choderaj$ cbio Last login: Thu May 5 10:42:33 2016 from 140.163.53.245

Welcome to the HAL cluster now located in the MSKCC Datacenter!

Please report any issues to: https://github.com/cBio/cbio-cluster/issues

Let people know the support mechanism is changing by email I, for one, am surprised at this.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/cBio/cbio-cluster/issues/373#issuecomment-217256903

jchodera commented 8 years ago

The transition is a gradual one as we also decide what to do with the more ‘social’ component that git provides.

I'm not sure what you mean by "social", but the ability for other users to assist in providing rapid responses and the ability to view the issues solved previously are both benefits of the current system.

To start we’ve created a slack channel where we’re inviting all cluster users. You’ll see an invitation to that soon.

I'm certainly willing to give that a try!

One other thing I noticed: You seem to have cloned part of the wiki docs at hpc.mskcc.org. Is that synced with the official cluster documentation, which we have been continuing to update? Is the hpc.mskcc.org the new official docs? If so, how do we update that, and shouldn't we deprecate the wiki docs?

I definitely understand the desire for gradual change, but there is also something to be said for:

identifying what is wrong with the old way of doing things and the requirements the old way needed to solve
deciding on the new way to do things that fixes those issues while also satisfying the original requirements
warning people thoroughly in advance
putting the new way of doing things into practice in a coordinated manner so as not to cause undue confusion

It could just be me, but it feels like there are a lot of changes going on where even heavy cluster users have absolutely no idea what's going on...

cBio / cbio-cluster