Doesn't work with OpenCV 4

[auscompgeek]

(Filing this so people don't waste time trying to get this working with OpenCV 4. tl;dr: it doesn't work and it's hard; stick with OpenCV 3 for now.)

OpenCV 4 appears to install headers to /usr/include/opencv4 instead of /usr/include. It seems like the intended way to discover the correct include path is to call pkg-config: pkg-config --cflags opencv4. (One can trick distutils to pass through a -I by setting $CFLAGS, but obviously that's a terrible suggestion.)

However, once you convince distutils to tell gcc to look for the OpenCV headers in the right spot...

src/ndarray_converter.cpp: In member function ‘cv::UMatData* NumpyAllocator::allocate(int, const int*, int, void*, size_t*, int, cv::UMatUsageFlags) const’:
src/ndarray_converter.cpp:111:75: error: invalid conversion from ‘int’ to ‘cv::AccessFlag’ [-fpermissive]
             return stdAllocator->allocate(dims0, sizes, type, data, step, flags, usageFlags);
                                                                           ^~~~~
In file included from /usr/include/opencv4/opencv2/core.hpp:59,
                 from /usr/include/opencv4/opencv2/core/core.hpp:48,
                 from src/ndarray_converter.h:5,
                 from src/ndarray_converter.cpp:4:
/usr/include/opencv4/opencv2/core/mat.hpp:473:69: note:   initializing argument 6 of ‘virtual cv::UMatData* cv::MatAllocator::allocate(int, const int*, int, void*, size_t*, cv::AccessFlag, cv::UMatUsageFlags) const’
                                void* data, size_t* step, AccessFlag flags, UMatUsageFlags usageFlags) const = 0;
                                                          ~~~~~~~~~~~^~~~~
src/ndarray_converter.cpp: In member function ‘bool NumpyAllocator::allocate(cv::UMatData*, int, cv::UMatUsageFlags) const’:
src/ndarray_converter.cpp:136:42: error: invalid conversion from ‘int’ to ‘cv::AccessFlag’ [-fpermissive]
         return stdAllocator->allocate(u, accessFlags, usageFlags);
                                          ^~~~~~~~~~~
In file included from /usr/include/opencv4/opencv2/core.hpp:59,
                 from /usr/include/opencv4/opencv2/core/core.hpp:48,
                 from src/ndarray_converter.h:5,
                 from src/ndarray_converter.cpp:4:
/usr/include/opencv4/opencv2/core/mat.hpp:474:54: note:   initializing argument 2 of ‘virtual bool cv::MatAllocator::allocate(cv::UMatData*, cv::AccessFlag, cv::UMatUsageFlags) const’
     virtual bool allocate(UMatData* data, AccessFlag accessflags, UMatUsageFlags usageFlags) const = 0;
                                           ~~~~~~~~~~~^~~~~~~~~~~
src/ndarray_converter.cpp: At global scope:
src/ndarray_converter.cpp:157:16: error: cannot declare variable ‘g_numpyAllocator’ to be of abstract type ‘NumpyAllocator’
 NumpyAllocator g_numpyAllocator;
                ^~~~~~~~~~~~~~~~
src/ndarray_converter.cpp:86:7: note:   because the following virtual functions are pure within ‘NumpyAllocator’:
 class NumpyAllocator : public MatAllocator
       ^~~~~~~~~~~~~~
In file included from /usr/include/opencv4/opencv2/core.hpp:59,
                 from /usr/include/opencv4/opencv2/core/core.hpp:48,
                 from src/ndarray_converter.h:5,
                 from src/ndarray_converter.cpp:4:
/usr/include/opencv4/opencv2/core/mat.hpp:472:23: note:         ‘virtual cv::UMatData* cv::MatAllocator::allocate(int, const int*, int, void*, size_t*, cv::AccessFlag, cv::UMatUsageFlags) const’
     virtual UMatData* allocate(int dims, const int* sizes, int type,
                       ^~~~~~~~
/usr/include/opencv4/opencv2/core/mat.hpp:474:18: note:         ‘virtual bool cv::MatAllocator::allocate(cv::UMatData*, cv::AccessFlag, cv::UMatUsageFlags) const’
     virtual bool allocate(UMatData* data, AccessFlag accessflags, UMatUsageFlags usageFlags) const = 0;
                  ^~~~~~~~

[auscompgeek]

There's https://github.com/matze/pkgconfig to help with finding the correct include dir, and opencv2/core/version.hpp defines CV_VERSION_{MAJOR,MINOR,REVISION}, which should be useful. This might not be as difficult as I expected.

[auscompgeek]

Darn. cscore itself doesn't support OpenCV 4.

cscore_src/cscore/src/main/native/cpp/Frame.cpp: In member function ‘cs::Image* cs::Frame::ConvertBGRToMJPEG(cs::Image*, int)’:
cscore_src/cscore/src/main/native/cpp/Frame.cpp:400:41: error: ‘CV_IMWRITE_JPEG_QUALITY’ was not declared in this scope
     m_impl->compressionParams.push_back(CV_IMWRITE_JPEG_QUALITY);
                                         ^~~~~~~~~~~~~~~~~~~~~~~
cscore_src/cscore/src/main/native/cpp/Frame.cpp: In member function ‘cs::Image* cs::Frame::ConvertGrayToMJPEG(cs::Image*, int)’:
cscore_src/cscore/src/main/native/cpp/Frame.cpp:431:41: error: ‘CV_IMWRITE_JPEG_QUALITY’ was not declared in this scope
     m_impl->compressionParams.push_back(CV_IMWRITE_JPEG_QUALITY);
                                         ^~~~~~~~~~~~~~~~~~~~~~~

[uutzinger]

cscore in allwpilib supports now opencv4.

• How do I modify setup.py so that it finds opencv includes?

I build opencv and allwpilib from source. Then I link

ln -fs ~/allwpilib/wpiutil/ ~/robotpy-cscore/cscore_src/wpiutil
ln -fs ~/allwpilib/cscore/ ~/robotpy-cscore/cscore_src/cscore

and finally python3 setup.y build but #include <opencv2/opencv.hpp> in src/_cscore.cpp is not found.

[auscompgeek]

You'll have to query pkg-config for the OpenCV include dir. See above for a package to help with that.

[uutzinger]

Ok need to change setup.py with

def cpp_flag(compiler):
    """Return the -std=c++[11/14/17] compiler flag.

    The c++17 is preferred over c++11 and c++14 (when it is available).
    """
    if has_flag(compiler, "-std=c++17"):
        return "-std=c++17"
    elif has_flag(compiler, "-std=c++14"):
        return "-std=c++14"
    elif has_flag(compiler, "-std=c++11"):
        return "-std=c++11"
    else:
        raise RuntimeError(
            "Unsupported compiler -- at least C++11 support " "is needed!"
        )

and

        include_dirs=[
            "pybind11/include",
            "cscore_src/cscore/src/main/native/include",
            "cscore_src/cscore/src/main/native/cpp",
            "cscore_src/wpiutil/src/main/native/include",
            "cscore_src/wpiutil/src/main/native/libuv/include",
            "/usr/local/include/opencv4",
            get_numpy_include(),
        ],

note: libuv needs to be changed also, not just adding opencv4 include folder

and now we need to fix ndarray_converter.cpp and ndarray_converter.h for invalid conversion from ‘int’ to ‘cv::AccessFlag’

[uutzinger]

And if you want to work with pkgconfig for opencv you need to compile opencv with -D OPENCV_GENERATE_PKGCONFIG=ON so it creates the opencv4.pc, but with modified setup.py as above, that is not needed.

[uutzinger]

patch ndarray_converter.cpp with (int accessFlags to AccessFlag accessFlags)

105c105
<     UMatData* allocate(int dims0, const int* sizes, int type, void* data, size_t* step, int flags, UMatUsageFlags usageFlags) const
---
>     UMatData* allocate(int dims0, const int* sizes, int type, void* data, size_t* step, AccessFlag flags, UMatUsageFlags usageFlags) const
134c134
<     bool allocate(UMatData* u, int accessFlags, UMatUsageFlags usageFlags) const
---
>     bool allocate(UMatData* u, AccessFlag accessFlags, UMatUsageFlags usageFlags) const

[virtuald]

I'm surprised that's the only change to the ndarray_converter.cpp script, are you sure OpenCV 4 doesn't have more changes? After all, the bulk of that was extracted from the OpenCV source code.

Adding /usr/local/include/opencv4 is not a good idea. What if it's installed via a system package? Using pkg-config makes a lot more sense to me.

However, an open question for me is whether we should migrate this to robotpy-build assuming that we migrate everything to binary bindings? In that case, we could utilize pkg-config with robotpy-build directly, and could get rid of the cscore subtree. However, not sure whether the upstream WPILib builds would be able to support python bindings or not.

[auscompgeek]

However, not sure whether the upstream WPILib builds would be able to support python bindings or not.

The upstream builds would be built against the WPILib-packaged OpenCV, so it would definitely not work with the system-installed OpenCV and won't have Python bindings.

I've pushed up what I had in my working tree in my experiments compiling against OpenCV 4: add61ae1f5dcb4ece6e9f4259e30c6da41d372ba

[uutzinger]

robotpy-cscore is used by https://github.com/wpilibsuite/FRCVision-pi-gen for python bindings to create a raspberrypi image. I was trying to use it with opencv4.

allwpilib has made changes to accomodate OpenCV4 but I am not sure if they just disabled functions. cscore has only 3 files with OpenCV issues in allwpilib/cscore/src/main/native/cpp/

• CvSinkImpl.cpp
• CvSourceImpl.cpp
• Frame.cpp

When adding and changing the opencv related includes it compiles. The JPEG_QUALITY issue above can be fixed by using the opencv4 equivalent name. One can make current release version of allwpilib compile with opencv4.

I have pkgconfig for opencv4 and its in the pkgconfig path but setup.py does not automatically use it. I don't know how to make that work.

You are correct there are more issues to solve because latest version of allwpilib on github introduces allwpilib/wpiutil/src/main/native/cpp/jni/WPIUtilJNI.cpp which includes jni.h (found in jvm jdk) and "edu_wpi_first_wpiutil_WPIUtilJNI.h" which appears not being a part of the allwpilib. Perhaps it can be somehow exclude from the robotpy-cscore.

I am no expert with creating system packages and perhaps the title of this issue will remain a while longer in effect.

[auscompgeek]

robotpy-cscore is used by https://github.com/wpilibsuite/FRCVision-pi-gen for python bindings to create a raspberrypi image.

The frcvision image has all of WPILib, OpenCV, and robotpy-cscore built from source. It does not use any of WPILib's binary artifacts.

[auscompgeek]

I just realised we will want to make sure we can still build on Windows as well, lest we undo all the work I did last season.

[boranseckin]

Hey, I was trying to install cscore on my mac (with OpenCV 4) and finally, I made it thanks to the previous workarounds on this thread. I am sharing what I have done so anyone who needs it can benefit.

I have done all the changes on this commit add61ae. Plus I changed the JPEG_QUALITY and added new imports as told on this thread.

It is successfully compiled and installed.

Changes (additional to add61ae): cscore_src/cscore/src/main/native/cpp/CvSinkImpl.cpp

13,14d12
< #include <opencv2/core/types_c.h>
< #include <opencv2/videoio/videoio_c.h>

cscore_src/cscore/src/main/native/cpp/CvSourceImpl.cpp

13,14d12
< #include <opencv2/core/types_c.h>
< #include <opencv2/videoio/videoio_c.h>

cscore_src/cscore/src/main/native/cpp/Frame.cpp

400c400
<     m_impl->compressionParams.push_back(cv::IMWRITE_JPEG_QUALITY);
---
>     m_impl->compressionParams.push_back(CV_IMWRITE_JPEG_QUALITY);
431c431
<     m_impl->compressionParams.push_back(cv::IMWRITE_JPEG_QUALITY);
---
>     m_impl->compressionParams.push_back(CV_IMWRITE_JPEG_QUALITY);

[auscompgeek]

This was done in #81.