File not found for opencv video tests in `0.37.0`

TheChymera commented 2 years ago

I get the following build log for =dandi-cli-0.37.0: build.log.

Only the video tests fail, all with what appears to be the same sort of issue:

Caught exception /var/tmp/portage/dev-vcs/dandi-cli-0.37.0/temp/pytest-of-portage/pytest-0/test_video_organize_common_cop0/video_files/test1_0.avi does not exist

I couldn't find the test1_0.avi file anywhere in the build and test sandbox, which seems to be because it's part of a fixture. Any idea what could be wrong?

yarikoptic commented 2 years ago

produced by a fixture: git grep is your friend

$> git grep -p test1_
dandi/tests/fixtures.py=def video_files(tmp_path):
dandi/tests/fixtures.py:        movie_file1 = video_path / f"test1_{no}.avi"

do you have opencv in dependencies?

$> git grep opencv
setup.cfg:    opencv-python

since so far we haven't ran into such problems, I think we could keep it mandatory for testing instead of skipping a test if not available. I will close this with the above, but feel welcome to reopen if there is more to the story

TheChymera commented 2 years ago

@yarikoptic yes, I had found it in the fixtures, was just wondering whether anybody knew what could be wrong. I tried to run the respective function stand-alone, and indeed it does not seem to create any files. Ofc I also have opencv ;)

chymera@decohost ~ $ cat lulu.py
def video_files(tmp_path):
    video_paths = []
    import cv2
    import numpy as np
    import os

    video_path = f"{tmp_path}/video_files"
    os.mkdir(video_path)
    for no in range(2):
        movie_file1 = f"{video_path}/test1_{no}.avi"
        movie_file2 = f"{video_path}/test2_{no}.avi"
        (nf, nx, ny) = (5, 10, 20)
        writer1 = cv2.VideoWriter(
            filename=str(movie_file1),
            apiPreference=None,
            fourcc=cv2.VideoWriter_fourcc(*"DIVX"),
            fps=25,
            frameSize=(ny, nx),
            params=None,
        )
        writer2 = cv2.VideoWriter(
            filename=str(movie_file2),
            apiPreference=None,
            fourcc=cv2.VideoWriter_fourcc(*"DIVX"),
            fps=25,
            frameSize=(ny, nx),
            params=None,
        )
        for k in range(nf):
            writer1.write(np.random.randint(0, 255, (nx, ny, 3)).astype("uint8"))
            writer2.write(np.random.randint(0, 255, (nx, ny, 3)).astype("uint8"))
        writer1.release()
        writer2.release()
        video_paths.append((movie_file1, movie_file2))
    return video_paths
chymera@decohost ~ $ python -c "import lulu; lulu.video_files('/tmp')"
[ INFO:0@0.004] global /var/tmp/portage/media-libs/opencv-4.5.5-r1/work/opencv-4.5.5/modules/videoio/src/videoio_registry.cpp (223) VideoBackendRegistry VIDEOIO: Enabled backends(7, sorted by priority): FFMPEG(1000); GSTREAMER(990); INTEL_MFX(980); V4L2(970); CV_IMAGES(960); CV_MJPEG(950); UEYE(940)
[ INFO:0@0.004] global /var/tmp/portage/media-libs/opencv-4.5.5-r1/work/opencv-4.5.5/modules/videoio/src/backend_plugin.cpp (369) getPluginCandidates VideoIO plugin (FFMPEG): glob is 'libopencv_videoio_ffmpeg*.so', 1 location(s)
[ INFO:0@0.009] global /var/tmp/portage/media-libs/opencv-4.5.5-r1/work/opencv-4.5.5/modules/videoio/src/backend_plugin.cpp (379) getPluginCandidates     - /usr/lib64: 0
[ INFO:0@0.009] global /var/tmp/portage/media-libs/opencv-4.5.5-r1/work/opencv-4.5.5/modules/videoio/src/backend_plugin.cpp (383) getPluginCandidates Found 0 plugin(s) for FFMPEG
[ INFO:0@0.010] global /var/tmp/portage/media-libs/opencv-4.5.5-r1/work/opencv-4.5.5/modules/videoio/src/backend_plugin.cpp (369) getPluginCandidates VideoIO plugin (GSTREAMER): glob is 'libopencv_videoio_gstreamer*.so', 1 location(s)
[ INFO:0@0.014] global /var/tmp/portage/media-libs/opencv-4.5.5-r1/work/opencv-4.5.5/modules/videoio/src/backend_plugin.cpp (379) getPluginCandidates     - /usr/lib64: 0
[ INFO:0@0.014] global /var/tmp/portage/media-libs/opencv-4.5.5-r1/work/opencv-4.5.5/modules/videoio/src/backend_plugin.cpp (383) getPluginCandidates Found 0 plugin(s) for GSTREAMER
[ INFO:0@0.014] global /var/tmp/portage/media-libs/opencv-4.5.5-r1/work/opencv-4.5.5/modules/videoio/src/backend_plugin.cpp (369) getPluginCandidates VideoIO plugin (INTEL_MFX): glob is 'libopencv_videoio_intel_mfx*.so', 1 location(s)
[ INFO:0@0.018] global /var/tmp/portage/media-libs/opencv-4.5.5-r1/work/opencv-4.5.5/modules/videoio/src/backend_plugin.cpp (379) getPluginCandidates     - /usr/lib64: 0
[ INFO:0@0.018] global /var/tmp/portage/media-libs/opencv-4.5.5-r1/work/opencv-4.5.5/modules/videoio/src/backend_plugin.cpp (383) getPluginCandidates Found 0 plugin(s) for INTEL_MFX
[ INFO:0@0.018] global /var/tmp/portage/media-libs/opencv-4.5.5-r1/work/opencv-4.5.5/modules/videoio/src/cap_images.cpp (282) icvExtractPattern Pattern: /tmp/video_files/test%01d_0.avi @ 1
[ INFO:0@0.018] global /var/tmp/portage/media-libs/opencv-4.5.5-r1/work/opencv-4.5.5/modules/videoio/src/cap_images.cpp (282) icvExtractPattern Pattern: /tmp/video_files/test%01d_0.avi @ 2
[ INFO:0@0.018] global /var/tmp/portage/media-libs/opencv-4.5.5-r1/work/opencv-4.5.5/modules/videoio/src/cap_images.cpp (282) icvExtractPattern Pattern: /tmp/video_files/test%01d_1.avi @ 1
[ INFO:0@0.018] global /var/tmp/portage/media-libs/opencv-4.5.5-r1/work/opencv-4.5.5/modules/videoio/src/cap_images.cpp (282) icvExtractPattern Pattern: /tmp/video_files/test%01d_1.avi @ 2
chymera@decohost ~ $ ls /tmp/video_files/

yarikoptic commented 2 years ago

thanks for reproducer.

hm, I don't see INFO outputs you see and it does produce files for me

```shell lena:/tmp $> xclip -o def video_files(tmp_path): video_paths = [] import cv2 import numpy as np import os video_path = f"{tmp_path}/video_files" os.mkdir(video_path) for no in range(2): movie_file1 = f"{video_path}/test1_{no}.avi" movie_file2 = f"{video_path}/test2_{no}.avi" (nf, nx, ny) = (5, 10, 20) writer1 = cv2.VideoWriter( filename=str(movie_file1), apiPreference=None, fourcc=cv2.VideoWriter_fourcc(*"DIVX"), fps=25, frameSize=(ny, nx), params=None, ) writer2 = cv2.VideoWriter( filename=str(movie_file2), apiPreference=None, fourcc=cv2.VideoWriter_fourcc(*"DIVX"), fps=25, frameSize=(ny, nx), params=None, ) for k in range(nf): writer1.write(np.random.randint(0, 255, (nx, ny, 3)).astype("uint8")) writer2.write(np.random.randint(0, 255, (nx, ny, 3)).astype("uint8")) writer1.release() writer2.release() video_paths.append((movie_file1, movie_file2)) return video_paths% 1 15671.....................................:Wed 30 Mar 2022 01:19:53 PM EDT:. lena:/tmp $> xclip -o > lulu.py 1 15672.....................................:Wed 30 Mar 2022 01:19:56 PM EDT:. lena:/tmp $> python3 -c "import lulu; print(lulu.video_files('/tmp'))" [('/tmp/video_files/test1_0.avi', '/tmp/video_files/test2_0.avi'), ('/tmp/video_files/test1_1.avi', '/tmp/video_files/test2_1.avi')] 1 15673.....................................:Wed 30 Mar 2022 01:20:15 PM EDT:. lena:/tmp $> ls video_files test1_0.avi test1_1.avi test2_0.avi test2_1.avi ```

this is with cv2 as shipped in debian. So may be issue version/OS dependent? may be you are lacking any codecs or smth needed to produce those files?

```shell $> welp cv2 PATH : /usr/lib/python3/dist-packages/cv2.cpython-39-x86_64-linux-gnu.so SRC PATH : /usr/lib/python3/dist-packages/cv2.cpython-39-x86_64-linux-gnu.so VERSION : Not found __version__: '4.5.4' PACKAGE : python3-opencv ii python3-opencv:amd64 4.5.4+dfsg-9+b3 amd64 Python 3 bindings for the computer vision library python3-opencv: Installed: 4.5.4+dfsg-9+b3 Candidate: 4.5.4+dfsg-9+b4 Version table: 4.5.4+dfsg-9+b4 900 900 http://deb.debian.org/debian bookworm/main amd64 Packages 600 http://http.debian.net/debian sid/main amd64 Packages *** 4.5.4+dfsg-9+b3 100 100 /var/lib/dpkg/status 4.5.1+dfsg-5 100 100 http://deb.debian.org/debian bullseye/main amd64 Packages ```

please check also may be there is some .close needed or alike to finish it all up cleanly?

TheChymera commented 2 years ago

@yarikoptic these are the optional features for opencv, some of which have pretty deep dependency stacks, so not that easy to test by process of elimination. Any idea which one it could be? Alternatively, would it be possible to just pre-ship the files, might be a bit less resource-intensive than requiring opencv for tests (even the minimal version I now have, has over 50MB of files).

[ Legend : U - final flag setting for installation]
[        : I - package is installed with flag     ]
[ Colors : set, unset                             ]
 * Found these USE flags for media-libs/opencv-4.5.5-r1:
 U I
 - - abi_x86_32                : 32-bit (x86) libraries
 - - contrib                   : Install user contributed scripts from opencv_contrib (Experimental!)
 - - contribcvv                : (Restricted to >=media-libs/opencv-3.1.0)
                                 CVV module requires Qt5
 - - contribdnn                : (Restricted to >=media-libs/opencv-3.3.0)
                                 DNN module contrib requires tiny dnn
 - - contribfreetype           : (Restricted to >=media-libs/opencv-4.4.0)
                                 Enable Drawing UTF-8 strings with freetype/harfbuzz
 - - contribhdf                : (Restricted to >=media-libs/opencv-3.1.0)
                                 HDF module requires sci-libs/hdf5
 - - contribsfm                : (Restricted to >=media-libs/opencv-3.1.0)
                                 SFM module requires eigen, gflags, and glog
 - - contribxfeatures2d        : (Restricted to >=media-libs/opencv-3.1.0)
                                 Enables xfeatures2d and autodownload of samples in contrib
 - - cpu_flags_x86_avx         : Adds support for Advanced Vector Extensions instructions
 - - cpu_flags_x86_avx2        : Adds support for Advanced Vector Extensions 2 instructions
 - - cpu_flags_x86_avx512f     : Adds support for AVX-512 Foundation instructions
 - - cpu_flags_x86_f16c        : Adds support for F16C instruction set for converting between half-precision and single-precision floats
 - - cpu_flags_x86_fma3        : Use the Fused Multiply Add 3 instruction set ([fma] in cpuinfo)
 - - cpu_flags_x86_popcnt      : Enable popcnt instruction support ([abm] or [popcnt] in cpuinfo)
 + + cpu_flags_x86_sse         : Use the SSE instruction set
 + + cpu_flags_x86_sse2        : Use the SSE2 instruction set
 - - cpu_flags_x86_sse3        : Use the SSE3 instruction set ([pni] in cpuinfo, NOT ssse3)
 - - cpu_flags_x86_sse4_1      : Enable SSE4.1 instruction support
 - - cpu_flags_x86_sse4_2      : Enable SSE4.2 instruction support
 - - cpu_flags_x86_ssse3       : Use the SSSE3 instruction set (NOT sse3/pni)
 - - cuda                      : Enable NVIDIA Cuda computations support (Experimental!)
 - - debug                     : Enable extra debug codepaths, like asserts and extra output. If you want to get meaningful backtraces see
                                 https://wiki.gentoo.org/wiki/Project:Quality_Assurance/Backtraces
 - - dnnsamples                : (Restricted to >=media-libs/opencv-3.4.0)
                                 Download dnn caffeemodel samples
 - - download                  : (Restricted to >=media-libs/opencv-4.1.2)
                                 Enable download during cmake configure
 + + eigen                     : Enable usage of dev-cpp/eigen for computations
 - - examples                  : Install examples, usually source code
 + + features2d                : Enable features2d module
 - - ffmpeg                    : Enable ffmpeg/libav-based audio/video codec support
 - - gdal                      : Enable support for sci-libs/gdal library
 - - gflags                    : (Restricted to >=media-libs/opencv-3.1.0)
                                 Use Google's C++ argument parsing library
 - - glog                      : (Restricted to >=media-libs/opencv-3.1.0)
                                 Use Google's C++ loggin library
 - - gphoto2                   : Add digital camera support
 - - gstreamer                 : Add support for media-libs/gstreamer (Streaming media)
 - - gtk3                      : Enable x11-libs/gtk+:3 support
 - - ieee1394                  : Enable FireWire/iLink IEEE1394 support (dv, camera, ...)
 - - java                      : Add support for Java
 + + jpeg                      : Add JPEG image support
 - - jpeg2k                    : Support for JPEG 2000, a wavelet-based image compression format
 - - lapack                    : Add support for the virtual/lapack numerical library
 - - lto                       : (Restricted to >=media-libs/opencv-4.4.0)
                                 Build using Link Time Optimizations (LTO)
 - - opencl                    : Add support for OpenCL
 - - opencvapps                : (Restricted to >=media-libs/opencv-4.1.2)
                                 Enable compilation with opencvapps
 - - openexr                   : Support for the OpenEXR graphics file format
 + - opengl                    : Add support for OpenGL (3D graphics)
 + + openmp                    : Build support for the OpenMP (support parallel computing), requires >=sys-devel/gcc-4.2 built with
                                 USE="openmp"
 + + png                       : Add support for libpng (PNG images)
 + + python                    : Add optional support/bindings for the Python language
 - - python_targets_python3_10 : Build with Python 3.10
 - - python_targets_python3_8  : Build with Python 3.8
 + + python_targets_python3_9  : Build with Python 3.9
 - - qt5                       : Add support for the Qt 5 application and UI framework
 - - tesseract                 : (Restricted to >=media-libs/opencv-3.1.0)
                                 Use Google's OCR Engine
 - - testprograms              : Build and install programs for testing OpenCV (performance)
 - - threads                   : Add threads support for various packages. Usually pthreads
 + + tiff                      : Add support for the TIFF image format
 + + v4l                       : Enable support for video4linux (using linux-headers or userspace libv4l libraries)
 - - vaapi                     : Enable Video Acceleration API for hardware decoding
 - - vtk                       : Build new 3D visualization module viz based on sci-libs/vtk
 - - webp                      : Add support for the WebP image format
 - - xine                      : Add support for the XINE movie libraries

yarikoptic commented 2 years ago

my bet would be on

 - - ffmpeg                    : Enable ffmpeg/libav-based audio/video codec support

as seems to be the only one really for video files. But please also check (ask opencv ppl?) if there is a way to improve the script so we get alerted whenever writing of the file cannot happen for some reason, as seems to be the case in your case. It is odd that it just continues silently

TheChymera commented 2 years ago

I reported the issue upstream, let's see if anything comes back: https://github.com/opencv/opencv/issues/21824

Also:

do we need it to use ffmpeg or would it be possible to generate a more minimal file?
is there a reason to re-generate the file as part of a fixture, couldn't we just have a few example files either under dandi/tests/data or as part of a testdata archive? I looked and it's 4 files each being 8KB, making it comparable to what we have in dandi/tests/metadata, so even the first option should be feasible. Maybe we could also make them smaller, as in (nf, nx, ny) = (3, 2, 4)?

yarikoptic commented 2 years ago

note -- files under dandi/tests/data are currently text files, so their content and diffs are very nicely compressed by git. For binary files such videos that wouldn't be the case. that is why we resorted to fixtures for sample .nwb and other binary files (although IIRC we had some committed directly in the stone age past).

If content of the video files does not really matter (it doesn't right @saksham20) -- indeed please check how small you could get them. But may be also please check if there is a way to sense cv2 to have ffmpeg support built in and if not -- skip . I just think that in the long run it might be beneficial to rely on generated test files here instead of on a handful of pre-generated

yarikoptic commented 2 years ago

re the idea from matrix chat on "test dataset" -- we have one already (https://gui.dandiarchive.org/#/dandiset/000027/draft) so in principle we could upload to that one! then it would be synced into https://github.com/dandisets/000027 so could even be datalad installed if so desired . I guess some sample data could go there, but then remember that it would 1. add delay due to network communications, 2. forbid offline testing (rare - but happens, e.g. while building debian packages etc).

yarikoptic commented 1 year ago

@TheChymera is there still anything to be done for this issue?

TheChymera commented 1 year ago

Nope, we can close it.

dandi / dandi-cli

File not found for opencv video tests in `0.37.0` #944