search

saimn / sigal

yet another simple static gallery generator
http://sigal.saimon.org/
MIT License
893 stars 170 forks source link

2.3: Issues with tests #489

Closed dvzrv closed 1 year ago

dvzrv commented 1 year ago

Hi! I'm currently rebuilding sigal due to Python 3.11 upgrade on Arch Linux. I noticed that (also with Python 3.10) I have the following tests failing:

=================================== FAILURES ===================================                                                                                                                                                                                                                                     [172/4274]
_________________________ test_gallery_max_img_pixels __________________________

settings = {'adjust_options': {'brightness': 1.0, 'color': 0.9, 'contrast': 1.0, 'sharpness': 0.0}, 'albums_sort_attr': 'name', 'albums_sort_reverse': False, 'atom_feed': {'feed_url': 'http://127.0.0.1:8000/feed.atom', 'nb_items': 10}, ...}
tmpdir = local('/tmp/pytest-of-builduser/pytest-0/test_gallery_max_img_pixels0')
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f0a4da4d540>

    def test_gallery_max_img_pixels(settings, tmpdir, monkeypatch):
        "Test the Gallery class with the max_img_pixels setting."
        # monkeypatch is used here to reset the value to the PIL default.
        # This value does not matter, other than it is "large"
        # to show that settings['max_img_pixels'] works.
        monkeypatch.setattr('PIL.Image.MAX_IMAGE_PIXELS', 100_000_000)

        with open(str(tmpdir.join('my.css')), mode='w') as f:
            f.write('color: red')

        settings['destination'] = str(tmpdir)
        settings['user_css'] = str(tmpdir.join('my.css'))
        settings['max_img_pixels'] = 5000

        logger = logging.getLogger('sigal')
        logger.setLevel(logging.DEBUG)
        try:
            with pytest.raises(PILImage.DecompressionBombError):
                gal = Gallery(settings, ncpu=1)
                gal.build()

            settings['max_img_pixels'] = 100_000_000
            gal = Gallery(settings, ncpu=1)
>           gal.build()

tests/test_gallery.py:387:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
sigal/gallery.py:871: in build
    result = [process_file(media_item) for media_item in medias]
sigal/gallery.py:871: in <listcomp>
    result = [process_file(media_item) for media_item in medias]
sigal/gallery.py:953: in process_file
    return processor(media)
sigal/video.py:230: in process_video
    generate_thumbnail(
sigal/video.py:186: in generate_thumbnail
    check_subprocess(cmd, source, outname)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

cmd = ['ffmpeg', '-i', '/tmp/pytest-of-builduser/pytest-0/test_gallery_max_img_pixels0/video/example video.webm', '-an', '-r', '1', ...]
source = '/tmp/pytest-of-builduser/pytest-0/test_gallery_max_img_pixels0/video/example video.webm'
outname = '/tmp/pytest-of-builduser/pytest-0/test_gallery_max_img_pixels0/video/thumbnails/example video.tn.jpg'

    def check_subprocess(cmd, source, outname=None):
        """Run the command to resize the video and remove the output file if the
        processing fails.

        """
        logger = logging.getLogger(__name__)
        try:
            res = subprocess.run(cmd, capture_output=True)
        except KeyboardInterrupt:
            logger.debug("Process terminated, removing file %s", outname)
            if outname and os.path.isfile(outname):
                os.remove(outname)
            raise

        if res.returncode:
            logger.debug("STDOUT:\n %s", res.stdout.decode("utf8"))
            logger.debug("STDERR:\n %s", res.stderr.decode("utf8"))
            if outname and os.path.isfile(outname):
                logger.debug("Removing file %s", outname)
                os.remove(outname)
>           raise SubprocessException("Failed to process " + source)
E           sigal.video.SubprocessException: Failed to process /tmp/pytest-of-builduser/pytest-0/test_gallery_max_img_pixels0/video/example video.webm

sigal/video.py:59: SubprocessException
------------------------------ Captured log setup ------------------------------
INFO     sigal.settings:settings.py:132 Reading settings ...
------------------------------ Captured log call -------------------------------
INFO     sigal.gallery:gallery.py:793 Using 1 cores
INFO     sigal.gallery:gallery.py:746 Skip empty album: <Album>(path='nonmedia_files', title='nonmedia_files')
INFO     sigal.gallery:gallery.py:746 Skip empty album: <Album>(path='empty', title='empty')
INFO     sigal.gallery:gallery.py:746 Skip empty album: <Album>(path='dir1/empty', title='empty')
DEBUG    sigal.gallery:gallery.py:770 Albums:
dict_values([<Album>(path='webp', title='webp'), <Album>(path='video', title='video'), <Album>(path='nomedia/ignored/recursively_ignored', title='recursively_ignored'), <Album>(path='nomedia/ignored', title='ignored'), <Album>(path='nomedia/created/ignored', title='ignored'), <Album>(path='nomedia/created', title='cre
ated'), <Album>(path='nomedia', title='nomedia'), <Album>(path='iptcTest', title='iptc test gallery'), <Album>(path='exifTest', title='exifTest'), <Album>(path='encryptTest', title='encryptTest'), <Album>(path='dir2', title='Another example gallery with a very long name'), <Album>(path='dir1/test3', title='01 First ti
tle alphabetically'), <Album>(path='dir1/test2', title='test2'), <Album>(path='dir1/test1', title='An example sub-category'), <Album>(path='dir1', title='An example gallery'), <Album>(path='accentué', title='accentué'), <Album>(path='.', title='Sigal test gallery')])
INFO     sigal.image:image.py:178 Processing /build/sigal/src/sigal-2.3/tests/sample/pictures/webp/_MG_7805_lossy80.webp
INFO     sigal.image:image.py:200 Failed to process: DecompressionBombError('Image size (452800 pixels) exceeds limit of 10000 pixels, could be decompression bomb DOS attack.')
INFO     sigal.gallery:gallery.py:793 Using 1 cores
INFO     sigal.gallery:gallery.py:746 Skip empty album: <Album>(path='nonmedia_files', title='nonmedia_files')
INFO     sigal.gallery:gallery.py:746 Skip empty album: <Album>(path='empty', title='empty')
INFO     sigal.gallery:gallery.py:746 Skip empty album: <Album>(path='dir1/empty', title='empty')
DEBUG    sigal.gallery:gallery.py:770 Albums:
dict_values([<Album>(path='webp', title='webp'), <Album>(path='video', title='video'), <Album>(path='nomedia/ignored/recursively_ignored', title='recursively_ignored'), <Album>(path='nomedia/ignored', title='ignored'), <Album>(path='nomedia/created/ignored', title='ignored'), <Album>(path='nomedia/created', title='cre
ated'), <Album>(path='nomedia', title='nomedia'), <Album>(path='iptcTest', title='iptc test gallery'), <Album>(path='exifTest', title='exifTest'), <Album>(path='encryptTest', title='encryptTest'), <Album>(path='dir2', title='Another example gallery with a very long name'), <Album>(path='dir1/test3', title='01 First ti
tle alphabetically'), <Album>(path='dir1/test2', title='test2'), <Album>(path='dir1/test1', title='An example sub-category'), <Album>(path='dir1', title='An example gallery'), <Album>(path='accentué', title='accentué'), <Album>(path='.', title='Sigal test gallery')])
INFO     sigal.image:image.py:178 Processing /build/sigal/src/sigal-2.3/tests/sample/pictures/webp/_MG_7805_lossy80.webp
DEBUG    sigal.image:image.py:126 Processor: ResizeToFit
DEBUG    sigal.image:image.py:150 Save resized image to /tmp/pytest-of-builduser/pytest-0/test_gallery_max_img_pixels0/webp/_MG_7805_lossy80.webp (WEBP)
DEBUG    sigal.image:image.py:170 Save thumnail image: /tmp/pytest-of-builduser/pytest-0/test_gallery_max_img_pixels0/webp/thumbnails/_MG_7805_lossy80.tn.webp (WEBP)
INFO     sigal.image:image.py:178 Processing /build/sigal/src/sigal-2.3/tests/sample/pictures/webp/_MG_7808_lossy80.webp
DEBUG    sigal.image:image.py:126 Processor: ResizeToFit
DEBUG    sigal.image:image.py:150 Save resized image to /tmp/pytest-of-builduser/pytest-0/test_gallery_max_img_pixels0/webp/_MG_7808_lossy80.webp (WEBP)
DEBUG    sigal.image:image.py:170 Save thumnail image: /tmp/pytest-of-builduser/pytest-0/test_gallery_max_img_pixels0/webp/thumbnails/_MG_7808_lossy80.tn.webp (WEBP)
DEBUG    sigal.video:video.py:90 Video size: 320, 240 -> 480, 360
DEBUG    sigal.video:video.py:125 Processing video: ffmpeg -i /build/sigal/src/sigal-2.3/tests/sample/pictures/video/example video.ogv -y -crf 10 -b:v 1.6M -qmin 4 -qmax 63 /tmp/pytest-of-builduser/pytest-0/test_gallery_max_img_pixels0/video/example video.webm
DEBUG    sigal.video:video.py:185 Create thumbnail for video: ffmpeg -i /tmp/pytest-of-builduser/pytest-0/test_gallery_max_img_pixels0/video/example video.webm -an -r 1 -ss 5 -vframes 1 -y /tmp/pytest-of-builduser/pytest-0/test_gallery_max_img_pixels0/video/thumbnails/example video.tn.jpg.tmp.jpg
DEBUG    sigal.video:video.py:54 STDOUT:

DEBUG    sigal.video:video.py:55 STDERR:
 ffmpeg version n6.0 Copyright (c) 2000-2023 the FFmpeg developers
  built with gcc 12.2.1 (GCC) 20230201
  configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-amf --enable-avisynth --enable-cuda-llvm --enable-lto --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libdav1d --enable
-libdrm --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libjack --enable-libjxl --enable-libmfx --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-libra
v1e --enable-librsvg --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxml2 --enable-libxvid --e
nable-libzimg --enable-nvdec --enable-nvenc --enable-opencl --enable-opengl --enable-shared --enable-version3 --enable-vulkan
  libavutil      58.  2.100 / 58.  2.100
  libavcodec     60.  3.100 / 60.  3.100
  libavformat    60.  3.100 / 60.  3.100
  libavdevice    60.  1.100 / 60.  1.100
  libavfilter     9.  3.100 /  9.  3.100
  libswscale      7.  1.100 /  7.  1.100
  libswresample   4. 10.100 /  4. 10.100
  libpostproc    57.  1.100 / 57.  1.100
[matroska,webm @ 0x55dfdf5e8ec0] Could not find codec parameters for stream 0 (Video: vp9, none(bt470bg/bt470bg/bt709, progressive), 320x240): unspecified pixel format
Consider increasing the value for the 'analyzeduration' (0) and 'probesize' (5000000) options
Input #0, matroska,webm, from '/tmp/pytest-of-builduser/pytest-0/test_gallery_max_img_pixels0/video/example video.webm':
  Metadata:
    ENCODER         : Lavf60.3.100
  Duration: 00:00:04.64, start: -0.004000, bitrate: 81 kb/s
  Stream #0:0: Video: vp9, none(bt470bg/bt470bg/bt709, progressive), 320x240, SAR 1:1 DAR 4:3, 25 fps, 25 tbr, 1k tbn
    Metadata:
      ENCODER         : Lavc60.3.100 libvpx-vp9
      DURATION        : 00:00:00.000000000
  Stream #0:1: Audio: opus, 48000 Hz, stereo, fltp
    Metadata:
      ENCODER         : Lavc60.3.100 libopus
      DURATION        : 00:00:04.643000000
Stream mapping:
  Stream #0:0 -> #0:0 (vp9 (native) -> mjpeg (native))
Press [q] to stop, [?] for help
Cannot determine format of input stream 0:0 after EOF
Error marking filters as finished
Conversion failed!
_____________________ test_generate_video_fit_height[mp4] ______________________

tmpdir = local('/tmp/pytest-of-builduser/pytest-0/test_generate_video_fit_height1')
fmt = 'mp4'

    @pytest.mark.parametrize("fmt", ['webm', 'mp4'])
    def test_generate_video_fit_height(tmpdir, fmt):
        """largest fitting dimension is height"""

        base, ext = os.path.splitext(TEST_VIDEO)
        dstfile = str(tmpdir.join(base + '.' + fmt))
        settings = create_settings(video_size=(80, 100), video_format=fmt)
        generate_video(SRCFILE, dstfile, settings)

        size_src = video_size(SRCFILE)
        size_dst = video_size(dstfile)

>       assert size_dst[0] == 80
E       assert 0 == 80

tests/test_video.py:79: AssertionError
______________________ test_generate_video_fit_width[mp4] ______________________

tmpdir = local('/tmp/pytest-of-builduser/pytest-0/test_generate_video_fit_width_1')
fmt = 'mp4'

    @pytest.mark.parametrize("fmt", ['webm', 'mp4'])
    def test_generate_video_fit_width(tmpdir, fmt):
        """largest fitting dimension is width"""

        base, ext = os.path.splitext(TEST_VIDEO)
        dstfile = str(tmpdir.join(base + '.' + fmt))
        settings = create_settings(video_size=(100, 50), video_format=fmt)
        generate_video(SRCFILE, dstfile, settings)

        size_src = video_size(SRCFILE)
        size_dst = video_size(dstfile)

>       assert size_dst[1] == 50
E       assert 0 == 50

tests/test_video.py:96: AssertionError
____________________ test_generate_video_dont_enlarge[mp4] _____________________

tmpdir = local('/tmp/pytest-of-builduser/pytest-0/test_generate_video_dont_enlar1')
fmt = 'mp4'

    @pytest.mark.parametrize("fmt", ['webm', 'mp4', 'ogv'])
    def test_generate_video_dont_enlarge(tmpdir, fmt):
        """Video dimensions should not be enlarged."""

        base, ext = os.path.splitext(TEST_VIDEO)
        dstfile = str(tmpdir.join(base + '.' + fmt))
        settings = create_settings(video_size=(1000, 1000), video_format=fmt)
        generate_video(SRCFILE, dstfile, settings)
        size_src = video_size(SRCFILE)
        size_dst = video_size(dstfile)

>       assert size_src == size_dst
E       assert (320, 240) == (0, 0)
E         At index 0 diff: 320 != 0
E         Full diff:
E         - (0, 0)
E         + (320, 240)
E         ?  ++   ++

tests/test_video.py:112: AssertionError
=============================== warnings summary ===============================
tests/test_cli.py::test_init
  /usr/lib/python3.10/site-packages/pkg_resources/__init__.py:121: DeprecationWarning: pkg_resources is deprecated as an API
    warnings.warn("pkg_resources is deprecated as an API", DeprecationWarning)

tests/test_cli.py: 1 warning
tests/test_gallery.py: 40 warnings
tests/test_image.py: 10 warnings
tests/test_zip.py: 10 warnings
  /usr/lib/python3.10/site-packages/pilkit/processors/resize.py:24: DeprecationWarning: ANTIALIAS is deprecated and will be removed in Pillow 10 (2023-07-01). Use LANCZOS or Resampling.LANCZOS instead.
    img = img.resize((self.width, self.height), Image.ANTIALIAS)

tests/test_cli.py: 1 warning
tests/test_gallery.py: 61 warnings
tests/test_image.py: 4 warnings
tests/test_video.py: 1 warning
tests/test_zip.py: 20 warnings
  /build/sigal/src/sigal-2.3/sigal/image.py:165: DeprecationWarning: ANTIALIAS is deprecated and will be removed in Pillow 10 (2023-07-01). Use LANCZOS or Resampling.LANCZOS instead.
    img = ImageOps.fit(img, box, PILImage.ANTIALIAS, centering=thumb_fit_centering)

tests/test_image.py::test_generate_thumbnail[KeckObservatory20071020.jpg-/build/sigal/src/sigal-2.3/tests/sample/pictures/dir2/KeckObservatory20071020.jpg-wide_size0-high_size0]
tests/test_image.py::test_generate_thumbnail[KeckObservatory20071020.jpg-/build/sigal/src/sigal-2.3/tests/sample/pictures/dir2/KeckObservatory20071020.jpg-wide_size0-high_size0]
tests/test_image.py::test_generate_thumbnail[example.gif-/build/sigal/src/sigal-2.3/tests/sample/pictures/dir1/test1/example.gif-wide_size1-high_size1]
tests/test_image.py::test_generate_thumbnail[example.gif-/build/sigal/src/sigal-2.3/tests/sample/pictures/dir1/test1/example.gif-wide_size1-high_size1]
  /build/sigal/src/sigal-2.3/sigal/image.py:167: DeprecationWarning: ANTIALIAS is deprecated and will be removed in Pillow 10 (2023-07-01). Use LANCZOS or Resampling.LANCZOS instead.
    img.thumbnail(box, PILImage.ANTIALIAS)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED tests/test_gallery.py::test_gallery_max_img_pixels - sigal.video.Subpr...
FAILED tests/test_video.py::test_generate_video_fit_height[mp4] - assert 0 == 80
FAILED tests/test_video.py::test_generate_video_fit_width[mp4] - assert 0 == 50
FAILED tests/test_video.py::test_generate_video_dont_enlarge[mp4] - assert (3...
================= 4 failed, 80 passed, 153 warnings in 19.37s ==================

I'll disable those tests for now, as I don't really know how to fix them. Maybe you have an idea?

saimn commented 1 year ago

Hmm seems to be a problem with ffmpeg, maybe related to the example\ video.ogv file we use in sigal tests, or with ffmpeg itself (I also using Archlinux):

❯ ffmpeg -loglevel error -i /home/simon/dev/sigal/tests/sample/pictures/video/example\ video.ogv -y -crf 10 -b:v 1.6M -qmin 4 -qmax 63 video.webm
[ogg @ 0x560920f06ec0] invalid pts -77120
[ogg @ 0x560920f06ec0] invalid pts -28992

❯ ffmpeg -loglevel error video.webm                                         
[out#0/webm @ 0x55a46c94bd80] Output file does not contain any stream

Will need more investigation.

saimn commented 1 year ago

It seems to be specific to the example ogv file that tests were using, I replaced that file with another one (0923a56a568c08fd05b0fae2e8c4b7247c1476e0).