Skip video on conversion failure

[miek770]

Hi, I'm having a lot of trouble building my gallery with sigal because it gets errors in video conversion. While the conversion error isn't caused by sigal, the fact that it crashes the program is annoying (and it doesn't build the website).

Ideally, I think sigal shouldn't crash when a conversion fails but instead skip to the next file.

Otherwise, what would be the recommended way to turn off video conversion, give an empty list to video_extensions in the configuration file? It'd be a quick fix, but it's disappointing that it would ignore all videos instead of just the ones which failed.

Thanks!

P.S.: I could share more details on my video conversion errors, but I think fixing them is not relevant in this project (it's not sigal's problem; it's a ffmpeg problem on the Odroid XU4 with Ubuntu 18.04). Ultimately, multiprocessing crashes with the following output:

[h264_v4l2m2m @ 0x47dbc0] driver 'exynos-gsc' on card 'exynos-gsc gscaler'
    Last message repeated 1 times
[h264_v4l2m2m @ 0x47dbc0] driver 's5p-jpeg' on card 's5p-jpeg decoder'
[h264_v4l2m2m @ 0x47dbc0] driver 's5p-jpeg' on card 's5p-jpeg encoder'
[h264_v4l2m2m @ 0x47dbc0] driver 's5p-jpeg' on card 's5p-jpeg decoder'
[h264_v4l2m2m @ 0x47dbc0] driver 's5p-jpeg' on card 's5p-jpeg encoder'
[h264_v4l2m2m @ 0x47dbc0] Could not find a valid device
[h264_v4l2m2m @ 0x47dbc0] can't configure decoder
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (h264_v4l2m2m) -> vp9 (libvpx-vp9))
  Stream #0:1 -> #0:1 (pcm_s16le (native) -> opus (libopus))
Error while opening decoder for input stream #0:0 : Invalid argument

DEBUG - Removing file /srv/sigal/Imprimante 3D/IMG_0007.webm
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/home/sigal/app/lib/python3.6/site-packages/sigal/gallery.py", line 776, in worker
    return process_file(args)
  File "/home/sigal/app/lib/python3.6/site-packages/sigal/gallery.py", line 768, in process_file
    ret = processor(*args[3:])
  File "/home/sigal/app/lib/python3.6/site-packages/sigal/video.py", line 171, in process_video
    options=settings.get(video_format + '_options'))
  File "/home/sigal/app/lib/python3.6/site-packages/sigal/video.py", line 127, in generate_video
    check_subprocess(cmd, source, outname)
  File "/home/sigal/app/lib/python3.6/site-packages/sigal/video.py", line 59, in check_subprocess
    raise SubprocessException('Failed to process ' + source)
sigal.video.SubprocessException: Failed to process /mnt/Photos/Imprimante 3D/IMG_0005.MOV
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/sigal/app/bin/sigal", line 8, in <module>
    sys.exit(main())
  File "/home/sigal/app/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/sigal/app/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/sigal/app/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/sigal/app/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/sigal/app/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/sigal/app/lib/python3.6/site-packages/sigal/__init__.py", line 157, in build
    gal.build(force=force)
  File "/home/sigal/app/lib/python3.6/site-packages/sigal/gallery.py", line 689, in build
    for res in self.pool.imap_unordered(worker, media_list):
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 735, in next
    raise value
sigal.video.SubprocessException: Failed to process /mnt/Photos/Imprimante 3D/IMG_0005.MOV

P.P.S.: My user is sigal, the app is in /home/sigal/app, the shared gallery is in /mnt/Photos, the website build is set to /srv/sigal, and user sigal has all relevant accesses, including to these directories and groups video and audio.

[miek770]

Well I set video_extensions to [] as mentioned above, but now an image also makes the whole process crash. I wish the program could simply skip those problematic files instead of making the build fail entirely..

INFO - Processing /mnt/Photos/000/Clique/06.jpg
INFO - Failed to process: UnidentifiedImageError("cannot identify image file '/mnt/Photos/000/Clique/06.jpg'",)
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/home/sigal/app/lib/python3.6/site-packages/sigal/gallery.py", line 776, in worker
    return process_file(args)
  File "/home/sigal/app/lib/python3.6/site-packages/sigal/gallery.py", line 768, in process_file
    ret = processor(*args[3:])
  File "/home/sigal/app/lib/python3.6/site-packages/sigal/image.py", line 184, in process_image
    generate_image(filepath, outname, settings, options=options)
  File "/home/sigal/app/lib/python3.6/site-packages/sigal/image.py", line 94, in generate_image
    img = _read_image(source)
  File "/home/sigal/app/lib/python3.6/site-packages/sigal/image.py", line 65, in _read_image
    im = PILImage.open(file_path)
  File "/home/sigal/app/lib/python3.6/site-packages/PIL/Image.py", line 2896, in open
    "cannot identify image file %r" % (filename if filename else fp)
PIL.UnidentifiedImageError: cannot identify image file '/mnt/Photos/000/Clique/06.jpg'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/sigal/app/bin/sigal", line 8, in <module>
    sys.exit(main())
  File "/home/sigal/app/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/sigal/app/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/sigal/app/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/sigal/app/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/sigal/app/lib/python3.6/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/sigal/app/lib/python3.6/site-packages/sigal/__init__.py", line 157, in build
    gal.build(force=force)
  File "/home/sigal/app/lib/python3.6/site-packages/sigal/gallery.py", line 689, in build
    for res in self.pool.imap_unordered(worker, media_list):
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 735, in next
    raise value
PIL.UnidentifiedImageError: cannot identify image file '/mnt/Photos/000/Clique/06.jpg'

[saimn]

In debug mode (which you were using at least in the first example) it is expected to fail, as a help to debug issues, but it should not fail otherwise, and just report files for which there was a problem.

[miek770]

I see, thanks. I wouldn't expect a --debug switch to actually change execution, only logging. If that's the desired behavior would you consider updating the doc to make accordingly? http://sigal.saimon.org/en/latest/getting_started.html#help-on-the-sigal-build-command

[saimn]

@miek770 - Yes this should be mentioned indeed. Fixed in 0e49dae02913495c0de69546c2725e180ecfede6.