🐛 Unable to thumbnail video file at time 0

What's happening?

Videos captured with React Native Vision Camera sometimes fail to thumbnail using AVAssetImageGenerator (via Expo Video Thumbnails). This appears to happen when there is no video frame at time zero. There may be no video frame at time zero when the audio and video streams require synchronization within a container that necessarily has a duration longer than either component stream. We can reproduce this situation by setting a frame rate range on Camera of [15, 30] and transitioning between well-lit and very poorly lit environments during recording.

Below are details for a video that successfully thumbs and a video that does not.

Working Video (well-lit throughout)

Video info

ffprobe ./filepath

ffprobe /Users/username/Downloads/com.app.name\ 2024-07-23\ 22:34.57.596.xcappdata/AppData/Documents/capture/0ECD1FBF-C111-48D7-8775-160200229BF0.mov
ffprobe version 7.0 Copyright (c) 2007-2024 the FFmpeg developers
  built with Apple clang version 15.0.0 (clang-1500.1.0.2.5)
  configuration: --prefix=/opt/homebrew/Cellar/ffmpeg/7.0_1 --enable-shared --enable-pthreads --enable-version3 --cc=clang --host-cflags= --host-ldflags='-Wl,-ld_classic' --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libaribb24 --enable-libbluray --enable-libdav1d --enable-libharfbuzz --enable-libjxl --enable-libmp3lame --enable-libopus --enable-librav1e --enable-librist --enable-librubberband --enable-libsnappy --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libspeex --enable-libsoxr --enable-libzmq --enable-libzimg --disable-libjack --disable-indev=jack --enable-videotoolbox --enable-audiotoolbox --enable-neon
  libavutil      59.  8.100 / 59.  8.100
  libavcodec     61.  3.100 / 61.  3.100
  libavformat    61.  1.100 / 61.  1.100
  libavdevice    61.  1.100 / 61.  1.100
  libavfilter    10.  1.100 / 10.  1.100
  libswscale      8.  1.100 /  8.  1.100
  libswresample   5.  1.100 /  5.  1.100
  libpostproc    58.  1.100 / 58.  1.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/Users/username/Downloads/com.app.name 2024-07-23 22:34.57.596.xcappdata/AppData/Documents/capture/0ECD1FBF-C111-48D7-8775-160200229BF0.mov':
  Metadata:
    major_brand     : qt
    minor_version   : 0
    compatible_brands: qt
    creation_time   : 2024-07-23T21:22:14.000000Z
    com.apple.quicktime.information: Captured with VisionCamera by mrousavy
  Duration: 00:00:15.77, start: 0.000000, bitrate: 1476 kb/s
  Stream #0:0[0x1](und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, mono, fltp, 63 kb/s (default)
      Metadata:
        creation_time   : 2024-07-23T21:22:14.000000Z
        handler_name    : Core Media Audio
        vendor_id       : [0][0][0][0]
  Stream #0:1[0x2](und): Video: hevc (Main) (hvc1 / 0x31637668), yuvj420p(pc, smpte170m/bt709/bt709), 1280x720, 1399 kb/s, 15.89 fps, 59.94 tbr, 600 tbn (default)
      Metadata:
        creation_time   : 2024-07-23T21:22:14.000000Z
        handler_name    : Core Media Video
        vendor_id       : [0][0][0][0]
        encoder         : HEVC
      Side data:
        displaymatrix: rotation of -90.00 degrees

Stream info

ffprobe -v error -show_entries format=duration -show_entries stream=codec_type,duration -of default ./filepath

audio
15.772500
video
15.773333
format
15.773333

Non-working video (transitions between light and dark)

Video info

ffprobe ./filepath

ffprobe /Users/username/Downloads/com.app.name\ 2024-07-23\ 22:34.57.596.xcappdata/AppData/Documents/capture/5AC793A6-EA31-4F43-9D76-9BEE236AEC33.mov
ffprobe version 7.0 Copyright (c) 2007-2024 the FFmpeg developers
  built with Apple clang version 15.0.0 (clang-1500.1.0.2.5)
  configuration: --prefix=/opt/homebrew/Cellar/ffmpeg/7.0_1 --enable-shared --enable-pthreads --enable-version3 --cc=clang --host-cflags= --host-ldflags='-Wl,-ld_classic' --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libaribb24 --enable-libbluray --enable-libdav1d --enable-libharfbuzz --enable-libjxl --enable-libmp3lame --enable-libopus --enable-librav1e --enable-librist --enable-librubberband --enable-libsnappy --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvmaf --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libspeex --enable-libsoxr --enable-libzmq --enable-libzimg --disable-libjack --disable-indev=jack --enable-videotoolbox --enable-audiotoolbox --enable-neon
  libavutil      59.  8.100 / 59.  8.100
  libavcodec     61.  3.100 / 61.  3.100
  libavformat    61.  1.100 / 61.  1.100
  libavdevice    61.  1.100 / 61.  1.100
  libavfilter    10.  1.100 / 10.  1.100
  libswscale      8.  1.100 /  8.  1.100
  libswresample   5.  1.100 /  5.  1.100
  libpostproc    58.  1.100 / 58.  1.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/Users/username/Downloads/com.app.name 2024-07-23 22:34.57.596.xcappdata/AppData/Documents/capture/5AC793A6-EA31-4F43-9D76-9BEE236AEC33.mov':
  Metadata:
    major_brand     : qt
    minor_version   : 0
    compatible_brands: qt
    creation_time   : 2024-07-24T03:48:14.000000Z
    com.apple.quicktime.information: Captured with VisionCamera by mrousavy
  Duration: 00:00:16.13, start: 0.045000, bitrate: 3175 kb/s
  Stream #0:0[0x1](und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, mono, fltp, 62 kb/s (default)
      Metadata:
        creation_time   : 2024-07-24T03:48:14.000000Z
        handler_name    : Core Media Audio
        vendor_id       : [0][0][0][0]
  Stream #0:1[0x2](und): Video: hevc (Main) (hvc1 / 0x31637668), yuvj420p(pc, smpte170m/bt709/bt709), 1280x720, 3141 kb/s, 19.73 fps, 15 tbr, 600 tbn (default)
      Metadata:
        creation_time   : 2024-07-24T03:48:14.000000Z
        handler_name    : Core Media Video
        vendor_id       : [0][0][0][0]
        encoder         : HEVC
      Side data:
        displaymatrix: rotation of -90.00 degrees

Stream info

ffprobe -v error -show_entries format=duration -show_entries stream=codec_type,duration -of default ./filepath

audio
15.914667
video
15.943333
format
16.132563

Other info

Relevant thumbnail code in Expo Video Thumbnails.
Related StackOverflow discussion of the error reproducing for similar, though not identical, reasons.

Reproduceable Code

`fps={[15, 30]}`

// Camera configuration is complicated but we've reduced the issue to the frame rate setting and dynamic FPS.

Relevant log output

I do not have logs at this time but will update the ticket once gathered.

Camera Device

I do not have logs at this time but will update the ticket once gathered. Test uses default multicam.

Device

iPhone 14 Pro

VisionCamera Version

4.4.3

Can you reproduce this issue in the VisionCamera Example app?

No, I have not tried to reproduce this issue in the example app, yet.

Additional information

[X] I am using Expo
[ ] I have enabled Frame Processors (react-native-worklets-core)
[X] I have read the Troubleshooting Guide
[X] I agree to follow this project's Code of Conduct
[X] I searched for similar issues in this repository and found none.

mrousavy / react-native-vision-camera