FFmpegFrameFilter can't get the correct number of frames in android devices

[zhuoran-guo]

I used FFmpegFrameFilter to set the fps filter to get the the frames in android devices. My video has 17s and set the fps to 5, thus it should get around 85 frames in this video, but I only got 69 frames, I want to know why it can not get correct numbers of frames? Thank you so much my code as below:

  public void inferenceIncamDistractionVideoSnapshot() throws Exception {
    final int interval = 200;
    long timestamp = 0;
    final String incamVideoSourcePath = dataRoot + "/images/" + incamVideoSourceName;
    FFmpegFrameGrabber grabber = new FFmpegFrameGrabber(inputStream);
    AndroidFrameConverter convertToBitmap = new AndroidFrameConverter();
    .....
    try {
      grabber.start();
      FFmpegFrameFilter frameFilter = new
              FFmpegFrameFilter("fps=fps=5", "anull", grabber.getImageWidth(), grabber.getImageHeight(), grabber.getAudioChannels());
      frameFilter.setSampleFormat(grabber.getSampleFormat());
      frameFilter.setSampleRate(grabber.getSampleRate());
      frameFilter.setPixelFormat(grabber.getPixelFormat());
      frameFilter.start();

      Frame capturedFrame;
      Frame pullFrame;
      while (true) {
        try {
          capturedFrame = grabber.grab();
          if (capturedFrame == null) {
            Log.w("!!! end cvQueryFrame..video:{}", incamVideoSourcePath);
            break;
          }
          if (capturedFrame.image != null) {
            frameFilter.push(capturedFrame);
          }
          if ((pullFrame = frameFilter.pull()) != null) {
            if (pullFrame.image != null) {
              Bitmap bitmap = convertToBitmap.convert(pullFrame);
              setImage(bitmap, incamRunner.getSurface(), false);
              timestamp += interval;
              incamRunner.setGps(gps.toBuilder().setMsecTimestamp(timestamp).build());
              IncamModelPredictResultProto result = incamRunner.run(timestamp);
              resultsIncam.add(result);
              .......
            }
          }
        } catch (Exception e) {
          e.printStackTrace();
        }
      }
      grabber.stop();
      frameFilter.stop();
    } catch (Exception e) {
      e.printStackTrace();
    }
  }

[saudet]

Check the log for any warning messages

[zhuoran-guo]

@saudet thank you the logcat is like this :

01-01 09:11:41.863  1557  1557 I com.android.phone: Looking for service android.hardware.radio@1.0::IRadio/slot1
01-01 09:11:41.866  1557  1557 I com.android.phone: Looking for service android.hardware.radio.deprecated@1.0::IOemHook/slot1
01-01 09:11:42.809  3184  3211 I ChartEdgeAI/Native: chart_edge_core::model::snpe: DLC model version: incam-distraction_39_8f9119ba
01-01 09:11:44.018  3184  3211 I ChartEdgeAI/Native: chart_edge_core::model::snpe: DLC model version: incam-distraction-stealth_39_803392fa
01-01 09:11:44.021  3184  3211 I SurfaceConsumer: end prepareSurface
01-01 09:11:44.177  3184  3208 E linker  : normalize_path - invalid input: "lib/", the input path should be absolute
01-01 09:11:44.177  3184  3208 W linker  : Warning: unable to normalize "lib/"
01-01 09:11:44.177  3184  3208 E linker  : normalize_path - invalid input: "lib/", the input path should be absolute
01-01 09:11:44.177  3184  3208 W linker  : Warning: unable to normalize "lib/"
01-01 09:11:44.230  3184  3208 W System.err: Warning: Version of org.bytedeco:ffmpeg could not be found.
01-01 09:11:44.379  3184  3208 E linker  : normalize_path - invalid input: "lib/", the input path should be absolute
01-01 09:11:44.379  3184  3208 W linker  : Warning: unable to normalize "lib/"
01-01 09:11:44.379  3184  3208 E linker  : normalize_path - invalid input: "lib/", the input path should be ab
.......
01-01 09:11:52.498  3184  3208 D ChartEdgeAISnapshot: height: 2160, width: 1920
01-01 09:11:52.499  3184  3208 D ChartEdgeAISnapshot: targetSurface.isValid() = true
01-01 09:11:52.508  3184  3208 D ChartEdgeAISnapshot: targetSurface.isValid() = true
01-01 09:11:52.624  3184  3189 I zygote64: Do full code cache collection, code=118KB, data=95KB
01-01 09:11:52.625  3184  3189 I zygote64: After code cache collection, code=117KB, data=88KB
01-01 09:11:53.214  3184  3208 D ChartEdgeAISnapshot: height: 2160, width: 1920
01-01 09:11:53.215  3184  3208 D ChartEdgeAISnapshot: targetSurface.isValid() = true
01-01 09:11:53.226  3184  3208 D ChartEdgeAISnapshot: targetSurface.isValid() = true
01-01 09:11:53.718   698   712 I mmid    : select timeout: wait for receiving msg
01-01 09:11:53.884  1557  1557 I com.android.phone: Looking for service android.hardware.radio@1.0::IRadio/slot1
01-01 09:11:53.887  1557  1557 I com.android.phone: Looking for service android.hardware.radio.deprecated@1.0::IOemHook/slot1
01-01 09:11:53.949  3184  3208 D ChartEdgeAISnapshot: height: 2160, width: 1920
01-01 09:11:53.949  3184  3208 D ChartEdgeAISnapshot: targetSurface.isValid() = true
01-01 09:11:53.959  3184  3208 D ChartEdgeAISnapshot: targetSurface.isValid() = true
01-01 09:11:54.697  3184  3208 D ChartEdgeAISnapshot: height: 2160, width: 1920
01-01 09:11:54.698  3184  3208 D ChartEdgeAISnapshot: targetSurface.isValid() = true
01-01 09:11:54.708  3184  3208 D ChartEdgeAISnapshot: targetSurface.isValid() = true

[saudet]

FFmpeg's log, not Android's log, make sure FFmpegLogCallback.set() has been called.

[zhuoran-guo]

@saudet thank you, I add FFmpegLogCallback.set() in my code as below : I run the code but there is no log shown in terminal is that right way to call the FFmpegLogCallback.set()? and how to find the log ?

public void inferenceIncamDistractionVideoSnapshot() throws Exception {
    ....
    FFmpegFrameGrabber grabber = new FFmpegFrameGrabber(inputStream);
    AndroidFrameConverter convertToBitmap = new AndroidFrameConverter();
    .....
    try {
      FFmpegLogCallback.set()
      grabber.start();
      FFmpegFrameFilter frameFilter = new
              FFmpegFrameFilter("fps=fps=5", "anull", grabber.getImageWidth(), grabber.getImageHeight(), grabber.getAudioChannels());
      frameFilter.setSampleFormat(grabber.getSampleFormat());
      frameFilter.setSampleRate(grabber.getSampleRate());
      frameFilter.setPixelFormat(grabber.getPixelFormat());
      frameFilter.start();

      Frame capturedFrame;
      Frame pullFrame;
      while (true) {
        try {
          capturedFrame = grabber.grab();
          if (capturedFrame == null) {
            Log.w("!!! end cvQueryFrame..video:{}", incamVideoSourcePath);
            break;
          }
          if (capturedFrame.image != null) {
            frameFilter.push(capturedFrame);
          }
          if ((pullFrame = frameFilter.pull()) != null) {
            if (pullFrame.image != null) {
              Bitmap bitmap = convertToBitmap.convert(pullFrame);
              setImage(bitmap, incamRunner.getSurface(), false);
              ......

[saudet]

In the case of Android you should see additional messages appear in your Android log.

[zhuoran-guo]

@saudet Thank you, but no strange log happened when I checked the log, but after I add this code the problem was solved: frameFilter.setFrameRate(grabber.getFrameRate());