search

bytedeco / javacv

Java interface to OpenCV, FFmpeg, and more
Other
7.44k stars 1.57k forks source link

FFmpegFrameGrabber.grab() missing audio frames #1971

Closed frankfliu closed 1 year ago

frankfliu commented 1 year ago

When I use FFmpegFrameGrabber.grab() audio frames, it stopped before hitting the end.

I'm using a flac file, it suppose contains 106 frame, but FFmpegFrameGrabber stopped at 94 frames:

URL url = new URL("https://github.com/openai/whisper/raw/main/tests/jfk.flac");
try (FFmpegFrameGrabber grabber = new FFmpegFrameGrabber(url)) {
    grabber.setAudioChannels(1);
    grabber.setSampleRate(16000);
    grabber.setAudioCodec(avutil.AV_SAMPLE_FMT_S16);
    grabber.start();
    int frameCount = 0;
    while (grabber.grab() != null) {
        ++frameCount;
    }
    System.out.println("Total frames: " + frameCount);
}

If I use ffmpeg command line it can read all 106 frames correctly:

ffmpeg -nostdin -threads 0 -i jfk.flac -f s16le -ac 1 -ar 16000 -y -loglevel debug out.bin

106 frames successfully decoded, 0 decoding errors
saudet commented 1 year ago

Fixed in commit https://github.com/bytedeco/javacv/commit/53d19bed18731be9e655de7ac96c20753f0f9739. Thanks for reporting!

frankfliu commented 1 year ago

@saudet Thanks for fixing the issue. I tried to use 1.5.9-SNAPSHOT version, but I get the following error:

java.lang.NoClassDefFoundError: Could not initialize class org.bytedeco.ffmpeg.global.avcodec

It looks like the error is caused by missing symbolic link:

~/.javacpp/cache/ffmpeg-5.1.2-1.5.9-SNAPSHOT-macosx-x86_64.jar/org/bytedeco/ffmpeg/macosx-x86_64/libavcodec.dylib
saudet commented 1 year ago

It's not a symbolic link, the libraries can't load for some reason. I'm guessing it's related to the recent addition of the AV1 codecs. @bradh @n-kai-cj Could you look into this?

frankfliu commented 1 year ago

In old version theres is a symbolic link from libavcodec.dylib -> libavcodec.59.dylib, this is missing in 1.5.9-SNAPSHOT version. And the Loader failed to find libavcodec.dylib

bradh commented 1 year ago

I just checked the readfewframes sample with the -SNAPSHOT version, and it worked fine. However that was on linux, and I have no test system for macosx.

My cache directory has the symlinks.

bradh@audax:~/.javacpp/cache/ffmpeg-5.1.2-1.5.9-SNAPSHOT-linux-x86_64-gpl.jar/org/bytedeco/ffmpeg/linux-x86_64-gpl$ ls -al
total 67652
drwxrwxr-x 2 bradh bradh     4096 Jan 17 11:15 .
drwxrwxr-x 3 bradh bradh     4096 Jan 17 11:15 ..
lrwxrwxrwx 1 bradh bradh       16 Jan 17 11:15 libavcodec.so -> libavcodec.so.59
-rw-rw-r-- 1 bradh bradh 54943288 Jan  8 20:08 libavcodec.so.59
lrwxrwxrwx 1 bradh bradh       17 Jan 17 11:15 libavformat.so -> libavformat.so.59
-rw-rw-r-- 1 bradh bradh  9568280 Jan  8 20:09 libavformat.so.59
lrwxrwxrwx 1 bradh bradh       15 Jan 17 11:15 libavutil.so -> libavutil.so.57
-rw-rw-r-- 1 bradh bradh   951488 Jan  8 20:09 libavutil.so.57
lrwxrwxrwx 1 bradh bradh       11 Jan 17 11:15 libdrm.so -> libdrm.so.2
-rw-rw-r-- 1 bradh bradh    70856 Jan  8 20:09 libdrm.so.2
-rw-rw-r-- 1 bradh bradh   699952 Jan  8 20:08 libjniavcodec.so
-rw-rw-r-- 1 bradh bradh   700128 Jan  8 20:09 libjniavformat.so
-rw-rw-r-- 1 bradh bradh  1219216 Jan  8 20:09 libjniavutil.so
-rw-rw-r-- 1 bradh bradh    88760 Jan  8 20:09 libjniswresample.so
-rw-rw-r-- 1 bradh bradh   113440 Jan  8 20:09 libjniswscale.so
lrwxrwxrwx 1 bradh bradh       18 Jan 17 11:15 libswresample.so -> libswresample.so.4
-rw-rw-r-- 1 bradh bradh   125168 Jan  8 20:09 libswresample.so.4
lrwxrwxrwx 1 bradh bradh       15 Jan 17 11:15 libswscale.so -> libswscale.so.6
-rw-rw-r-- 1 bradh bradh   616736 Jan  8 20:09 libswscale.so.6
lrwxrwxrwx 1 bradh bradh       14 Jan 17 11:15 libva-drm.so -> libva-drm.so.1
-rw-rw-r-- 1 bradh bradh    11400 Jan  8 20:09 libva-drm.so.1
lrwxrwxrwx 1 bradh bradh       10 Jan 17 11:15 libva.so -> libva.so.1
-rw-rw-r-- 1 bradh bradh   132144 Jan  8 20:09 libva.so.1

@frankfliu Can you try setting the org.bytedeco.javacpp.logger.debug property to true and providing the relevant parts?

frankfliu commented 1 year ago 
    Debug: Loading class org.bytedeco.javacpp.presets.javacpp
    Debug: Loading class org.bytedeco.javacpp.Loader
    Debug: Loading library jnijavacpp
    Debug: Failed to load for jnijavacpp: java.lang.UnsatisfiedLinkError: no jnijavacpp in java.library.path: [/Users/ec2-user/Library/Java/Extensions, /Library/Java/Extensions, /Network/Library/Java/Extensions, /System/Library/Java/Extensions, /usr/lib/java, .]
    Debug: Could not load Loader: java.lang.UnsatisfiedLinkError: no jnijavacpp in java.library.path: [/Users/ec2-user/Library/Java/Extensions, /Library/Java/Extensions, /Network/Library/Java/Extensions, /System/Library/Java/Extensions, /usr/lib/java, .]
    Debug: Loading class org.bytedeco.javacpp.presets.javacpp
    Debug: Loading class org.bytedeco.ffmpeg.global.avutil
    Debug: Loading class org.bytedeco.javacpp.presets.javacpp
    Debug: Loading class org.bytedeco.ffmpeg.global.avutil
    Debug: Loading /Users/ec2-user/.javacpp/cache/ffmpeg-5.1.2-1.5.9-SNAPSHOT-macosx-x86_64.jar/org/bytedeco/ffmpeg/macosx-x86_64/libavutil.57.dylib
    Debug: Loading /Users/ec2-user/.javacpp/cache/ffmpeg-5.1.2-1.5.9-SNAPSHOT-macosx-x86_64.jar/org/bytedeco/ffmpeg/macosx-x86_64/libjniavutil.dylib
    Debug: Loading class org.bytedeco.javacpp.presets.javacpp
    Debug: Loading class org.bytedeco.javacpp.Pointer
    Debug: Loading class org.bytedeco.javacpp.presets.javacpp
    Debug: Loading class org.bytedeco.ffmpeg.global.avutil
    Debug: Loading class org.bytedeco.ffmpeg.avutil.AVChannelLayout
    Debug: Registering org.bytedeco.ffmpeg.avutil.AVChannelLayout[address=0x600003e4b7a0,position=0,limit=1,capacity=1,deallocator=org.bytedeco.javacpp.Pointer$NativeDeallocator[ownerAddress=0x600003e4b7a0,deallocatorAddress=0x12fafe860]]
    Debug: Loading class org.bytedeco.javacpp.presets.javacpp
    Debug: Loading class org.bytedeco.ffmpeg.global.avutil
    Debug: Loading class org.bytedeco.ffmpeg.global.swresample
    Debug: Loading class org.bytedeco.javacpp.presets.javacpp
    Debug: Loading class org.bytedeco.ffmpeg.global.avutil
    Debug: Loading class org.bytedeco.ffmpeg.global.swresample
    Debug: Loading /Users/ec2-user/.javacpp/cache/ffmpeg-5.1.2-1.5.9-SNAPSHOT-macosx-x86_64.jar/org/bytedeco/ffmpeg/macosx-x86_64/libswresample.4.dylib
    Debug: Loading /Users/ec2-user/.javacpp/cache/ffmpeg-5.1.2-1.5.9-SNAPSHOT-macosx-x86_64.jar/org/bytedeco/ffmpeg/macosx-x86_64/libjniswresample.dylib
    Debug: Loading class org.bytedeco.javacpp.presets.javacpp
    Debug: Loading class org.bytedeco.ffmpeg.global.avutil
    Debug: Loading class org.bytedeco.ffmpeg.global.swresample
    Debug: Loading class org.bytedeco.ffmpeg.global.avcodec
    Debug: Loading class org.bytedeco.javacpp.presets.javacpp
    Debug: Loading class org.bytedeco.ffmpeg.global.avutil
    Debug: Loading class org.bytedeco.ffmpeg.global.swresample
    Debug: Loading class org.bytedeco.ffmpeg.global.avcodec
    Debug: Loading /Users/ec2-user/.javacpp/cache/ffmpeg-5.1.2-1.5.9-SNAPSHOT-macosx-x86_64.jar/org/bytedeco/ffmpeg/macosx-x86_64/libavcodec.59.dylib
    Debug: Failed to load /Users/ec2-user/.javacpp/cache/ffmpeg-5.1.2-1.5.9-SNAPSHOT-macosx-x86_64.jar/org/bytedeco/ffmpeg/macosx-x86_64/libavcodec.59.dylib: java.lang.UnsatisfiedLinkError: Can't load library: /Users/ec2-user/.javacpp/cache/ffmpeg-5.1.2-1.5.9-SNAPSHOT-macosx-x86_64.jar/org/bytedeco/ffmpeg/macosx-x86_64/libavcodec.59.dylib
    Debug: Loading library avcodec
    Debug: Failed to load for avcodec@.59: java.lang.UnsatisfiedLinkError: no avcodec in java.library.path: [/Users/ec2-user/Library/Java/Extensions, /Library/Java/Extensions, /Network/Library/Java/Extensions, /System/Library/Java/Extensions, /usr/lib/java, .]
    Debug: Loading /Users/ec2-user/.javacpp/cache/ffmpeg-5.1.2-1.5.9-SNAPSHOT-macosx-x86_64.jar/org/bytedeco/ffmpeg/macosx-x86_64/libjniavcodec.dylib
    Debug: Failed to load /Users/ec2-user/.javacpp/cache/ffmpeg-5.1.2-1.5.9-SNAPSHOT-macosx-x86_64.jar/org/bytedeco/ffmpeg/macosx-x86_64/libjniavcodec.dylib: java.lang.UnsatisfiedLinkError: Can't load library: /Users/ec2-user/.javacpp/cache/ffmpeg-5.1.2-1.5.9-SNAPSHOT-macosx-x86_64.jar/org/bytedeco/ffmpeg/macosx-x86_64/libjniavcodec.dylib
    Debug: Loading library jniavcodec
    Debug: Failed to load for jniavcodec: java.lang.UnsatisfiedLinkError: no jniavcodec in java.library.path: [/Users/ec2-user/Library/Java/Extensions, /Library/Java/Extensions, /Network/Library/Java/Extensions, /System/Library/Java/Extensions, /usr/lib/java, .]
    Debug: Loading class org.bytedeco.javacpp.presets.javacpp
    Debug: Loading class org.bytedeco.ffmpeg.global.avutil
    Debug: Loading class org.bytedeco.ffmpeg.global.swresample
    Debug: Loading class org.bytedeco.ffmpeg.global.avcodec
frankfliu commented 1 year ago

Here is my cache folder:

[~/.javacpp/cache/ffmpeg-5.1.2-1.5.9-SNAPSHOT-macosx-x86_64.jar/org/bytedeco/ffmpeg/macosx-x86_64]>ls -la
total 62872
drwxr-xr-x  10 lufen  staff       320 Jan 16 11:41 .
drwxr-xr-x   3 lufen  staff        96 Jan 16 11:41 ..
-rw-r--r--   1 lufen  staff  29413456 Jan  8 01:45 libavcodec.59.dylib
-rw-r--r--   1 lufen  staff    718400 Jan  8 01:45 libavutil.57.dylib
lrwxr-xr-x   1 lufen  staff        18 Jan 16 11:41 libavutil.dylib -> libavutil.57.dylib
-rw-r--r--   1 lufen  staff    648560 Jan  8 01:45 libjniavcodec.dylib
-rw-r--r--   1 lufen  staff   1130728 Jan  8 01:45 libjniavutil.dylib
-rw-r--r--   1 lufen  staff    106920 Jan  8 01:45 libjniswresample.dylib
-rw-r--r--   1 lufen  staff    152768 Jan  8 01:45 libswresample.4.dylib
lrwxr-xr-x   1 lufen  staff        21 Jan 16 11:41 libswresample.dylib -> libswresample.4.dylib
frankfliu commented 1 year ago

Just tested on Linux, it works fine and missing audio frames issue is fixed.

bradh commented 1 year ago

@frankfliu that cache directory looks really incomplete (like, where is libavformat?). There are definitely more files than that in the jar file I downloaded.

Can you try removing and retrying it?

saudet commented 1 year ago

The problem is this error. JavaCPP bails out after that.

Debug: Failed to load /Users/ec2-user/.javacpp/cache/ffmpeg-5.1.2-1.5.9-SNAPSHOT-macosx-x86_64.jar/org/bytedeco/ffmpeg/macosx-x86_64/libavcodec.59.dylib: java.lang.UnsatisfiedLinkError: Can't load library: /Users/ec2-user/.javacpp/cache/ffmpeg-5.1.2-1.5.9-SNAPSHOT-macosx-x86_64.jar/org/bytedeco/ffmpeg/macosx-x86_64/libavcodec.59.dylib

Usually we get more useful errors on Mac, but not for this one it seems. There's something it doesn't like about AV1...

frankfliu commented 1 year ago

I cleaned the cached and still the same. I do see other files (e.g. libjniavformat.dylib) in the jar, but not in the cache folder.

otool -L libavcodec.59.dylib

libavcodec.59.dylib:
    @rpath/libavcodec.59.dylib (compatibility version 59.0.0, current version 59.37.100)
    @rpath/libswresample.4.dylib (compatibility version 4.0.0, current version 4.7.100)
    @rpath/libavutil.57.dylib (compatibility version 57.0.0, current version 57.28.100)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)
    /System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox (compatibility version 1.0.0, current version 1000.0.0)
    @rpath/libSvtAv1Enc.1.dylib (compatibility version 1.0.0, current version 1.4.1)
    /System/Library/Frameworks/VideoToolbox.framework/Versions/A/VideoToolbox (compatibility version 1.0.0, current version 1.0.0)
    /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1856.105.0)
    /System/Library/Frameworks/CoreMedia.framework/Versions/A/CoreMedia (compatibility version 1.0.0, current version 1.0.0)
    /System/Library/Frameworks/CoreVideo.framework/Versions/A/CoreVideo (compatibility version 1.2.0, current version 1.5.0)
    /System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0.0, current version 1141.1.0)
    /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1200.3.0)
saudet commented 1 year ago

Ah, here's the problem:

@rpath/libSvtAv1Enc.1.dylib (compatibility version 1.0.0, current version 1.4.1)

@bradh Please make sure the build doesn't create a shared library for SVT-AV1.

bradh commented 1 year ago

@bradh Please make sure the build doesn't create a shared library for SVT-AV1.

Looks like we are missing a bunch of configuration options (including -DBUILD_SHARED_LIBS=OFF) that were meant to be in LIBSVTAV1_CONFIG but never got applied. Running a test build on that.

saudet commented 1 year ago

@frankfliu The builds have been fixed with pull https://github.com/bytedeco/javacpp-presets/pull/1312

frankfliu commented 1 year ago

The snapshot build seems not atomic, I hit the following error:

Could not find ffmpeg-5.1.2-1.5.9-SNAPSHOT-linux-x86_64.jar (org.bytedeco:ffmpeg:5.1.2-1.5.9-SNAPSHOT:20230120.060711-153).
     Searched in the following locations:
         https://oss.sonatype.org/content/repositories/snapshots/org/bytedeco/ffmpeg/5.1.2-1.5.9-SNAPSHOT/ffmpeg-5.1.2-1.5.9-20230120.060711-153-linux-x86_64.jar
bradh commented 1 year ago

That is not the latest snapshot update at https://oss.sonatype.org/content/repositories/snapshots/org/bytedeco/ffmpeg/5.1.2-1.5.9-SNAPSHOT/

As of writing, I see https://oss.sonatype.org/content/repositories/snapshots/org/bytedeco/ffmpeg/5.1.2-1.5.9-SNAPSHOT/ffmpeg-5.1.2-1.5.9-20230120.070542-158-linux-x86_64.jar

Possibly you have something cached?

saudet commented 1 year ago

The snapshot build seems not atomic, I hit the following error:

By atomic if you mean something transactional like a staging repository, Nexus doesn't support that for snapshots, no.

frankfliu commented 1 year ago

I think what you are doing is publishing the SNAPSHOT version for each platform, and then re-publish them again as a whole.

The publishing takes more than one hour. If my build kick in during your publishing time, only one platform is available for the latest version. and gradle will fail to resolve other platform jars.

In the repo, -158.pom contain all the platform, but -175.pom only has on platform. If my build kick in at the time -158.pom is not ready, my build will fail.

saudet commented 1 year ago

Yes, I'm aware, that's the best we can do with Nexus unfortunately. We keep getting back to these things, but the answer is always the same. No one, not even Oracle nor Amazon, is investing in Java anymore, so we're stuck with poor tools like this and we have to come up with hacks like you're doing in DJL that don't work a lot better, that require a lot more maintenance just to keep them half working... I mean if DJL could offer an alternative to Nexus, that's great, but let's be honest, it's not going to happen.

saudet commented 1 year ago

The fix has been released with JavaCV 1.5.9! Enjoy and thanks again for reporting