Test proposed shared buffer PR in JavaFX with VLCJ

[hohonuuli]

JavaFX has a new pull request to support native rendering by supporting WritableImages backed by NIO bytebuffers. This would allow JavaFX to avoid copying data from native buffers to a WritableImage, greatly reducing CPU load. The details of the PR are at https://mail.openjdk.java.net/pipermail/openjfx-dev/2019-June/023347.html.

I thought VLCJ would be provide a good test for this PR.

[caprica]

This is interesting, thanks for the tip!

[caprica]

I had a bit of time to make this work:

https://github.com/caprica/vlcj-javafx/tree/pixelbuffer-test

Feedback welcome, especially if anyone runs performance tests or if a JavaFX expert has any more tips for a more optimal solution.

[hohonuuli]

Awesome! I'm not a JavaFX expert. Still, I hope to carve out a moment tonight to take your test for spin. Thanks for putting that together.

[hohonuuli]

I went to run the demo and I'm getting this compile error in JavaFXDirectRenderingTest:

Error:(224, 86) java: cannot find symbol
  symbol:   method getNativeBuffers()
  location: variable videoSurface of type uk.co.caprica.vlcj.javafx.test.JavaFXDirectRenderingTest.JavaFxVideoSurface

The offending line is:

pixelBuffer = new PixelBuffer(bufferWidth, bufferHeight, videoSurface.getNativeBuffers()[0], pixelFormat);

There's no method getNativeBuffers() ... any tips?

[hohonuuli]

OK making progress, I went to modify the vlcj source to add a method CallbackVideoSurface.getNativeBuffers(). But when I run mvn install I get an error:

Caused by: java.lang.UnsatisfiedLinkError: Unable to load library 'vlc':
dlopen(libvlc.dylib, 9): image not found
dlopen(libvlc.dylib, 9): image not found
Native library (darwin/libvlc.dylib) not found in resource path

I have VLC installed. What's the incantation to get libvlc on the lib path for vlcj's maven build?

[hohonuuli]

Nevermind ... I didn't pay attention that libvlc was only used for tests. I was able to compile with:

mvn install -Dmaven.javadoc.skip=true -Dmaven.test.skip=true

[hohonuuli]

I ran a quick test with the following:

tl;dr: Best rending I've seen from vlcj on a Mac. 👍

I have some 4k ProRes files I'd like to run tests with, but I'm off on vacation for a week. I'll run those tests when I get back.

Video Source

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'V4066_20170822T135738Z_h264.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    creation_time   : 2017-08-22T13:57:38.000000Z
    encoder         : Lavf56.22.100
  Duration: 00:15:00.42, start: 0.000000, bitrate: 26185 kb/s
    Stream #0:0(und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p, 1920x1080 [SAR 1:1 DAR 16:9], 26175 kb/s, 29.97 fps, 29.97 tbr, 30k tbn, 59.94 tbc (default)
    Metadata:
      creation_time   : 2017-08-22T13:57:38.000000Z
      handler_name    : VideoHandler
    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 3 kb/s (default)
    Metadata:
      creation_time   : 2017-08-22T13:57:38.000000Z
      handler_name    : SoundHandler

JDK

openjdk version "12.0.1" 2019-04-16
OpenJDK Runtime Environment AdoptOpenJDK (build 12.0.1+12)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 12.0.1+12, mixed mode, sharing)

OS

Test profiling

[hohonuuli]

The only change I a had to make to vlcj was:

--- a/src/main/java/uk/co/caprica/vlcj/player/embedded/videosurface/CallbackVideoSurface.java
+++ b/src/main/java/uk/co/caprica/vlcj/player/embedded/videosurface/CallbackVideoSurface.java
@@ -79,26 +85,33 @@ public class CallbackVideoSurface extends VideoSurface {
         libvlc_video_set_callbacks(mediaPlayer.mediaPlayerInstance(), lock, unlock, display, null);
     }

+    public ByteBuffer[] getNativeBuffers() {
+        return nativeBuffers.buffers();
+    }

[hohonuuli]

For grins, I ran a test with the new Shenandoah GC ... note the memory usage!

[caprica]

I'll make a corresponding branch in vlcj some time later today to include that change I forgot, thanks.

And thanks for the feedback.

[caprica]

You need the 4.2.0-SNAPSHOT that (at time of writing) is the current vlcj master.

Use:

mvn -Dmaven.test.skip=true clean install

You must use that snapshot build rather than a release build (as per the comments above).

[caprica]

So that profile chart posted by @hohonuuli shows 9Gb heap usage which is insane, but interestingly the CPU load seems to about 8% on average which is somewhat remarkable.

[hohonuuli]

9Gb heap usage which is insane

Just to clarify ... that was when I tested with the new, experimental Shenandoah garbage collector. Otherwise memory usage was less than 300MB using the default Java 12 GC settings.

Shenandoah can be explicitly enabled using -XX:+UnlockExperimentalVMOptions -XX:+UseShenandoahGC. I was pretty shocked by the memory usage of it. I'm sure there's a way to tune it down though.

[caprica]

See #886 for the current state. The vlcj master branch now contains the necessary changes - note that it was done differently (better) than what was described here, the changes should be very easy to accommodate.

[caprica]

Regarding the Shenandoah garbage collector, I saw similar behaviour on Linux, upper heap size looked like 7Gb+ IIRC. I did many forced GC while the video was playing back and it certainly looked like no pauses/stuttering in the video when the GC was running.

It's somewhat hilarious because the demo app runs in about 20Mb of heap ordinarily.

[caprica]

So now I really hope this new PixelBuffer becomes a part of the next JavaFX release.

I would now be very surprised if this JavaFX solution does not out-perform the Swing/Java2D direct-rendering solution.

[caprica]

Link to the pull request for convenient reference: https://github.com/javafxports/openjdk-jfx/pull/472

[caprica]

Reminder to compare performance with ARGB buffer format since it seems it will be supported if this gets included in JFX.

[caprica]

New EA release build:

https://mail.openjdk.java.net/pipermail/openjfx-dev/2019-July/023438.html

[caprica]

It looks like this should make it into JavaFX 13.

[mipastgt]

Hi This is a very simple example of a video player which uses the new WritableImage of JavaFX with support for Buffers. The idea is to let VLC directly render into this buffer and use the image directly in an ImageView without any explicit rendering into a canvas or such thing. Only this brings the desired performance boost. I did this because I think the examples I have seen here are conceptionally wrong. Please correct me if I should have missed something.

What I have not considered yet is any kind of synchronization. I think an extension of the PixelBuffer to support some kind of double-buffering would be the right thing to do but it also runs smoothly without it. I also don't need any timers in this example.

WritableImageVideoDemo.java.zip

[caprica]

Looks very interesting, thanks. I was hoping someone who knew JavaFX would offer something like this. I'll look it over in detail in due course, but I have an immediate question - with your approach, is it still easy to paint an overlay on top of the video?

[hohonuuli]

@mipastgt Thanks for posting your demo! @caprica I think since he's using a StackPane (line 75) It should be easy to throw another AnchorPane or Canvas to draw overlays on top of the ImageView that's rendering the video. When I get a few minutes, I'll give it a try and see how it performs.

[mipastgt]

@caprica Yes, you can use the video image inside an ImageView like any other Node. In a previous version of this demo I had a rotating Label on top of the video inside an AnchorPane. No problem. That's actually the whole purpose of this aproach, to integrate an arbitrary image source into the normal scene graph. You could even use the image as a texture inside a triangle mesh in 3D. (But I haven't tried that myself yet.)

[mipastgt]

Here is the same example with a rotating label on top of the video. WritableImageVideoDemo2.java.zip You can also resize the window and even maximize it and the video will be rescaled on the fly.

[caprica]

Thanks a lot for this.

[mipastgt]

I'd be interested in some performance comparisons. I tried playing a 4k movie trailer with my code on an old MacBook Pro from 2012 and the result was still smooth, the cpu load moderate even with the rotating label on top of it.

[caprica]

I have a 4k sample that had high CPU load on my Linux PC, I will do some comparisons at the weekend.

[caprica]

So this example is really nice. It is a bit simpler than what we already have.

I ran some very brief performance comparisons between this and the version currently in the vlcj-javafx project.

When playing 4K video the CPU usage was broadly comparable. This new version was no better or no worse on average.

I did see mild tearing and stuttering when playing 4K, in both versions as it turns out.

So, in short:

1. I noticed no performance difference
2. This new example is simpler, and is probably more "correct" in terms of how the native buffer should be used (with inherent rescaling, no specific painting and so on).

I still think there needs to be a proper solution to sync renders with vblank, and maybe ideally there is buffer switching to incorporate too. But I am unclear on all of that with JavaFX.

Performance (I mean relative performance between the current example and this new one) seemed slightly better with HD video rather than 4K, but the difference wasn't much.

[mipastgt]

I obtained the following results on my old MacBook Pro from 2012 with a FullHD video:

Old solution: 85-95% CPU New solution: 45-55% CPU VLC native: 12% CPU (just as a reference) (100% would mean 1 core fully utilized.)

I did not observe any stuttering for any of the solutions. (Did you use VLC 3.0.8?)

I agree that double-buffering would be nice. How would that be configured in VLC and how would the API tell me which buffer is currently in use? On the JavaFX side this can probably only be implemented by providing a special version of PixelBuffer which can only be done internally of JavaFX due to a lot of private API in there. But I could have a look at it.

I also tried to get the examples running on Windows 10 because I have attached an LG 5k/2k monitor their, but I could not get any of the examples working. I did not get past a black window.

[caprica]

I used an 8-core i7 with a mid-range GPU (at least at the time it was bought, a few years ago), on Linux with native nVidia drivers.

I don't see stuttering or tearing with full HD, only with 4K.

I guess that that the Linux code path is not as well optimised as OSX or Windows as I see double the CPU usage you report using either solution.

I see no significant difference when comparing one solution over the other, and certainly nowhere near a difference of nearly 100% that you are reporting. I don't doubt your figures, by the way. Performance is very sensitive to the execution environment of course.

As to the details on how to make multi-buffer work, I have no idea really.

[caprica]

Regarding your problems in Windows - I seem to remember reading something in the VLC development list that this new LibVLC code path was not yet functional on Windows.

[mipastgt]

Just some further data: My machine is a Intel Core i7, 2.6 GHz, 4 cores, 8 GB RAM, NVIDIA GeForce GT 650M 1 GB, 2880 x 1800 Retina With my new solution I can also play a 4k video without stuttering but have to reduce the window size so that it fits on my smaller screen. The CPU usage goes up to 80-90% then. With the previous solution I cannot play this video at all. I just see an initial frame as a still image and I can hear the sound but that's it. As for Windows: The native VLC player works without problem.

[caprica]

Linux performance definitely appears to be sub-par. On paper my CPU specs look slightly better, I don't know about the GPU, but even then my CPU usage appears higher. I'm no expert at performance analysis but I expected lower CPU.

Comparing the native VLC player is not a like-for-like comparison, it is using a completely different code path to render video than what we're talking about here.

Well, thanks for your feedback, it was very interesting.

I like your simplified implementation a lot, and it's likely the "right" way to do it. I'm just not seeing big improvements with my own setup which I'm kinda sad about.

[wimdeblauwe]

@mipastgt Thank you so much for the demo source code. I just have run it with Java 11 using OpenJFX 13-ea+14b and VLCJ 4.2.0 and seems to be running fine on my macbook.

CPU Usage was around 30%, compared to 7% if I play the same video on VLC directly. Is this what to be expected or should CPU usage differ less ?

[caprica]

So my view... this is a good solution, it's simple, it performs well, but it will never match the performance of VLC's optimal native rendering. I don't know how close to par your 30% CPU usage is, but I would be interested to know just how close to native performance could be achieved with this approach.

Honestly though, I have more hope for LibVLC 4.x's upcoming OpenGL rendering callbacks, IMO that should give superior performance. But how to integrate that into a JavaFX app I don't know.

So in short, I didn't really answer your question, sorry, but what is shown here is likely to be the "best" approach for using vlcj with JavaFX, no matter what it's performance characteristics are you probably won't find any other solution that's better.

[mipastgt]

@wimdeblauwe Yes, I think that is the expected performance. Its exact percentage depends on the general performance of your maching but the relative performance is pretty much what I would expect. @caprica I agree with you as far as the status quo goes. But if you have the oportunity in the future to render a video directly in OpenGL, then there may be an even better solution. Have a look at this project: https://github.com/eclipse-efx/efxclipse-drift and here https://tomsondev.bestsolution.at/2018/12/04/announcing-efxclipse-driftfx-integrating-native-rendering-pipelines-into-javafx/ I would consider this project being in its early stages. I have seen the demo in Zürich myself and it was very impressive. When I tried the software myself however it was not yet really stable (the Java 11 branch) but that will probably change after some more time. If it works, this will be the ultimate solution with probably the best performance you can get in JavaFX. I will follow this project and see how it evolves.

[wimdeblauwe]

@caprica @mipastgt I was not complaining about the performance, just wanted to share. I am very happy that something like this exists. Using the demo code, I was able to build a simple POC in an hour with very acceptable performance.

[wimdeblauwe]

For the code in https://github.com/caprica/vlcj/issues/883#issuecomment-523197593 -> It uses a trick to work around the Java module system. For those that want to avoid that, you just need to put the code in a package of your liking, remove the workaround and add a module-info.java file with something like this:

module myapplication {
    requires javafx.controls;
    requires javafx.fxml;
    requires javafx.web;
    requires vlcj;

    exports com.company.app;
}

Where myapplication is any name you like to call your "module" (which is your application) and com.company.app is the package where you have put the demo code.

To have it running easily, this is a Maven pom.xml that works:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>mygroup</groupId>
    <artifactId>myartifact</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>jar</packaging>

    <name>My App</name>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <java.version>11</java.version>
        <javafx.version>13-ea+14b</javafx.version>

        <maven.compiler.source>${java.version}</maven.compiler.source>
        <maven.compiler.target>${java.version}</maven.compiler.target>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.openjfx</groupId>
            <artifactId>javafx-controls</artifactId>
            <version>${javafx.version}</version>
        </dependency>
        <dependency>
            <groupId>org.openjfx</groupId>
            <artifactId>javafx-fxml</artifactId>
            <version>${javafx.version}</version>
        </dependency>
        <dependency>
            <groupId>org.openjfx</groupId>
            <artifactId>javafx-web</artifactId>
            <version>${javafx.version}</version>
        </dependency>
        <dependency>
            <groupId>uk.co.caprica</groupId>
            <artifactId>vlcj</artifactId>
            <version>4.2.0</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.8.1</version>
                <configuration>
                    <release>${java.version}</release>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

[caprica]

Whilst that is very useful, so thanks for posting it, it just shows to me the needless baggage introduced with the Java Module System. I understand the claimed benefits but in reality I don't see the point.

[hohonuuli]

re: Java Module System ... tl;dr yes, it's a huge PITA. But there's no way around it now.

Unfortunately, the javapackager tool provided in JDK 8 has been dropped. It was a handy tool for generating "native" looking java apps. It's in-development replacement, jpackage, uses jlink under-the-hood to build the native apps. I've been porting several JavaFX apps from JDK 8 to 11 and packaging them using jpackage. One lesson learned is that library developers need to be at least module-aware and follow some best practices, especially around reflection, package names, and the naming of resource directories in the jar. Otherwise, at best, it requires developers to do funky contortions to get a library to work or, at worst, the pain of using a library is no longer worth it. I had to rip out a number of 3rd party libraries during the migration from 8 to 11.

[mipastgt]

@hohonuuli I fully agree with your opinion about the module system but what exactly are your problems? I am building a non-modular 100+ jar-file application with jpackage and do not have any problem with the module system because I strictly avoid it. You can use jpackage and jlink without fully going modular and suffering from all the pain.

[hohonuuli]

@mipastgt Yeah, I just went all in with the modules which probably was a mistake. Some example pain points were:

• Two different modules had resources in /images ... that was a no-no.
• Guice ... not worth the struggle of making it work in a modular project
• slf4j ... jlink couldn't decide if the module name was slf4j.api or org.slf4j. Took way to long to work through that. Also, logback has a dependency on the servlet api ... the workaround is either to explicitly include it or manually specify what goes in the merged module.
• A module had a swing component that uses an image in it's own module. Another module subclassed that component but then couldn't access the image causing instantiation to fail.
• Different modules importing the same classes from different jars (e.g. javax.activation)
• Some 3rd party jars that used JavaFX would include both mac and linux javafx dependencies. These had to be manually excluded or jlink would fail.
• etc.

[mipastgt]

Yes, that is exactly the kind of problems which I also experienced when I tried to go modular. As I also didn't see any benefit in the module system which could not also be achieved in a less painfull way, I decided to completely avoid it and that worked for me so far and with the GraalVM at the horizon the module system becomes even less important.

[hohonuuli]

Running tests using @mipastgt code. I've packaged that code up at https://github.com/hohonuuli/vlcfx to make it simple for folks to play with. Here's two test cases. Both videos are from one of our autonomous underwater vehicles running video transects at around 700m below the ocean's surface. The vehicle is running relatively fast while looking at very small animals, so any stuttering in video playback is very noticeable. Both videos are stored on the same external FireWire SSD. A typical video frame looks like:

Video 1 (2048x1080, 60fps)

Video looks great. Like really great. The playback is very smooth. On my Mac computer there's no subjective difference between this and playback from QuickTime player.

Codec info:

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/Volumes/LBD1/M3/master/i2MAP/2019/07/20190709/i2MAP_20190709T175042Z_701m_F031_9.mov':
  Metadata:
    major_brand     : qt
    minor_version   : 537199360
    compatible_brands: qt
    creation_time   : 2019-07-09T17:50:41.000000Z
  Duration: 00:06:35.56, start: 0.000000, bitrate: 505949 kb/s
    Stream #0:0(eng): Audio: pcm_s24le (in24 / 0x34326E69), 48000 Hz, 1 channels, s32 (24 bit), 1152 kb/s (default)
    Metadata:
      creation_time   : 2019-07-09T17:50:41.000000Z
      handler_name    : AJA Sound Media Handler
    Stream #0:1(eng): Audio: pcm_s24le (in24 / 0x34326E69), 48000 Hz, 1 channels, s32 (24 bit), 1152 kb/s (default)
    Metadata:
      creation_time   : 2019-07-09T17:50:41.000000Z
      handler_name    : AJA Sound Media Handler
    Stream #0:2(eng): Video: prores (HQ) (apch / 0x68637061), yuv422p10le(tv, unknown/reserved/reserved, progressive), 2048x1080, 502855 kb/s, SAR 1:1 DAR 256:135, 59.94 fps, 59.94 tbr, 60k tbn, 60k tbc (default)
    Metadata:
      creation_time   : 2019-07-09T17:50:41.000000Z
      handler_name    : AJA Video Media Handler
      encoder         : Apple ProRes 422 (HQ)
      timecode        : 17:50:41:12
    Stream #0:3(eng): Data: none (tmcd / 0x64636D74)
    Metadata:
      rotate          : 0
      creation_time   : 2019-07-09T17:50:41.000000Z
      handler_name    : AJA Time Code Handler
      reel_name       : 082
      timecode        : 17:50:41:12

Profiling info

Video 2 (4096x2160, 30fps)

Noticeable stuttering and dropped frames. Video is certainly usable, but the playback is decidedly not smooth. 4K playback isn't quite there yet.

Codec info:

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/Volumes/LBD1/M3/master/i2MAP/2019/05/20190503/i2MAP_20190503T202328Z_600m_F031_13.mov':
  Metadata:
    major_brand     : qt
    minor_version   : 537199360
    compatible_brands: qt
    creation_time   : 2019-05-03T20:23:28.000000Z
  Duration: 00:06:34.43, start: 0.000000, bitrate: 902929 kb/s
    Stream #0:0(eng): Audio: pcm_s24le (in24 / 0x34326E69), 48000 Hz, 1 channels, s32 (24 bit), 1152 kb/s (default)
    Metadata:
      creation_time   : 2019-05-03T20:23:28.000000Z
      handler_name    : AJA Sound Media Handler
    Stream #0:1(eng): Audio: pcm_s24le (in24 / 0x34326E69), 48000 Hz, 1 channels, s32 (24 bit), 1152 kb/s (default)
    Metadata:
      creation_time   : 2019-05-03T20:23:28.000000Z
      handler_name    : AJA Sound Media Handler
    Stream #0:2(eng): Video: prores (HQ) (apch / 0x68637061), yuv422p10le(tv, unknown/reserved/reserved, progressive), 4096x2160, 900193 kb/s, SAR 1:1 DAR 256:135, 29.97 fps, 29.97 tbr, 30k tbn, 30k tbc (default)
    Metadata:
      creation_time   : 2019-05-03T20:23:28.000000Z
      handler_name    : AJA Video Media Handler
      encoder         : Apple ProRes 422 (HQ)
      timecode        : 20:23:28:13
    Stream #0:3(eng): Data: none (tmcd / 0x64636D74)
    Metadata:
      rotate          : 0
      creation_time   : 2019-05-03T20:23:28.000000Z
      handler_name    : AJA Time Code Handler
      reel_name       : 082
      timecode        : 20:23:28:13

Profiling info

[mipastgt]

@hohonuuli Thanks for the tests. Maybe I should have mentioned that I also uploaded my code on GitHub :-) https://github.com/mipastgt/JFXToolsAndDemos I tried to separate the demo code from the actual functionality to make it a bit more usable.

[mipastgt]

@hohonuuli You have used the Apple ProRes 422 (HQ) codec for the video. So I would not say that 4k in general is not yet usable but with this codec I am even astonished that it did work at all because this codec is very bandwidth-hungry :-) I doubt that I could even play that with Quicktime on my machine.

[hohonuuli]

@mipastgt Yep, it's a pretty monster codec. We do have reasons for it though. For the type of science we do, our researchers have found that compression artifacts from video codecs can make it hard to do their work. We have a pretty fancy video target, with lots of different lines, grids, patterns, etc, to test for artifacts produced by different cameras and codecs. ProRes HQ was the winner, and we use it as a master codec. We do write mezzanine and proxy versions of each video too though.

We're going to be doing a lot of machine learning work on our video next year. For that, I really need a good toolkit for drawing on top of video. I'm rooting for VLCJ so I don't have to go write something in Swift/AVFoundation or C/C++

[caprica]

Very interesting, thanks.

[caprica]

@hohonuuli by the way, your project sounds amazing!