QVideoSink not working on linux

[wolfseifert]

Running the following program on linux prints out four times called and the image is frozen.

import io.qt.gui.*;
import io.qt.multimedia.*;
import io.qt.multimedia.widgets.*;
import io.qt.widgets.*;

public class VideoSink {
  public static void main(String... args) {
    QApplication.initialize(args);
    var mainWindow = new QMainWindow();
    var videoSink = new QVideoSink();
    var session = new QMediaCaptureSession();
    var camera = new QCamera();
    camera.start();
    session.setCamera(camera);
    session.setVideoOutput(videoSink);
    var label = new QLabel();
    videoSink.videoFrameChanged.connect(frame -> {
      var img = frame.toImage();
      System.out.println("called");
      label.setPixmap(QPixmap.fromImage(img));
    });
    mainWindow.setCentralWidget(label);
    mainWindow.show();
    QApplication.exec();
    QApplication.shutdown();
  }
}

This happens on different hardware and different linux distros (always exactly four times called;-).

This does not happen on windows, it works fine on windows.

The C++ equivalent works fine on linux.

When using a Slot0 as an argument for connect it prints called indefinitely. But then there is no way to access the frame.

The following program works fine on linux:

import io.qt.multimedia.*;
import io.qt.multimedia.widgets.*;
import io.qt.widgets.*;

public class VideoWidget {
  public static void main(String... args) {
    QApplication.initialize(args);
    var videoWidget = new QVideoWidget();
    var session = new QMediaCaptureSession();
    var camera = new QCamera();
    camera.start();
    session.setCamera(camera);
    session.setVideoOutput(videoWidget);
    videoWidget.show();
    QApplication.exec();
    QApplication.shutdown();
  }
}

System

• Manjaro Linux, Linux Mint, OpenSUSE (newest)
• Java 19
• QtJambi 6.5.0
  • Qt 6.5.0

[omix]

The frame argument seems to require an invalidation after use in signal videoFrameChanged. It works well when calling frams.dispose(); at the end of the slot. Since you have a C++ equivalent, could you please test what happens when copying video frame and let it persist?

Please do new QVideoFrame(frame); within the slot. Does the C++ program now have the same behavior?

[wolfseifert]

Yes, calling frame.dispose() at the end of the slot does the trick for Java!

And doing an extra new QVideoFrame(frame) breaks also the C++ version.

So this comes from the difference between Java and C++ with respect to cleanup/destruction of resources.

Is it possible to handle these kind of issues on the QtJambi-side or is it something for the documentation?

[omix]

I try to find a cause of this strange behavior in Qt sources but I don't see it. As long as a copy of the video frame exists the frame's video buffer exists. I think QAbstractVideoBuffer has platform dependent implementation. Its destructor is empty but, maybe, the Linux-implementation does something. I guess, there is an instance count limit (4) in Linux.

In QtJambi const T& types are provided as non-const copies in Java. the copy is Java-owned and thus deleted by garbage collection at arbitrary time in the future. By this, the video buffer persists in memory.

[omix]

The same issue may appear with QImageCapture::imageAvailable

[omix]

I will do an object invalidation after slot return. This will solve the issue at least for lambda slots. When connecting to QObject methods you need to dispose the frame at the end manually.

[omix]

I also added a comment to both signals.

[wolfseifert]

If you commit to the github repo I can pull, build and test.

[wolfseifert]

After a pull and build the result is still the same: it works with frame.dispose(); as the last line in the lambda, but does not work without it.

To verify the upgrade I grepped for Make sure to dispose frame object at the end of the slot! and found it in QImageCapture.java and QVideoSink.java.

I rebuilt everything from scratch, giving it version 6.5.1 in my local sonatype repository, deleting 6.5.0. So the build should be fine.

Are you sure that it should work now without frame.dispose();?

[omix]

If a lambda expression is connected to videoFrameChanged (as in the example code above) it should work automatically. I tested it on Windows with success. I collected all frame objects and tested for disposal after closing the window.

public class TestVideoSink {
    public static void main(String... args) {
        QApplication.initialize(args);
        var camera = new QCamera();
        camera.start();
        switch (camera.error()) {
        case CameraError:
            System.out.println(camera.errorString());
            break;
        case NoError:
            var mainWindow = new QMainWindow();
            var session = new QMediaCaptureSession();
            session.setCamera(camera);
            var videoSink = new QVideoSink();
            session.setVideoOutput(videoSink);
            var label = new QLabel();
            List<QVideoFrame> list = new ArrayList<>();
            class Receiver extends QObject {
                void onVideoFrameChanged(QVideoFrame frame) {
                    var img = frame.toImage();
                    label.setPixmap(QPixmap.fromImage(img));
                    list.add(frame);
                    mainWindow.close();
                }
            }
            Receiver receiver = new Receiver();
            videoSink.videoFrameChanged.connect(receiver::onVideoFrameChanged);
            mainWindow.setCentralWidget(label);
            mainWindow.show();
            QApplication.exec();
            for (QVideoFrame frame : list) {
                if (!frame.isDisposed())
                    System.out.println("Undisposed frame: "+frame);
            }
            mainWindow.dispose();
            break;
        default:
            break;
        }
        QApplication.shutdown();
    }
}

When connecting videoFrameChanged to a Java method that is captured by QMetaObject system (moc) (i.e. an invokable member function of a QObject class) the frams is not disposed after use. This is because not the argument conversion of videoFrameChanged signal is applied but argument conversion of the meta method onVideoFrameChanged. And here, no invalidation after use is done autiomatically. Thus, the user has to make sure to dispose frame as stated in the signal's Java API docs.

Another way is to use non-moc functions by excluding them from QMetaObject with @QtUninvokable annotation.

[wolfseifert]

Running TestVideoSink on linux yields:

Undisposed frame: QVideoFrame(QSize(1280, 720), Format_Jpeg, NoHandle, NotMapped, 00:00.00 - 00:00.33333) 
Undisposed frame: QVideoFrame(QSize(1280, 720), Format_Jpeg, NoHandle, NotMapped, 00:00.64019 - 00:00.97352) 

This issue always was an issue on linux only, so you should test it on linux, not on windows.

In the meantime I patched QtJambi myself: dispose_after_lambda.txt This not nice, but at least it works for lambdas on linux.

[omix]

Yes, I know but frams disposal should work on WIndows either.

[omix]

Your patch is not a good solution. You cause it to check parameters of every single signal argument for every signal no matter if it is videoFrameChanged or not. Also, why don't you cast to QtObjectInterface?

Well, what you are doing is already done in QVideoFrame native code. However, it is only used when the connected slot is not invokable by meta object system.

[wolfseifert]

Ok, I personally can live with an extra dispose(), maybe hidden in some nice Kotlin extension function like this:

fun <A : QVideoFrame> QObject.Signal1<A>.connect(
  connectionType: ConnectionType = ConnectionType.AutoConnection,
  slot: QMetaObject.Slot1<A>,
): QMetaObject.Connection = connect({ a: A ->
  try {
    slot(a)
  } finally {
    a.dispose()
  }
}, connectionType)

So I am closing this issue now...

[wolfseifert]

Here comes the final version:

import io.qt.gui.*;
import io.qt.multimedia.*;
import io.qt.multimedia.widgets.*;
import io.qt.widgets.*;

public class VideoSink {
  public static void main(String... args) {
    QApplication.initialize(args);
    var mainWindow = new QMainWindow();
    var videoSink = new QVideoSink();
    var session = new QMediaCaptureSession();
    var camera = new QCamera();
    camera.start();
    session.setCamera(camera);
    session.setVideoOutput(videoSink);
    var label = new QLabel();
    videoSink.videoFrameChanged.connect(frame -> {
      var image = frame.toImage();
      var pixmap = QPixmap.fromImage(image);
      label.setPixmap(pixmap);
      pixmap.dispose();
      image.dispose();
      frame.dispose();
    });
    mainWindow.setCentralWidget(label);
    mainWindow.show();
    QApplication.exec();
    QApplication.shutdown();
  }
}

All three dispose() are necessary: omitting one of them causes a huge memory-leak, omitting frame.dispose() freezes the image on linux.