On iOS devices, rendering of videotoolbox decoded frames stops after 30000 frames.

[geminixdev]

Qt 6.4.3 and Qt 6.5.2, on iOS devices, real devices:

Decoding works fine, decoded frames are AV_PIX_FMT_VIDEOTOOLBOX. When sending to display using videoSink->setVideoFrame(), they are redeclared (not converted) to NV12 in the code of QtAVPlayer in QAVVideoFrame::operator QVideoFrame() :

        ...
        case AV_PIX_FMT_D3D11:
        case AV_PIX_FMT_VIDEOTOOLBOX:
        case AV_PIX_FMT_NV12:
            format = VideoFrame::Format_NV12;
            break;
        ...

Then they are taken for display by videoSink->setVideoFrame().

Works fine, plays smooth on iPad and iPhone. Looks perfect!

But after 20 minutes or apparently exactly 30000 frames fed into videoSink->setVideoFrame() the displayed picture and the complete UI is freezing, but not due to something hogging CPU (CPU usage drops significantly).

Now there is only about one UI update per minute, or per two minutes, happening. The UI is unresponsive, but the player continues to play, QtAVPlayer delivers audio and video frames, the audio continues to play, only the pictures are not updated.

Different to running on iOS simulator or Mac the memory usage is increasing slowly up until the freezing. But the freezing occurs at the 30000 frames mark, and not when a certain memory threshold is reached. It freezes also no matter if HD or SD video is played.

When sending dummy frames like this:

videoSink->setVideoFrame(QVideoFrame());

then there is no freeze, and memory usage increases significantly less.

When converting the AV_PIX_FMT_VIDEOTOOLBOX frames to AV_PIX_FMT_YUV420P or AV_PIX_FMT_NV12 before displaying with

convertTo(AV_PIX_FMT_YUV420P);

or

convertTo(AV_PIX_FMT_NV12);

which results in displaying these frames as software frames, then there is also no freeze (but the display is jerky, too slow, so that is unfortunately not a workaround).

So this seems to be quite clearly a problem in QtMM videoSink->setVideoFrame(). However when using QtMM alone, directly, the QtMM black box mediaplayer, then there is no freeze, at least not with Qt 5, it plays for hours and smooth.

The question is now, is there something special to consider when sending the AV_PIX_FMT_VIDEOTOOLBOX frames to setVideoFrame()? May be they need not to be relabelled as NV12, but as something different?

Any ideas?

[geminixdev]

PS. XCode Instruments does not show leaks, nothing.

[valbok]

PS. XCode Instruments does not show leaks, nothing.

Interesting can we trust this? Btw, are you saying it might happening due to mem leak somewhere?

[geminixdev]

PS. XCode Instruments does not show leaks, nothing.

Interesting can we trust this?

Yes, I think so, I have good experience with xcode Instruments, it is even more helpful than valgrind. What I cannot see is if memory is allocated, still accessible, but should have been released because it's not needed anymore. Meaning memory which is not leaked but just unused waste. I don't see that here.

Also I did let the code run on MacOS over night, more than 8 hours, and it's still running and memory usage is stable (according to Activity Monitor in MacOS). If there would be any significant memory leak it should have shown. The code on MacOS is 100% the same, only difference is the targeting of MacOS instead of iOS in the .pro file.

Btw, are you saying it might happening due to mem leak somewhere?

Yes, but not RAM, not memory, but resources, something like file descriptors, file handles (everyting on unix/linux is a file). As the tests confirm every time, the freeze happens after almost exactly 30000 frames are sent to display. The correct value seems to be more 29980 or a little less.

It happens too when I pause playing in between, multiple pauses, and also when I switch the input source (which means resetting the Codecs in QtAVPlayer).

Whatever I do on the frame producing side seems not to matter. When a total of almost 30000 frames, as produced by videotoolbox, without frame format conversion, is sent to display with videoSink->setVideoFrame(), then it freezes the UI.

But the freeze is not 100%, there are very short updates, just about one frame, and some UI elements, every minute or so - I guess because some resources got available again, 2 or 3 file descriptors or similar. Such an update from time to time is unlikely if something crashed or got locked.

And while this is happening in the display thread, the threads handling download and processing of the data run and run without problems whatsoever.

[geminixdev]

Unless there are better ideas, I will try to confirm if this is happening too when using standard QtMM mediaplayer with Qt 6.5.2, on iOS devices, or not.

• If it does happen -> Open bug with Qt.

• If it does not happen -> What is different, what are they doing differently.

  (This will be time consuming, I don't like to go that route, but is here a choice?)

[valbok]

https://github.com/valbok/QtAVPlayer/blob/master/src/QtAVPlayer/qavhwdevice_videotoolbox.mm might be possible that some resources there should be cleaned up or reused. F.e. textures[i] = quint64([m_hw->device newTextureWithDescriptor:desc iosurface:surface plane:i]); also might be device m_hw->device ?

This is very suspicious CVPixelBufferRetain(m_hw->pbuf); because it should be released ? CVPixelBufferRelease (see dtor CVPixelBufferRelease(d->pbuf);

[geminixdev]

https://github.com/valbok/QtAVPlayer/blob/master/src/QtAVPlayer/qavhwdevice_videotoolbox.mm might be possible that some resources there should be cleaned up or reused. F.e. textures[i] = quint64([m_hw->device newTextureWithDescriptor:desc iosurface:surface plane:i]); also might be device m_hw->device ?

This is very suspicious CVPixelBufferRetain(m_hw->pbuf); because it should be released ? CVPixelBufferRelease (see dtor CVPixelBufferRelease(d->pbuf);

Yes, exactly, there might be the problem hidden somewhere. I have not much experience with Objective C, but I'll think through the code there.

[geminixdev]

This is very suspicious CVPixelBufferRetain(m_hw->pbuf); because it should be released ? CVPixelBufferRelease (see dtor CVPixelBufferRelease(d->pbuf);

As a test, I commented out the line with CVPixelBufferRetain(m_hw->pbuf); It seemed to be not useful anyway, because m_hw->pbuf doesn't go out of scope.

// TEST: is this needed???        CVPixelBufferRetain(m_hw->pbuf);

However when testing on Mac, it crashes directly after showing the first frame:

Thread 7 Crashed:: QSGRenderThread
0   CoreFoundation                         0x19643a8c4 _CFRelease + 216
1   libavcodec.59.37.100.dylib             0x10229b154 0x101e04000 + 4813140
2   Player                                 0x100133704 VideoBuffer_MTL::~VideoBuffer_MTL() + 68
3   Player                                 0x100125f90 QAVVideoFramePrivate::~QAVVideoFramePrivate() + 52
4   Player                                 0x1001249f0 QAVStreamFrame::~QAVStreamFrame() + 40
5   Player                                 0x100126080 PlanarVideoBuffer::~PlanarVideoBuffer() + 52
6   QtMultimedia                           0x101585f50 QVideoFrame::operator=(QVideoFrame const&) + 92
7   QtMultimediaQuick                      0x1010a49f0 0x101098000 + 51696
8   QtMultimediaQuick                      0x1010a4b8c 0x101098000 + 52108
9   QtQuick                                0x10393422c QSGBatchRenderer::Renderer::updateMaterialDynamicData(QSGBatchRenderer::ShaderManagerShader*, QSGMaterialShader::RenderState&, QSGMaterial*, QSGBatchRenderer::Batch const*, QSGBatchRenderer::Element*, int, int) + 168
10  QtQuick                                0x1039356ec QSGBatchRenderer::Renderer::prepareRenderMergedBatch(QSGBatchRenderer::Batch*, QSGBatchRenderer::Renderer::PreparedRenderBatch*) + 720
11  QtQuick                                0x103937c30 QSGBatchRenderer::Renderer::prepareRenderPass(QSGBatchRenderer::Renderer::RenderPassContext*) + 1692
12  QtQuick                                0x103937514 QSGBatchRenderer::Renderer::render() + 36
13  QtQuick                                0x10394cf28 QSGRenderer::renderScene() + 344
14  QtQuick                                0x103902a34 QQuickWindowPrivate::renderSceneGraph() + 780
15  QtQuick                                0x103a8eab4 0x103818000 + 2583220
16  QtQuick                                0x103a8f980 0x103818000 + 2587008
17  QtCore                                 0x103f16f84 0x103d3c000 + 1945476
18  libsystem_pthread.dylib                0x196287fa8 _pthread_start + 148
19  libsystem_pthread.dylib                0x196282da0 thread_start + 8

So the CVPixelBufferRelease without prior CVPixelBufferRetain crashes it.

So now I test (again first on the Mac) with both commented out. And it doesn't crash. And memory usage doesn't shoot up. I'll let it run for a while and watch.

[valbok]

pbuf seems ok than, but newTextureWithDescriptor interesting does it require to release this texture, or it creates everytime new.

[geminixdev]

if there is a change in memory usage, then it seems to be slightly less, but not more.

Other than that, unchanged, it runs on Mac forever, but on iPad the UI freeze still happens. This time after exactly 29800 frames sent to display. The CVPixelBufferRelease() / CVPixelBufferRetain() does not cause it.

[geminixdev]

pbuf seems ok than, but newTextureWithDescriptor interesting does it require to release this texture, or it creates everytime new.

            textures[i] = quint64([m_hw->device newTextureWithDescriptor:desc iosurface:surface plane:i]);

That line above? I don't understand it yet, but yes, this looks like being the next candidate to examine.

[geminixdev]

To check, if it might stop the freezing if that textures line would not run, I commented them out:

// TEST:  MTLTextureDescriptor *desc = [MTLTextureDescriptor texture2DDescriptorWithPixelFormat:f width:w height:h mipmapped:NO];

// TEST:  textures[i] = quint64([m_hw->device newTextureWithDescriptor:desc iosurface:surface plane:i]);

I expected that the video might play, just no picture showing, but to my surprise. the images show... video plays.

The textures returned are as set earlier there:

        QList<QVariant> textures = { 0, 0 };

nevertheless, the pictures are displayed. This code might not be needed, at least not in the test case, Qt 6.5.2 on real iPad device.

[geminixdev]

It passed the 32000 frames without freezing !!!!!!

If textures are not needed, possibly nothing in that function is really needed, so new test with the complete function changed to this:

    QVariant handle(QRhi */*rhi*/) const override
    {
        QList<QVariant> textures = { 0, 0 };
        return textures;
    }

Nevertheless the video still plays fine and smooth. I'll let it run now longer to check if there might be still freezing, just later.

[valbok]

QList textures = { 0, 0 };

Exactly means that no textures are used and no HW accel for rendering. Means textures are leaking here.

[geminixdev]

QList textures = { 0, 0 };

Exactly means that no textures are used and no HW accel for rendering.

You mean that the QtMM code handling the display (which called handle()) sees that there are no textures returned, and switches to a different way of rendering? Very probable, and if needed, verifiable in QtMM source.

Rendering seems to be very efficient though, may be because the image to render is already in GPU memory?

In any case, it plays perfectly smooth on the devices I tested so far, iPhone XS Max, iPhone 11 PM, 4th gen iPad 11 Pro. On a very old iPad 12 pro it stutters, but that might have other reasons, I still have to dig in there.

If this performance is confirmed over all iOS devices then it's good enough for use.

Means textures are leaking here.

Yes, but that's then in QtMM code, right? Or, as a workaround, can we keep a handle on these textures and release their memory later, after they had been displayed?

[valbok]

You mean that the QtMM code handling the display (which called handle()) sees that there are no textures returned, and switches to a different way of rendering? Very probable, and if needed, verifiable in QtMM source.

Yes, this is how it is implemented, it checks for texture first, if nothing, gets back to map() which downloads data from GPU if any.

[valbok]

Means textures are leaking here.

Yes, but that's then in QtMM code, right? Or, as a workaround, can we keep a handle on these textures and release their memory later, after they had been displayed?

No, it is somewhere here, we are responsible to free resources, so I will try to look a bit later, might be possible need to release textures in dtor

[valbok]

does it mean that MacOS does not reproduce the issue? regardless how long it works?

[geminixdev]

does it mean that MacOS does not reproduce the issue? regardless how long it works?

Yes, the issue does

• not show on MacOS, and
• not on the iOSSimulator (but that cannot use videotoolbox).

[valbok]

could you check if you return empty textures QList textures = { 0, 0 };

but keep calling textures[i] = quint64([m_hw->device newTextureWithDescriptor:desc iosurface:surface plane:i]); ?

trying to understand where it should be cleared.

[valbok]

maybe it needs double CVPixelBufferRelease(m_hw->pbuf); ?

[valbok]

or there is an issue inside QtMM, can you try just QtMM?

[geminixdev]

could you check if you return empty textures QList textures = { 0, 0 };

but keep calling textures[i] = quint64([m_hw->device newTextureWithDescriptor:desc iosurface:surface plane:i]); ?

trying to understand where it should be cleared.

With this the freeze still happens at the same frame count.

    QVariant handle(QRhi */*rhi*/) const override
    {
        QList<QVariant> textures = { 0, 0 };
        QList<QVariant> textures2 = { 0, 0 };

        CVPixelBufferRelease(m_hw->pbuf);
        m_hw->pbuf = (CVPixelBufferRef)frame().frame()->data[3];
        CVPixelBufferRetain(m_hw->pbuf);

        if (!m_hw->pbuf)
            return textures;

        if (CVPixelBufferGetDataSize(m_hw->pbuf) <= 0)
            return textures;

        auto format = CVPixelBufferGetPixelFormatType(m_hw->pbuf);
        if (format != '420v') {
            qWarning() << "420v is supported only";
            return textures;
        }

        if (!m_hw->device)
            m_hw->device = MTLCreateSystemDefaultDevice();

        IOSurfaceRef surface = CVPixelBufferGetIOSurface(m_hw->pbuf);
        int planes = CVPixelBufferGetPlaneCount(m_hw->pbuf);
        for (int i = 0; i < planes; ++i) {
            int w = IOSurfaceGetWidthOfPlane(surface, i);
            int h = IOSurfaceGetHeightOfPlane(surface, i) ;
            MTLPixelFormat f = i ?  MTLPixelFormatRG8Unorm : MTLPixelFormatR8Unorm;
            MTLTextureDescriptor *desc = [MTLTextureDescriptor texture2DDescriptorWithPixelFormat:f width:w height:h mipmapped:NO];

            textures2[i] = quint64([m_hw->device newTextureWithDescriptor:desc iosurface:surface plane:i]);
        }

        return textures;
    }

I only added textures2, so that textures can get returned empty and textures2 will get filled.

I would have expected that textures would/should get released later, in the code which calls handle(). As far as I can see, here we can only return them, and don't see them again.

[geminixdev]

maybe it needs double CVPixelBufferRelease(m_hw->pbuf); ?

where would you insert that?

[geminixdev]

or there is an issue inside QtMM, can you try just QtMM?

I never had such freezing when using QtMM / QMediaPlayer alone.

I have my old stuff running with Qt 5.15.x and also Qt 6.4.3 QtMM, using QMediaplayer in C++ linked to QML Videooutput. The QMediaPlayer part is now replaced with QtAVPlayer, otherwise almost identical, thus comparable.

This freezing issue did not show up, never. It was extensively tested on real iOS devices also with Qt 6.4.3. What I experienced there was on some devices almost flawless playing, but on many devices often stop and go. QtMM is calling the IOS internal Player, and somewhere there it got hiccups, often, too often. Also when the input has timecodes which are not always perfect, such as when you load streams over the internet and some parts get lost, or the stream is bad quality, or input resetted, then QtMM and the iOS internal Player always stumble.

This is all fixed now with using QtAVPlayer/ffmpeg, it plays for hours without issues. What I hoped for when I stumbled over QtAVPlayer, it got fulfilled, finally a stable Player for Android and iOS.

(Disclaimer: I have modified the PTS syncing a little to adapt to input from constant streams with sometimes messed up PTS values. Such an adaption was not possible with QtMM, but with QtAVPlayer all is possible. Thanks for that!).

[geminixdev]

does it mean that MacOS does not reproduce the issue? regardless how long it works?

Yes, the issue does

• not show on MacOS, and

I have to correct that. The issue shows very similar on the Mac, but it takes much longer to show. I only noticed it when playing streams which should play for hours.

I have not noticed this earlier, because before, at the time we discussed that here, I simply did not test long enough. And then I took over the code from iOS for the Mac version of my App, and that code has the patch for iOS in qavhwdevice_videotoolbox.mm:

    QVariant handle(QRhi */*rhi*/) const override
    {
        QList<QVariant> textures = { 0, 0 };
        return textures;
    }

If I use instead of this patch the original code from qavhwdevice_videotoolbox.mm, it will freeze after a while, it could be about 1 hour, or less.

Tested with Qt 6.6.3, ffmpeg 6.1.1, code compiled for x86_64, on Intel iMac with Big Sur 11.7.10, and on M1 Mac mini with Sonoma 14.5.