[topic lin & win] RES memory keep increasing hen running gst-play-1.0 with playlist

[SiewHoon]

Gstreamer framework: 1.12.3 gst-msdk : topic_linux_and_window Environment: wayland weston.

The test.list contains 200 sequence of H264 video command: gst-play-1.0 --playlist /test.list --videosink="mfxsink"

The memory is measured using "top -d 1 | grep -i gst".

The RES is keeping increase when running using gst-play-1.0.

This two fixed patch required to include in gst-plugins-base and gst-plugin-good. https://bugzilla.gnome.org/show_bug.cgi?id=789547 https://bugzilla.gnome.org/show_bug.cgi?id=788759

The RES no long keep increasing after apply this two patches fixed gst-plugins-base and gst-plugins-good for software decode, gst-msdk plugin (master branch), but the RES still keep increase in topic_linux_and_window branch.

[HSD-ES: 1805749008]

[SiewHoon]

I will provide a drop to validation to verify this test with this code changed: https://github.com/ishmael1985/gstreamer-media-SDK/commit/0223ebb3c1a61957c93758028e2be732f5fadd69

[SiewHoon]

Pending to test result.

[ghost]

I tested this with commit id 5392929 and 53cc6d1 from issue #60 applied to topic branch.

The commit id 5392929 does not seem to have resolved the issue in wayland weston environment. Executed the command: gst-play-1.0 --playlist test.list --videosink="mfxsink" in which test.list contains 200 of the same H264 videos in sequence and it is observed that the RES increases gradually over time.

RES increased from ~45000 to ~114400 by the end of the 200th video played. RES_issue_57.log

Rebuilt gst-msdk in debug buildtype and collected the following valgrind logs: valgrind_issue_57.log valgrind_issue_57_2.log

However, while running valgrind, i sometimes get the following error during video playback which I am not sure if it would affect the debug logging: WARNING A lot of buffers are being dropped. WARNING debug information: gstbasesink.c(2901): gst_base_sink_is_too_late (): /GstPlayBin:playbin/GstPlaySink:playsink/GstBin:vbin/GstMfxSinkBin:mfxsinkbin0/Gst MfxSink:mfxsink0: There may be a timestamping problem, or this computer is too slow.

Occasionally the process gets killed as well when running valgrind. I have tried rebooting the system and reinstalling gst-msdk topic branch but the above error still occurs. I tried to use GST_DEBUG="GST_TRACER:7" option to collect debug logs but the logs seems to be not meaningful. The command I used is GST_DEBUG="GST_TRACER:7" GST_TRACERS="leaks" GST_DEBUG_FILE=trace.log gst-play-1.0 --playlist test.list --videosink="mfxsink"

Attaching the trace.log: trace.log

[ishmael1985]

Would you get similar results if you ran the test on X11 environment? https://github.com/intel/gstreamer-media-SDK/commit/53929290a39e7917f8fe55d2a3fa1db04de0b637 only partially fixes the issue, the leak could be related to https://github.com/intel/gstreamer-media-SDK/issues/62

[ghost]

In X11 environment, it looks like the memory leaks are occuring at a faster rate compared to wayland. Did 2 runs to confirm this behaviour and within 30 video playbacks, the RES increased from ~41200 to ~101000. x11_res.log x11_res_2.log

Collected the following valgrind logs in X11: valgrind_issue_57_X11.log

In the valgrind logs, definite lost can be observed in: ==2832== by 0x8B32C1D: MFX::mfx_dll_free(void) (in /usr/lib/gstreamer-1.0/libgstmfx.so) ==2832== by 0x8B31B4D: MFX_DISP_HANDLE::UnLoadSelectedDLL() (in /usr/lib/gstreamer-1.0/libgstmfx.so) ==2832== by 0x8B31B98: MFX_DISP_HANDLE::Close() (in /usr/lib/gstreamer-1.0/libgstmfx.so) ==2832== by 0x8B2EBE1: MFXClose (in /usr/lib/gstreamer-1.0/libgstmfx.so) ==2832== by 0x19B04DC5: MFXInitEx (in /opt/intel/mediasdk/lib64/libmfxhw64-p.so.1.25) ==2832== by 0x8B32022: MFX_DISP_HANDLE::LoadSelectedDLL(char const, eMfxImplType, int, int, mfxInitParam&) (in /usr/lib/gstreamer-1.0/libgstmfx.so) ==2832== by 0x8B2F6B9: MFXInitEx (in /usr/lib/gstreamer-1.0/libgstmfx.so) ==2832== by 0x8B3064A: MFXInit (in /usr/lib/gstreamer-1.0/libgstmfx.so)

This looks similar to the valgrind logs in wayland and also for issue #62 in either environment.

Another observation is that, the video playback lags while using valgrind to collect the logs. Sometimes valgrind crashes and gets killed after just one video playback while using playlist. This is the same case in either wayland or X11.

[ishmael1985]

See my last comment in #62 .

[ishmael1985]

With GStreamer version 1.12.4 that incorporates the two patches as indicated by @SiewHoon in the first post, and the current latest GST-MFX version in the topic_linux_and_window branch, the memory leak issue seems to have been fixed. Could you please verify if this issue has been resolved with the latest GST-MFX version, @kuhanh91 ?

[SiewHoon]

We run with 200 times looping on under wayland weston only. Still seeing RES continue increasing. Still not yet full resolve memory issue.

[ishmael1985]

This was done on GStreamer version 1.12.4 (with corresponding plugins packages)? Or to be sure that this could be a GStreamer component issue itself, maybe you can run the playlist under 1.12.3 with the 2 patches indicated by @SiewHoon at the start of this thread.

[eero-t]

Sometimes valgrind crashes and gets killed after just one video playback while using playlist.

Did you check (from dmesg) whether it got OOM-killed?

If it was OOM-kill, have you checked whether kernel side DRI/GEM object allocations increase: sudo watch cat /sys/kernel/debug/dri/0/i915_gem_objects ?

Note: Above debugfs entry is the only thing listing these allocations, they don't show anywhere on the system except by reducing amount of free memory and causing things to pushed to swap (because data for GEM objects isn't swappable): https://bugs.freedesktop.org/show_bug.cgi?id=106136

When using multiple streams, I've found user-space visible memory usage to less of a problem than the GEM allocations for the GPU memory.

[ishmael1985]

@eero-t thanks for the tips! I've been a little busy these days but I'll try to debug this when I've got the time.