search

Xilinx / video-sdk

https://xilinx.github.io/video-sdk
Other
31 stars 14 forks source link

XmaSession statistics resources are not released #88

Open Keukhan opened 7 months ago

Keukhan commented 7 months ago

Hello. experts!

I'm developing a streaming server and using XMA. While creating and destroying the encoder repeatedly, the process suddenly terminated and the following log was output.

...
[XMA-Session-Stats] WARNING: Session id: 16037, type: encoder still has unused completd cu cmds
[XMA-Session-Stats] WARNING: Session id: 16038, type: encoder still has unused completd cu cmds
[XMA-Session-Stats] WARNING: Session id: 16044, type: encoder still has unused completd cu cmds
[XMA-Session-Stats] WARNING: Session id: 16045, type: encoder still has unused completd cu cmds
[XMA-Session-Stats] WARNING: Session id: 16046, type: encoder still has unused completd cu cmds
[XMA-Session-Stats] WARNING: Session id: 16047, type: encoder still has unused completd cu cmds
[XMA-Session-Stats] WARNING: Session id: 16053, type: encoder still has unused completd cu cmds
[XMA-Session-Stats] WARNING: Session id: 16054, type: encoder still has unused completd cu cmds
[XMA-Session-Stats] WARNING: Session id: 16055, type: encoder still has unused completd cu cmds
[XMA-Session-Stats] WARNING: Session id: 16056, type: encoder still has unused completd cu cmds
[XMA-Session-Stats] WARNING: Session id: 16062, type: encoder still has unused completd cu cmds
[XMA-Session-Stats] WARNING: Session id: 16063, type: encoder still has unused completd cu cmds
[XMA-Session-Stats] WARNING: Session id: 16064, type: encoder still has unused completd cu cmds
...

Analyzing the log, it was the xmaapi.cpp file, and it seems that sessions were accumulated in the g_xma_singleton->all_sessions_vec vector and could not be released. In the developed code, the session = xma_enc_session_create function and xma_enc_session_destroy(session) are called normally, but the XmaSession of g_xma_singleton->all_sessions_vec does not seem to be removed when the xma_enc_session_destroy function is called.

    for (auto& itr1: g_xma_singleton->all_sessions_vec) {
        xclLogMsg(NULL, XRT_INFO, "XMA-Session-Stats", "--------");
        XmaHwSessionPrivate *priv1 = (XmaHwSessionPrivate*) itr1.hw_session.private_do_not_use;
        if (priv1->kernel_complete_count != 0 && !priv1->using_cu_cmd_status) {
            xclLogMsg(NULL, XRT_WARNING, "XMA-Session-Stats", "Session id: %d, type: %s still has unused completd cu cmds", itr1.session_id, 
                xma_core::get_session_name(itr1.session_type).c_str());
        }

Issue It is unavoidable if the policy is to continuously store statistical information, but as a result, CPU usage increases and processes suddenly terminate.

Is there a way to release resources when decoder, scaler and encoder are repeatedly create/destroy without terminating the process, as in my case? Please help.

NastoohX commented 7 months ago

Hi Thank you for bringing this issue to our attention. I have informed our engineering group of this problem and will provide an update as soon as we have one. In the mean time, could you, kindly, describe your use case in more details, e.g., frequency of creation, whether requested resources are monotonically incrementing, etc.? Cheers,

Keukhan commented 7 months ago

@NastoohX

Please understand that it is too late to report additional information

I'm not using the XMA API directly, but FFmpeg's AVCodec (mpsoc_vcu_h264).

The way I reproduce the issue is to repeatedly open/close the codec without terminating the process, and the memory increases monotonically. If the codec Open Close is performed once every 5 seconds, the process seems to terminate within 6 hours.

  1. avcodec_alloc_context3
  2. avcodec_open2 (mpsoc_vcu_h264, encoder)
  3. while(true) { avcodec_send_frame -> avcodec_receive_packet }
  4. avcodec_close
  5. avcodec_free_context

I tried to report the issue by writing a reproducible sample code, but it was too simple, so I only wrote pseudo code as above. Has enough information been provided? Please let me know if you need more information to analyze the cause. I am waiting for good news!

Thanks you.

NastoohX commented 7 months ago

Hi, Thank you for the extra info and steps to reproduce this issue. I have opened an internal ticket to track this issue and will provide updates here, as soon as they are available. Cheers, Internal ticket: CR-1191227

cdeshpan-xlnx commented 4 months ago

@Keukhan Could you please provide exact source code of your app. I am interested in trying out your application.

Keukhan commented 4 months ago

@cdeshpan-xlnx

Thank you for your reply. I maintain an open source project called OvenMediaEngine, and XMA is used in this project. It is used by using the mpsoc_vcu_h264/hevc codec integrated in FFmpeg. Please let me know if you need more information to resolve the issue. I'm waiting for good news. :)