Reliability and stability issues when using a multi video pipeline

Hi there.

I have an application that on command is to display a selection of 4 video cameras, and on a command from the user, swap back and forth between this view and a full screen view.

The video cameras provide an H264 video feed via rtsp. As this is running on an iMX6, I am using the following accelerated elements: imxvpudec, imxipuvideotransform, imxg2dtextoverlay, imxg2dcompositor and imxeglvivsink.

I attempted to export a pipeline from my application using GST_DEBUG_BIN_TO_DOT_FILE and GST_DEBUG_BIN_TO_DOT_FILE_WITH_TS, and although a .dot file was produced, it was of size 0 and my application failed to work correctly.

Removing the above macros restored the functionality I am describing here.

To create my quad view, I create the following bins: Video Source (x4): rtspsrc ! rtph264depay ! h264parse ! imxvpudec Video Sink: imxipuvideotransform ! imxg2dtextoverlay ! imxg2dtextoverlay ! imxg2dtextoverlay ! imxg2dtextoverlay ! imxeglvivsink

I connect them together in a pipeline like this (pseudocode): imxg2dcompositor queue 1-4 ! Video Sink Video Source ! queue 1 Video Source ! queue 2 Video Source ! queue 3 Video Source ! queue 4

I also have something that can create image bins: Image Source: filesrc ! pngdec ! imagefreeze ! capsfilter ! videorate

I connect an error callback to the bus, and in case an rtspsrc isn't available, I remove the Video Source bin and replace it with an Image Source bin.

Although this mechanism doesn't cover all possible scenarios, it's good enough for what we need right now.

The issue that I have is some of the time, the first time I issue the command to start the above process, no image appears on the screen, and the virtual address space increases its allocation by 1 megabyte a second.

If I execute the command to stop the above process and then start it again, it starts and displays correctly, with no resource leakage.

Does anyone have any ideas on what to look for as to what's going on?

I've attached the console output from my program that shows the above happening.

At 00:21, I issue the command to start the video the 1st time. At 00:26, two of the cameras tiemout, and the file bins are created. At 00:27, we're in steady state with another thread polling to see if the video is playing (which is not in fact on the screen). This is the state with 1 megabyte per second resource leak (allocated, but not used), but not working properly. At 01:04, I issue the command to cancel the video, and the pipeline is torn down by 01:05. At 01:29 I issue the identical command to start the video the 2nd time. At 01:32.8 we see the two cameras timeout, and the bin substitution occurs. At 01:34, we're in steady state with no resource leakage and the quad video screen showing what is expected; two video cameras and two static images saying that those cameras weren't connected.

We can see that gst_element_get_state times out when there is the issue in question.

So, what can I do to get it to work like it does the 2nd time every time?

One thing that I noticed that might be relevant is what thread is doing what.

If I compare what happened in the log when issuing the command to start, the two regions of the log are almost identical until 0:00:23.25897 and 0:01:29.95989. It is at this time when other threads are created and thrown into the mix, and I'm wondering if this is part of the issue.

Can anyone offer some insight on how to get this to be more stable? Thanks. console.log

Freescale / gstreamer-imx

Reliability and stability issues when using a multi video pipeline #266