Can only get imx*compositor to work with videotestsrc in application

[tyounger24]

I have been having some challenges creating a fully functioning Gstreamer application.

I have prototyped all the functions using gst-launch-1.0, and have had success, but my final solution needs to be compiled into an application.

I have been able to get every single element in my test pipelines to function in a compiled application, the issue is that I can't get all of the elements to work together. When I say I've written the examples into an application, I'm creating all the elements and a pipeline, adding to the pipeline, linking, etc., not just passing the pipeline command into a parse command.

For example, I have written the following pipeline into my application:

gst-launch-1.0 rtspsrc location=rtsp://10.1.0.92:554/MainStream ! rtph264depay ! h264parse ! imxvpudec ! imxeglvivsink borderless-window=true sync=false

After sorting out sometimes pads, my application works great.

I have also compiled this into my application:

gst-launch-1.0 -v imxipucompositor name=comp \
sink_0::xpos=0 sink_0::ypos=0 \
sink_1::xpos=512 sink_1::ypos=396 \
! video/x-raw,width=1024,height=768 ! imxeglvivsink native-display=:0 borderless-window=true sync=false \
videotestsrc pattern=0 ! video/x-raw,width=512,height=384 ! queue ! comp.sink_0 \
videotestsrc pattern=1 ! video/x-raw,width=512,height=384 ! queue ! comp.sink_1

After sorting out request pads, my application works great.

I have also compiled this into my application:

gst-launch-1.0 -v $MIXER name=comp \
sink_0::xpos=0 sink_0::ypos=0 sink_0::width=512 sink_0::height=288 \
sink_1::xpos=512 sink_1::ypos=0 sink_1::width=512 sink_1::height=288 \
sink_2::xpos=0 sink_2::ypos=288 sink_2::width=512 sink_2::height=288 \
sink_3::xpos=512 sink_3::ypos=288 sink_3::width=512 sink_3::height=288 \
! video/x-raw,width=1024,height=576 ! imxipuvideotransform ! imxeglvivsink borderless-window=true sync=false \
rtspsrc location=rtsp://10.1.0.92:554/MainStream ! rtph264depay ! h264parse ! imxvpudec ! imxipuvideotransform ! video/x-raw,width=512,height=288 ! queue ! comp.sink_0 \
rtspsrc location=rtsp://10.1.0.94:554/MainStream ! rtph264depay ! h264parse ! imxvpudec ! imxipuvideotransform ! video/x-raw,width=512,height=288 ! queue ! comp.sink_1 \
rtspsrc location=rtsp://10.1.0.94:554/MainStream ! rtph264depay ! h264parse ! imxvpudec ! imxipuvideotransform ! video/x-raw,width=512,height=288 ! queue ! comp.sink_2 \
rtspsrc location=rtsp://10.1.0.92:554/MainStream ! rtph264depay ! h264parse ! imxvpudec ! imxipuvideotransform ! video/x-raw,width=512,height=288 ! queue ! comp.sink_3

When $MIXER is set to "videomixer", my application functions correctly, however CPU usage is high. When $MIXER is set to imxg2dcompositor or imxipucompositor, the pipeline does not work in my application.

What do I need to do differently with the imx compositor than I do with the videomixer?

If I can get the imx compositor to work with h264 live video, I would expect I could have a simpler pipeline:

gst-launch-1.0 -v imxg2dcompositor name=comp \
sink_0::xpos=0 sink_0::ypos=0 sink_0::width=512 sink_0::height=288 \
sink_1::xpos=512 sink_1::ypos=0 sink_1::width=512 sink_1::height=288 \
sink_2::xpos=0 sink_2::ypos=384 sink_2::width=512 sink_2::height=288 sink_2::keep-aspect-ratio=false sink_2::rotation=2 \
sink_3::xpos=512 sink_3::ypos=384 sink_3::width=512 sink_3::height=288 sink_3::keep-aspect-ratio=false sink_3::rotation=2 \
! "video/x-raw, width=1024, height=768" ! imxeglvivsink borderless-window=true sync=false \
rtspsrc location=rtsp://10.1.0.92:554/MainStream ! rtph264depay ! h264parse ! imxvpudec ! queue ! comp.sink_0 \
rtspsrc location=rtsp://10.1.0.94:554/MainStream ! rtph264depay ! h264parse ! imxvpudec ! queue ! comp.sink_1 \
rtspsrc location=rtsp://10.1.0.92:554/SubStream ! rtph264depay ! h264parse ! imxvpudec ! queue ! comp.sink_2 \
rtspsrc location=rtsp://10.1.0.94:554/SubStream ! rtph264depay ! h264parse ! imxvpudec ! queue ! comp.sink_3

This does not work with the videomixer element though as the videomixer element won't scale.

Does anyone have any pointers on how the videomixer and imx*compositor differ, especially when it comes to the difference between an RTSP live video feed and a videotestsrc?

[DuBistKomisch]

Could you elaborate on "does not work"? Got any GST_DEBUG logs?

I've been testing an extremely similar pipeline to your last snippet and it works fine. The only differences are that I'm using 480p videos from a file instead of the network, 854x480 sinks and 1708x960 output, and imxipuvideosink.

[DuBistKomisch]

Actually could it just be that you're missing a queue between the compositor and sink?

[tyounger24]

I've done some further development and have more insight into the issue.

As can be seen, my application is expecting 4 different video cameras. Unfortunately, I have only two video cameras for development. In fact, my application differs slightly from the example in a way that turns out to be critical. The application uses /SubStream for all cameras (MainStream is intended for full screen view, SubStream for the quad view).

I added additional features so that if a camera is not available, I substitute a PNG image indicating such.

After testing this feature, it turns out that the imxg2dcompositor does work.

In order to reproduce my problem, use two cameras twice each.