gst_pipeline node running but audio topic is not populated

[gc-robotics]

Hi Brett,

The gst_pipeline node is running:

[INFO] [pipeline_node-1]: process started with pid [29862] [pipeline_node-1] [INFO] [1607597739.073032956] [pipeline_node]: setting pipeline to PLAYING

It is pointing to the example.config.yaml provided in gst_bridge.

If I understand right, all these pipelines should be executed:

# assign the properties to each event handler named above audio_test: descr: 'audiotestsrc volume=0.3 is-live=true wave=red-noise ! queue ! audioconvert ! alsasink' video_test: descr: 'videotestsrc is-live=true pattern=ball ! queue ! ximagesink' mic_rossink_test: descr: 'alsasrc ! audioconvert ! rosaudiosink ros-name="audio_node" ros-topic="audio" ros-encoding="S16C2"' rossrc_rossink_test: descr: 'audiotestsrc volume=0.3 is-live=true wave=red-noise ! tee name=t ! queue ! rosaudiosink provide-clock=False ros-name="audio_node" ros-topic="audio" ros-encoding="S16C2" t. ! queue ! audioconvert ! alsasink' rossink_live_test: descr: 'audiotestsrc volume=0.3 is-live=true wave=red-noise ! tee name=t ! queue ! audioconvert ! alsasink t. ! queue ! rosaudiosink ros-name="audio_node" ros-topic="audio" ros-encoding="S16C2"' rossink_test: descr: 'audiotestsrc volume=0.3 is-live=true wave=red-noise ! queue ! audioconvert ! rosaudiosink ros-name="audio_node" ros-topic="audio" ros-encoding="S16C2"'

I checked the status of the "audio" ros topic with RQT but I can see that the queue is not being populated by ros-gst-bridge/gst_pipeline.

I tried to set GST_DEBUG=4 and this is the output: [INFO] [launch]: All log files can be found below /home/alpha/.ros/log/2020-12-10-11-04-13-123191-regulus-30172 [INFO] [launch]: Default logging verbosity is set to INFO [INFO] [pipeline_node-1]: process started with pid [30175] [pipeline_node-1] 0:00:00.000054957 30175 0x14c0900 INFO GST_INIT gst.c:586:init_pre: Initializing GStreamer Core Library version 1.16.2 [pipeline_node-1] 0:00:00.000079741 30175 0x14c0900 INFO GST_INIT gst.c:587:init_pre: Using library installed in /usr/lib/x86_64-linux-gnu [pipeline_node-1] 0:00:00.000088859 30175 0x14c0900 INFO GST_INIT gst.c:605:init_pre: Linux regulus 5.4.0-56-generic #62-Ubuntu SMP Mon Nov 23 19:20:19 UTC 2020 x86_64 [pipeline_node-1] 0:00:00.000254935 30175 0x14c0900 INFO GST_INIT gstmessage.c:128:_priv_gst_message_initialize: init messages [pipeline_node-1] 0:00:00.000634112 30175 0x14c0900 INFO GST_INIT gstcontext.c:84:_priv_gst_context_initialize: init contexts [pipeline_node-1] 0:00:00.000760664 30175 0x14c0900 INFO GST_PLUGIN_LOADING gstplugin.c:318:_priv_gst_plugin_initialize: registering 0 static plugins [pipeline_node-1] 0:00:00.000823189 30175 0x14c0900 INFO GST_PLUGIN_LOADING gstplugin.c:226:gst_plugin_register_static: registered static plugin "staticelements" [pipeline_node-1] 0:00:00.000830013 30175 0x14c0900 INFO GST_PLUGIN_LOADING gstplugin.c:228:gst_plugin_register_static: added static plugin "staticelements", result: 1 [pipeline_node-1] 0:00:00.000842666 30175 0x14c0900 INFO GST_REGISTRY gstregistry.c:1720:ensure_current_registry: reading registry cache: /home/alpha/.cache/gstreamer-1.0/registry.x86_64.bin [pipeline_node-1] 0:00:00.013644460 30175 0x14c0900 INFO GST_REGISTRY gstregistrybinary.c:621:priv_gst_registry_binary_read_cache: loaded /home/alpha/.cache/gstreamer-1.0/registry.x86_64.bin in 0.012783 seconds [pipeline_node-1] 0:00:00.013699742 30175 0x14c0900 INFO GST_REGISTRY gstregistry.c:1579:scan_and_update_registry: Validating plugins from registry cache: /home/alpha/.cache/gstreamer-1.0/registry.x86_64.bin [pipeline_node-1] 0:00:00.015309480 30175 0x14c0900 INFO GST_REGISTRY gstregistry.c:1678:scan_and_update_registry: Registry cache has not changed [pipeline_node-1] 0:00:00.015375461 30175 0x14c0900 INFO GST_REGISTRY gstregistry.c:1755:ensure_current_registry: registry reading and updating done, result = 1 [pipeline_node-1] 0:00:00.015404662 30175 0x14c0900 INFO GST_INIT gst.c:806:init_post: GLib runtime version: 2.64.3 [pipeline_node-1] 0:00:00.015413240 30175 0x14c0900 INFO GST_INIT gst.c:808:init_post: GLib headers version: 2.63.1 [pipeline_node-1] 0:00:00.015418407 30175 0x14c0900 INFO GST_INIT gst.c:810:init_post: initialized GStreamer successfully [pipeline_node-1] 0:00:00.102366408 30175 0x14c0900 INFO GST_ELEMENT_FACTORY gstelementfactory.c:358:gst_element_factory_create: creating element "pipeline" named "rospipe" [pipeline_node-1] [INFO] [1607598253.724772016] [pipeline_node]: setting pipeline to PLAYING [pipeline_node-1] 0:00:00.104539371 30175 0x14c0900 INFO GST_STATES gstelement.c:2647:gst_element_continue_state:<rospipe> committing state from NULL to READY, pending PLAYING, next PAUSED [pipeline_node-1] 0:00:00.104550143 30175 0x14c0900 INFO GST_STATES gstelement.c:2575:_priv_gst_element_state_changed:<rospipe> notifying about state-changed NULL to READY (PLAYING pending) [pipeline_node-1] 0:00:00.104573111 30175 0x14c0900 INFO GST_STATES gstelement.c:2655:gst_element_continue_state:<rospipe> continue state change READY to PAUSED, final PLAYING [pipeline_node-1] 0:00:00.104581661 30175 0x14c0900 INFO GST_STATES gstelement.c:2647:gst_element_continue_state:<rospipe> committing state from READY to PAUSED, pending PLAYING, next PLAYING [pipeline_node-1] 0:00:00.104586082 30175 0x14c0900 INFO GST_STATES gstelement.c:2575:_priv_gst_element_state_changed:<rospipe> notifying about state-changed READY to PAUSED (PLAYING pending) [pipeline_node-1] 0:00:00.104591719 30175 0x14c0900 INFO GST_STATES gstelement.c:2655:gst_element_continue_state:<rospipe> continue state change PAUSED to PLAYING, final PLAYING [pipeline_node-1] 0:00:00.104609045 30175 0x14c0900 INFO GST_EVENT gstevent.c:1449:gst_event_new_latency: creating latency event 0:00:00.000000000 [pipeline_node-1] 0:00:00.104622170 30175 0x14c0900 INFO bin gstbin.c:2783:gst_bin_do_latency_func:<rospipe> configured latency of 0:00:00.000000000 [pipeline_node-1] 0:00:00.104631137 30175 0x14c0900 INFO GST_STATES gstelement.c:2675:gst_element_continue_state:<rospipe> completed state change to PLAYING [pipeline_node-1] 0:00:00.104636249 30175 0x14c0900 INFO GST_STATES gstelement.c:2575:_priv_gst_element_state_changed:<rospipe> notifying about state-changed PAUSED to PLAYING (VOID_PENDING pending)

It seems that the pipeline stops somehow. am I right? Any suggestion?

Thanks!

[BrettRD]

The examples are a bit of a mess (and somewhat underwhelming) at the moment, I think it may be a bug that it ran an audio sink node at all.

Only the bins listed under simple_bins: and webrtc_bins: will get generated, I've set it up this way so that extremely complex pipelines like webrtc can get turned on and off with a single config line. I could have made that clearer in the comments.

On the latest commit, ros2 launch gst_bridge example.launch.py will run two bins inside one pipeline in a ros node. One bin will send static to the speaker, the other will render test images.

Neither bin will output ros topics, but you can turn other bins on by naming their configs under simple_bins:

    simple_bins:
      - audio_test
      - video_test
      - rossink_test

There is an issue where a pipeline that consumes its own output will deadlock on preroll, but I don't think that's the problem here.

[gc-robotics]

Ok, I have added the webrtc_example_bin and rossink_live_test at the same level of audio_test and video_test (under simple_bins). ` simple_bins:

• audio_test
• video_test
• rossink_live_test
• webrtc_example_bin ` The pipeline runs and state is set to "playing". Yet, in rqt there is no audio topic at all.

[BrettRD]

Importantly, the webrtc plugin cannot be loaded by simple_bin. the webrtc plugin has a bunch of signals that need dealing with at runtime, that's why there's a whole loader for that plugin, and a separate list in the config.

simple_bin only reads the description and hands it over to gst-launch. You can simplify the case by running the bin description through gst-launch directly:

gst-launch-1.0 --gst-plugin-path=install/gst_bridge/lib/gst_bridge/ videotestsrc is-live=true pattern=ball ! queue ! ximagesink

and

gst-launch-1.0 --gst-plugin-path=install/gst_bridge/lib/gst_bridge/ audiotestsrc volume=0.3 is-live=true wave=red-noise ! queue ! audioconvert ! rosaudiosink ros-name="audio_node" ros-topic="audio" ros-encoding="S16C2"

This needs to be run from your ros2 workspace for that plugin path to make sense.

[gc-robotics]

Thanks! All clear on webrtc_bin. I am building and running against Rolling. When I run:

gst-launch-1.0 --gst-plugin-path=install/gst_bridge/lib/gst_bridge/ audiotestsrc volume=0.3 is-live=true wave=red-noise ! queue ! audioconvert ! rosaudiosink ros-name="audio_node" ros-topic="audio" ros-encoding="S16C2"

works fine.

It also works when I run:

gst-launch-1.0 --gst-plugin-path=install/gst_bridge/lib/gst_bridge/ rosaudiosrc ! queue ! audioconvert ! alsasink

But when I use the config file and the ros gst_pipeline, everything freezes. I might be wrong, but it looks to me it could be somehow related to rosaudiosrc.

What do you think the issue could be here? What info could I provide to identify the cause? I am not getting any useful message on logs, unfortunately.

Thanks.

[gc-robotics]

Could it be that rosaudiosrc is somehow keeping the pipeline in a pending state? Thanks.

[BrettRD]

That's right, rosaudiosrc won't complete preroll until it receives its first message. I've added a mechanism to load a caps format into the element at load and then skip preroll, but it's really hard to use. If you start src in one terminal, then start sink in another, src should complete preroll.

[gc-robotics]

Not sure I am getting what you mean.

I am executing the plugin from within ROS using gst_pipeline.

I have three bins being define: video_in, audio_in and audio_out

The first two receive images and audio from video and audio src and sink them into ros (rosimagesink and rosaudiosink being the last elements). The first bin relies on rosaudiosrc to wait for an audio message that plays subsequently on alsasink.

The GST pipeline node is unique and loads one single configuration. So, when from launch file in ROS I load the pipeline node, it runs all the bins that I have define in config.

For a moment, I see the cam source going up, I can grab a frame, but then (as soon as the audio_out bin is loaded) all freezes. What should I do at that point? use your way to init the plugin at cold start or just throw an audio message in the ROS topic monitored by audio_out (so that the callback is triggered and the first message arrives?)

Thanks in advance.

[BrettRD]

One approach I've been using to relieve the deadlock is to load two gst_pipeline nodes.

Expand the launch to fire two gst_pipelines with different config files. write the first config with a pipeline that can start immediately like your audio_in and video_in bins, then write a second config for the other pipeline that can wait for the first ROS messages to complete its preroll, like your audio_out bin.

[gc-robotics]

Fantastic! Thanks. I will give it a try and will let you know how it goes.

Any long-term idea on how to implement this in a better and simplified way?

Thanks.

[BrettRD]

Long term, I'd like to better explore options skip preroll entirely and change formats on the fly.

At the moment, it's possible to set init-caps to the expected format and then tell the bin it should skip preroll. This is a little fragile because all of the bins need to be configured to skip preroll or it gets unhappy. This also reduces the flexibility rather a lot because the caps are fully descriptive of the image container so you you need to duplicate config in strange places.

Ideally, the src nodes will be able to default the pipeline to live-mode, and request a change of caps throughout the pipeline at runtime and init-caps would be made redundant. This is possible, but there are precious few examples of how to do it.

[BrettRD]

If preroll remains a major inconvenience, we'll open another issue to address it directly

[gc-robotics]

One approach I've been using to relieve the deadlock is to load two gst_pipeline nodes.

Expand the launch to fire two gst_pipelines with different config files. write the first config with a pipeline that can start immediately like your audio_in and video_in bins, then write a second config for the other pipeline that can wait for the first ROS messages to complete its preroll, like your audio_out bin.

Having two pipeline nodes does the job for now. thanks!