Pipeline switching not working

[Esbob]

I'm using gst-interpipe to switch between two playing webcams (both Logitech C920) by setting the "listen-to" property and I've made it work with the following:

Source pipeline 1: v4l2src device=/dev/video0 ! interpipesink name=webcam0 Source pipeline 2: v4l2src device=/dev/video1 ! interpipesink name=webcam1 Sink pipeline: interpipesrc name=src listen-to=webcam0 ! videoconvert ! xvimagesink

where I start by getting the webcam0 images on screen, subsequently setting listen-to=webcam1, and the sink correctly receives webcam1 images. So far, so good. But when I change the sink pipeline to the following, it stops working:

interpipesrc name=src listen-to=webcam0 format=time ! queue ! video/x-h264,framerate=30/1 ! rtph264pay config-interval=1 pt=96 ! queue ! udpsink host=127.0.0.1 port=8004"

That is, it works fine until I try to set listen-to=webcam1: the webcam1 pipeline then stops playing and the sink receives no data. I can switch back to the other webcam by setting listen-to=webcam0 and the streaming works again as expected. With my limited understanding of gst-interpipe, I'm guessing there must be some pipeline synchronization problems during the switch, and I've tried to change different parameters but without success. I can't tell if this is a bug or if I'm just missing some configuration.

Any help would be greatly appreciated.

I'm using:

• gstreamer version 1.14.1
• gst-interpipe version 1.0.4

[TopperBG]

Did you switch src pipeline like that? gstd-client element_set name_of_pipe_sink src listen-to webcam0/webcam1 Here is a good example how to construct and manage pipelines.

[Esbob]

I'm using gstreamer-sharp and setting the listen-to property through the indexer:

pipeline["listen-to"] = "webcam1";

When I look at gstreamer-sharp examples online, this seems to be the ordinary way to set element properties. Besides, it does work, just not when the sink pipeline contains my particular x-h264/rtph264pay/udpsink configuration. That is also why I'm thinking it might be solved if I knew how to properly set the combination of gstreamer parameters such as is-live, allow-renegotiation, etc.

[TopperBG]

If I debug first will try it command line and tools like gstd-client to be sure pipe architecture is correct. Give it a try. Also are you sure that stream from your cameras (both) are h264 compressed ? What version of GSTD did you use, can you tail the /usr/local/var/log/gstd/gstd.log

[Esbob]

I'm not using gstd and I know my pipeline architecture is correct since I can send streamed webcam data to my Janus server using this pipeline. But it only works with the first interpipesink that the interpipesrc listens to (the one the pipeline is initialized with). In other words, this works: v4l2src device=/dev/video0 ! interpipesink name=webcam0 + interpipesrc name=src listen-to=webcam0 format=time ! queue ! video/x-h264,framerate=30/1 ! rtph264pay config-interval=1 pt=96 ! queue ! udpsink host=127.0.0.1 port=8004

But switching to webcam1 does not work... My two C920 webcams are identical and generate h264 data by themselves.

[TopperBG]

I'm using similar setup but in use of gstd and don't have a problem. Also have more 3 or 4 (depends from source) pipes for record or re-encoding.

[michaelgruner]

I does sound like a timestamping mismatch. Can you try setting interpipesrc enable-sync=true? This will try to adjust the incoming timestamps to match the src pipeline running time. This way the downstream elements will see a continuous, monotonic flow of timestamps.

[Esbob]

I've tried this before, but with no luck. Are there other parameters that could help synchronize the two pipelines?

Following @TopperBG 's recommendation, I tried modifying the gstd example he linked to, and it works as long as the sink pipeline is using xvimagesink. When I use the configuration below, the pipelines are successfully created and playing, but no data is sent to the port:

gstd-client pipeline_create pipe_1_src v4l2src device=/dev/video0 \
! "video/x-raw, framerate=15/1, width=640, height=480" ! queue ! interpipesink name=src_1 \
caps=video/x-raw,width=640,height=480,framerate=15/1 sync=false async=false

gstd-client pipeline_create pipe_2_src v4l2src device=/dev/video1 \
! "video/x-raw, framerate=15/1, width=640, height=480" ! queue ! interpipesink name=src_2 \
caps=video/x-raw,width=640,height=480,framerate=15/1 sync=false async=false

gstd-client pipeline_create pipe_4_sink interpipesrc name=interpipesrc1 listen-to=src_1 \
is-live=true allow-renegotiation=true enable-sync=true ! queue ! rtph264pay config-interval=1 pt=96 ! udpsink async=false sync=false qos=false port=8004 host=127.0.0.1 \
auto-multicast=true

Btw, thanks a lot for all the advice you guys are providing :-)

[TopperBG]

Look at that case in /usr/local/var/gstd/gstd.log for errors. It's very heavy debug log ;)

[Esbob]

OK, I've debugged the following two pipelines:

gstd-client pipeline_create pipe_1_src v4l2src device=/dev/video0 ! queue ! interpipesink name=src_1 +

gstd-client pipeline_create pipe_4_sink interpipesrc name=interpipesrc1 listen-to=src_1 \
is-live=true allow-renegotiation=true enable-sync=true ! queue ! rtph264pay config-interval=1 pt=96 ! udpsink host=127.0.0.1 port=8004

There are no hard errors in the log file I've attached below and the only thing I notice is this: gst_base_sink_chain_unlocked:<src_1> object unref after render which I read somewhere else might mean a missing packet.

debug.log

Any ideas?

[TopperBG]

Try with: gstd-client pipeline_create pipe_1_src v4l2src device=/dev/video0 ! queue ! interpipesink name=src_1 async=true Also is good to place caps at src pipe as per your video caps=video/x-h264 ........ etc.

[Esbob]

Setting caps and sync=false async=true on the source + enable-sync=true on the sink actually made it work with gstd. The only issue is that when I switch between sources, there are some nasty image artifacts leaking from one source to the next, meaning that the image is not looking nice for 5 seconds after the switch - is there a remedy for that?

[michaelgruner]

This looks like the encoder’s inter-prediction losing its reference. However I did not see where in the pipeline you are placing the encoder.

I assume, if this is happening, that you have an encoder on each interpipesink pipeline? I would factor out the encoder to the interpipesrc pipeline, so the encoding reference is not lost in the transition.

On Feb 12, 2019, at 9:27 AM, Esben Visfeldt notifications@github.com wrote:

Setting caps and sync=false async=true on the source + enable-sync=true on the sink actually made it work with gstd. The only issue is that when I switch between sources, there are some nasty image artifacts leaking from one source to the next, meaning that the image is not looking nice for 5 seconds after the switch - is there a remedy for that?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/RidgeRun/gst-interpipe/issues/27#issuecomment-462803443, or mute the thread https://github.com/notifications/unsubscribe-auth/AAr0OKGU1BQlvozciujwZjFTxOhjGj01ks5vMt1VgaJpZM4azJov.

[Esbob]

I haven't added an encoder since the Logitech C920 webcam delivers a H.264 encoded video stream - the very reason I bought it. I want to send this H.264 data directly to the UDP sink to be picked up by a Janus media server and redistributed as a webRTC stream. I'm new to gstreamer, so it's very possible that I have forgotten something basic - do I still need an encoder? For instance, this works fine without an encoder: gst-launch-1.0 -v -e v4l2src device=/dev/video1 ! queue ! video/x-h264,width=1280,height=720,framerate=30/1 ! rtph264pay config-interval=1 pt=96 ! udpsink host=127.0.0.1 port=8004

Can you suggest the optimal way to split the above line into interpipe source and sink?

[TopperBG]

As I said - is good to put caps like this video/x-h264,width=1280,height=720,framerate=30/1 in source pipes. Also if you don't want broken frames you could issu EOS before switch pipelines. Look at that https://developer.ridgerun.com/wiki/index.php?title=GStreamer_Daemon_-_EOS and that, looks exactly your case https://developer.ridgerun.com/wiki/index.php?title=GstInterpipe_-_Dynamic_Switching

And no - you don't need addition encoder in case you don't want to reencode in another format stream.

[Esbob]

The caps did help, thanks @TopperBG . Sounds interesting with this EOS, but I need a little help: Which pipelines should it be sent to and if I send it, won't it close the pipelines? Will I have to set the pipelines to play again? I guess I don't really understand the consequences of issuing an EOS.

[TopperBG]

First send EOS event to output pipeline to cut off the stream, then switch input pipeline. Just a guess, my arch is opposite - one source, few outputs.

[Esbob]

I've tried the EOS idea, but I can't figure out how to make it work and flush the sink. This must be a relatively common problem when using the listen-to property, yet I can't find any examples illustrating how to handle the broken frames when switching. Can anyone point me to an example or explain to me more detailed how I can configure the sources and the sink in order to get a 'clean' switch? Any help is much appreciated :-)

[TopperBG]

Like that? https://gstreamer.freedesktop.org/data/events/gstreamer-conference/2017/David%20Soto%20-%20GStreamer%20Daemon%20-%20Building%20a%20media%20server%20under%2030min.pdf

[Esbob]

I can't get it to work properly - switching still has a tendency to temporarily corrupt the images - but I'm not sure it's an interpipe problem, so I'll close the issue. Thanks for your help and patience @TopperBG :-)

[anilgond]

@Esbob , its still pending or are you able to do switching? . I am also stuck at same point, any help will be appreciated. Thanks in advance.

[Esbob]

@Esbob , its still pending or are you able to do switching? . I am also stuck at same point, any help will be appreciated. Thanks in advance.

I dropped gst-interpipe and used input-selector instead: Source Pipeline (one for each webcam): uvch264src device=[webcam ID] initial-bitrate=3000000 average-bitrate=3000000 iframe-period=1000 name=uvcsrc[source ID] auto-start=true uvcsrc[source ID].vidsrc ! video/x-h264, framerate=30/1, width=1280, height=720 ! queue ! selector.sink_[sink ID] Sink Pipeline: input-selector name=selector sync-mode=1 ! rtph264pay config-interval=1 pt=96 ! udpsink host=127.0.0.1 port=8004

This worked for me, though there is still a short moment of disrupted frames when switching. Code: https://github.com/Esbob/WebCamStreamer

[michaelgruner]

@fasm22 please provide a working set of pipelines for future readers.

[dutsuwak]

Hi,

Please notice the use of input video caps is really important. If using different camera source videos the camera caps must match the same format. This specially important when switching between different camera types, for examples the change between an USB camera and a MIPI camera.

Use the following example as a reference for camera switching (the example uses Gstd):

First camera: gstd-client pipeline_create campipe nvarguscamerasrc ! 'video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)NV12, framerate=(fraction)30/1' ! interpipesink name=eo_camera sync=false async=true

Second camera: gstd-client pipeline_create campipe2 nvarguscamerasrc sensor-id=1 ! 'video/x-raw(memory:NVMM), width=(int)1280, height=(int)720, format=(string)NV12, framerate=(fraction)30/1' ! interpipesink name=ir_camera sync=false async=true

Output pipeline: gstd-client pipeline_create camsinkpipe interpipesrc name=interpipe_sink listen-to=eo_camera ! nvvidconv ! xvimagesink sync=false async=true

Start pipelines: gstd-client pipeline_play campipe gstd-client pipeline_play campipe2 gstd-client pipeline_play camsinkpipe

Then, you can change between pipelines with:

gst-client element_set camsinkpipe interpipe_sink listen-to ir_camera gst-client element_set camsinkpipe interpipe_sink listen-to eo_camera

The following gif shows the behaviour when switching between two cameras. Test setup (works in other platforms as well: desktop, Xavier, Nano, etc):

• Jetson TX2
• Jetpack 4.3
• Gst-Interpipe 1.1.0
• Two IMX219 Cameras

[anilgond]

I am writing this using C code, as follows:

Following are pipelines: ret = gstc_pipeline_create (client, "pipe1", "videotestsrc pattern=smpte is-live=true ! video/x-raw, framerate=15/1, width=640, height=480 ! qu eue ! interpipesink name=src_1 sync=false async=false");

ret = gstc_pipeline_create (client, "pipe2", "videotestsrc pattern=ball is-live=true ! video/x-raw, framerate=15/1, width=640, height=480 ! qu eue ! interpipesink name=src_2 sync=false async=false");

ret = gstc_pipeline_create (client, "pipe3", "videotestsrc pattern=snow is-live=true ! video/x-raw, framerate=15/1, width=640, height=480 ! qu eue ! interpipesink name=src_3 sync=false async=false");

ret = gstc_pipeline_create (client, "pipe_sink", "interpipesrc name=interpipesrc1 listen-to=src_3 is-live=true allow-renegotiation=true ! queu e ! autovideosink async=false sync=false");

I am calling following statements in while loop and taking the input from UDP packet, gstc_element_set(client, "pipe_sink","interpipesrc1", "listen-to","src_1"); gstc_element_set(client, "pipe_sink","interpipesrc1", "listen-to","src_2"); gstc_element_set(client, "pipe_sink","interpipesrc1", "listen-to","src_3");

Only for first input it will switch to another pipeline after that it will freeze the current frame whenever second input received.

[dutsuwak]

Hi @anilgond

Here is a gif with a working version of the code you provided:

You can use the following C code as a working reference:

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

#include "libgstc.h"

int main (int argc, char *argv[]){
  GstClient *client;
  GstcStatus ret;
  const char *address = "127.0.0.1";
  const unsigned int port = 5000;
  const long wait_time = -1;
  const int keep_open = 1;

  ret = gstc_client_new (address, port, wait_time, keep_open, &client);
  if (GSTC_OK != ret) {
    fprintf (stderr, "There was a problem creating a GstClient: %d\n", ret);
    goto out;
  }

  ret = gstc_client_ping (client);
  if (GSTC_OK == ret) {
    printf ("GStreamer daemon is alive!\n");
  } else {
    fprintf (stderr, "Unable to contact server: %d\n", ret);
    goto free_client;
  }

  ret = gstc_pipeline_create (client, "pipe1", "videotestsrc \
        pattern=smpte is-live=true ! video/x-raw, framerate=15/1, width=640,\
        height=480 ! queue ! interpipesink name=src_1 sync=false async=false");
  if (GSTC_OK != ret) {
    fprintf (stderr, "Error creating pipeline: %d\n", ret);
    goto free_client;
  }

  ret = gstc_pipeline_create (client, "pipe2", "videotestsrc pattern=ball \
        is-live=true ! video/x-raw, framerate=15/1, width=640, height=480 ! \
        queue ! interpipesink name=src_2 sync=false async=false");
  if (GSTC_OK != ret) {
    fprintf (stderr, "Error creating pipeline: %d\n", ret);
    goto free_client;
  }

  ret = gstc_pipeline_create (client, "pipe3", "videotestsrc pattern=snow \
        is-live=true ! video/x-raw,framerate=15/1, width=640, height=480 ! \
        queue ! interpipesink name=src_3 sync=false async=false");
  if (GSTC_OK != ret) {
    fprintf (stderr, "Error creating pipeline: %d\n", ret);
    goto free_client;
  }

  ret = gstc_pipeline_create (client, "pipe_sink", "interpipesrc name=interpipesrc1 \
        listen-to=src_3 is-live=true allow-renegotiation=true ! queue ! autovideosink \
        async=false sync=false");
  if (GSTC_OK != ret) {
    fprintf (stderr, "Error creating pipeline: %d\n", ret);
    goto free_client;
  }

  ret = gstc_pipeline_play (client, "pipe1");
  if (GSTC_OK != ret) {
    fprintf (stderr, "Error playing pipeline: %d\n", ret);
    goto free_client;
  }
  ret = gstc_pipeline_play (client, "pipe2");
  if (GSTC_OK != ret) {
    fprintf (stderr, "Error playing pipeline: %d\n", ret);
    goto free_client;
  }
  ret = gstc_pipeline_play (client, "pipe3");
  if (GSTC_OK != ret) {
    fprintf (stderr, "Error playing pipeline: %d\n", ret);
    goto free_client;
  }
  ret = gstc_pipeline_play (client, "pipe_sink");
  if (GSTC_OK != ret) {
    fprintf (stderr, "Error playing pipeline: %d\n", ret);
    goto free_client;
  }

  for(int i=0; i < 30; i++){
    if(i%4 == 0){
      printf("Change to src3\n");
      gstc_element_set(client, "pipe_sink","interpipesrc1", "listen-to","src_3");
    }
    else if(i%3 == 0){
      printf("Change to src1\n");
      gstc_element_set(client, "pipe_sink","interpipesrc1", "listen-to","src_1");
    }
    else if(i%2 == 0){
      printf("Change to src2\n");
      gstc_element_set(client, "pipe_sink","interpipesrc1", "listen-to","src_2");
    }

    sleep(2);
  }

  printf ("Press any enter to stop pipeline...\n");
  getchar ();

  ret = gstc_pipeline_stop (client, "pipe");
  if (GSTC_OK == ret) {
    printf ("Pipeline set to null!\n");
  } else {
    fprintf (stderr, "Unable to stop pipeline: %d\n", ret);
    goto free_client;
  }

  ret = gstc_pipeline_delete (client, "pipe");
  if (GSTC_OK == ret) {
    printf ("Pipeline deleted!\n");
  } else {
    fprintf (stderr, "Unable to delete ipeline: %d\n", ret);
    goto free_client;
  }

 free_client:
  gstc_client_free (client);

 out:
  return ret;
}

[anilgond]

Hi @fasm22 I have tried your code also and it didn't worked for me, means the problem is in my setup. Now, I have setup a new machine and its working fine for your code as well as for my code. Thanks for your help :)

[anilgond]

Hi @fasm22 I am getting following error: gstd error :

---

0:00:13.191517576 788 0x2683850 ERROR interpipesink gstinterpipesink.c:803:gst_inter_pipe_sink_add_listener: Can not add listener interpipesrc1 because our caps are not defined yet 0:00:13.191538508 788 0x2683850 ERROR interpipesink gstinterpipesink.c:870:gst_inter_pipe_sink_remove_listener: Listener interpipesrc1 is not registered in node src_1 0:00:13.191911730 788 0x2683850 ERROR interpipesrc gstinterpipesrc.c:267:gst_inter_pipe_src_set_property: Could not listen to node src_2

---

I am trying to play videos back to back. I have taken 2 pipes and creating , playing and destroying one by one. I have used interpipes. Following is the description of my source pipe : Pipe description : "filesrc location=%s ! capsfilter name=cf ! decodebin ! queue ! interpipesink name=%s forward-eos=true sync=false async=false"

Caps : gstc_element_set(client, pipe1,"cf","caps","video/x-raw,width=%d,height=%d,framerate=%d/%d",640,480,30,1);

Sink pipe : "interpipesrc name=interpipesrc1 listen-to=src_1 allow-renegotiation=true accept-eos-event=true do-timestamp=true ! autovideosink async=false sync=false"

I have used listen-to and after this i have added following code : gstc_pipeline_bus_wait (client, "pipe_sink", "eos", -1, &message);

Any help will be appreciated. Thanks in advance :)

[jcaballeros]

Hi,

The latest release v1.1.3 included a fix for the error:

"Can not add listener interpipesrc1 because our caps are not defined yet"

Please test with the latest sources. I'm closing this issue, please reopen if you still encounter a problem.