Audio synchronization issue

[vixp24]

Hi all,

I've converted an existing(working) gstreamer pipeline to use gst-interpipes with the following split. When running this combination of pipelines, I seem to be getting periodic audio blipping/skipping, which sounds like a forced synchronization being done by interpipesrc. The audio blipping is removed when stream-sync=1, but then the audio & video are completely out of sync.

Video pipeline: rtspsrc ! rtph264depay ! h264parse ! queue ! interpipesink name=video

Audio pipeline: alsasrc ! queue ! audioconvert ! avenc_ac3 ! queue ! interpipesink name=audio

Combined pipeline: interpipesrc listen-to=video is-live=true allow-renegotiation=true stream-sync=0 ! queue ! h264parse ! mp4mux name=mux ! filesink \ interpipesrc listen-to=audio is-live=true allow-renegotiation=true stream-sync=0 ! queue ! mux.

Does anyone see any obvious issue, or some way I can achieve proper audio/video sync without interruption?

Thank you!

[jcaballeros]

Hi @vixp24,

Could you test having the video and the audio interpipesinks in the same pipeline so that they share the same clock and use stream-sync=1?

For instance:

pipeline_create pipe1 rtspsrc location=<your stream location> ! rtph264depay ! h264parse ! queue ! interpipesink name=video alsasrc ! queue ! audioconvert ! avenc_ac3 ! queue ! interpipesink name=audio

pipeline_create pipe2 interpipesrc listen-to=video is-live=true allow-renegotiation=true stream-sync=1 ! queue ! h264parse ! mp4mux name=mux ! filesink location=/tmp/test.mp4 interpipesrc listen-to=audio is-live=true allow-renegotiation=true stream-sync=1 ! queue ! mux.

pipeline_play pipe2
pipeline_play pipe1

[vixp24]

Hi @jcaballeros ,

I just tested that configuration, and yes that seems to work, and the video/audio are more in sync with no blipping. However since I am porting this over a C++ code base, the audio & video source pipelines need to be separate objects/classes. Is there a way for them to share a clock if in different pipelines?

When keeping the source pipelines separate, I have noticed when stream-sync=1, I do not experience the audio blipping issue, but the audio/video time sync is off by ~700-1000 ms. I did a ffprobe on the recording, and noticed the start_pts and start_time, were not 0 for the video stream, while the audio was at 0. Do you know how the PTS of the video stream can be adjusted in gstreamer to start in sync with the audio?

Any help is appreciated. Thanks!

[vixp24]

Hi @jcaballeros ,

I'm still having some problems with A/V syncing. Following the same idea as above, I called gst_pipeline_use_clock() on the video source pipeline to use the audio source pipeline's clock. I also had to keep stream-sync=1 for the audio's corresp. interpipesrc and stream-sync=2 for the video's corresp. interpipesrc. With these changes, the A/V is in sync for the first 20 ish seconds, but then the sync drifts and becomes worse over time.

What could be causing the clock drift, if the two source pipelines should be sharing the clock? Am I missing something?

Thanks

[jcaballeros]

Hi @vixp24 ,

Interesting, I think that might be related to using stream-sync=compensate-ts in only one of the interpipesrc, let me run some tests and get back to you

[vixp24]

Thank you,

It is worth mentioning that when using stream-sync=0 for both audio & video interpipsinks, it works fine (no blipping), when I use a different USB microphone. This may give some clue as to what could be wrong.

But I know for sure that the audio does not blip when there is no timestamp alterations (ie. when audio interpipesink's stream-sync=1)... So it is strange why one mic is handling the stream-sync=0 better than the other.

I've tried adding audiorate, and rate caps in my audio source pipeline, without much luck.

[jcaballeros]

Hi @vixp24,

stream-sync=compensate-ts adjusts the incoming buffer timestamp taking into account the interpipesrc pipe basetime (the clock time when the pipe went to the play state), while stream-sync=restart-ts discards the incoming buffer timestamp and re-timestamps based only on the interpipesrc timestamp, then using both in the interpipesrcs pipeline will likely lead to sync issues.

I'd suggest trying with both interpipesrc's to passthrough-ts, or both to compensate-ts, given that the two interpipesink pipes share the same clock.

By the way, I've not been able to reproduce the audio issue with stream-sync=restart-ts, is this issue noticeable in the first few seconds of a recording?

[jcaballeros]

Hi @vixp24,

There is an alsasrc property called provide-clock that is in true by default and may be causing problems in your case, I'd try setting it to false.

Also, try tunning the alsasrc properties buffer-time and latency-time.

[jcaballeros]

Closing issue. Feel free to reopen if necessary.