Different RTSP streams in sync

[ajimenezherrero]

Dear all,

I have built a video wall over IP using rpis. I am using omxplayer to play different RTSP streams in sync. In order to get them properly synchronized I had to modify OMXReader ConverterTime method in order to set PTS and DTS according the wallclock provided by the RTSP session, in fact, I am adding "start_time_real_time" from AVFormatContext to PTS and DTS. Do you think it could be feasible to set up a new option to omxplayer to use wallclock timestamps?

I can easily provide a patch of the ConverterTime changes I did and try to come up with a proposal of setting a flag in OMXReader in order to use "start_time_real_time" or not.

Regards

[popcornmix]

This is a very niche use case, and it would only be considered if changes are very minor and localised. I'm not convinced that what you have suggested will actually be sufficient. Each pi generates it's own media time (see OMXMediaTime) which is derived from number of audio samples played and has an accuracy based on its oscillator (~100ppm or 0.01%). You will get drift between the different Pi's (~0.36s per hour).

See: https://github.com/popcornmix/omxplayer/issues/85

[davidcassany]

I'm collegue of ajimenezherrero, agree that this is a niche use case. We synchronize the Pi's clock using PTP, as they are all in the same local network, so I guess this way we can solve the deviation you mentioned (is that correct? In addition, we are not using audio). We did a very simple and localized modification over the code, we'll post it.

85 is pretty interesting, the fact is that we managed to do it without using the "--live" option, it has been enough to use the NTP timestamp sent over RTCP from the server and avoid the substraction of AVFormatContext.start_time in PTS and DTS, this way we get the same timestamp in all Pi's with a minimum deviation (less than one frame, around 15~20ms).

Thanks!!

[popcornmix]

I don't see your scheme working without something like --live. NTP may synchronise the wall clock on the Pi's but it won't affect the audio clock which determines media time. --live uses audio resampling to manage buffer size, which will have the effect of forcing the media time to sync with the server's version of time.

[davidcassany]

@popcornmix ,

You were right, our configuration is not properly working, although it looks like many times. Applying the suggested modification helped a lot to get close to a multi device synchronization, however I form what you explained I don't see why (maybe due to the "start_time" substraction, which I understand it is only used to start at "0"). Our inital idea was that using realtime timestamps and setting some delay over them would force the video_scheduler to display the frames at a certain wallclock instant (I was assmuming OMXMediaTime was somehow related to the wallclock), however it is only related to audio timestamps (like assuming audio is the wallclack). So I understant that the video_scheduler module it only schedules video against OMXMediaTime (based on audio), in order to synchronize audio and video. Does the video_scheduler have any effect if no audio available?

I am pasting the modification of the ConvertTimestamp method we were talking about, just a simple modification to use the "start_time_realtime" timestamp from AVFormatContext in case there is a flag activated (we included this flag as an attribute to OMXReader), otherwise works as usuall.

@@ -1081,15 +1082,26 @@ double OMXReader::ConvertTimestamp(int64_t pts, int den, int num)
   // do calculations in floats as they can easily overflow otherwise
   // we don't care for having a completly exact timestamp anyway
   double timestamp = (double)pts * num  / den;
-  double starttime = 0.0f;

-  if (m_pFormatContext->start_time != (int64_t)AV_NOPTS_VALUE)
-    starttime = (double)m_pFormatContext->start_time / AV_TIME_BASE;
+  if(m_streams_sync)
+  {
+    double starttimerealtime = 0.0f;
+    if (m_pFormatContext->start_time != (int64_t)AV_NOPTS_VALUE)
+      starttimerealtime = (double)m_pFormatContext->start_time_realtime/AV_TIME_BASE;

-  if(timestamp > starttime)
-    timestamp -= starttime;
-  else if( timestamp + 0.1f > starttime )
-    timestamp = 0;
+    timestamp += starttimerealtime;
+  }
+  else
+  {
+    double starttime = 0.0f;
+    if (m_pFormatContext->start_time != (int64_t)AV_NOPTS_VALUE)
+      starttime = (double)m_pFormatContext->start_time / AV_TIME_BASE;
+
+    if(timestamp > starttime)
+      timestamp -= starttime;
+    else if( timestamp + 0.1f > starttime )
+      timestamp = 0;
+  }

   return timestamp*DVD_TIME_BASE;
 }

[popcornmix]

If there is no audio then timing falls back to using the system timer (STC) which is just a free-running 1MHz clock (again not synced to wall clock).

If there is no audio, then m_av_clock->OMXSetSpeed still adjusts the media speed, however it just adjusts the STC rate, and doesn't have to resample the audio to fit. So --live should still work.