Synchronized playback over a network

[jdb-vsn]

Description source: http://www.mplayerhq.hu/DOCS/HTML/en/networksync.html

Multiple instances of MPlayer can synchronize playback over a network. This is useful for creating "video walls" with multiple screens controlled by different computers. Each MPlayer instance can play a different video, but they all will try to stay at the same time offset in the file. It is recommended but not necessary to encode the video files using the same codec and parameters.

The relevant options are -udp-master, -udp-slave, -udp-ip, -udp-port, and -udp-seek-threshold.

If -udp-master is given, MPlayer sends a datagram to -udp-ip (default: 127.0.0.1) on -udp-port (default: 23867) just before playing each frame. The datagram indicates the master's position in the file. If -udp-slave is given, MPlayer listens on -udp-ip/-udp-port and matches the master's position. Setting -udp-ip to the master's broadcast address allows multiple slaves having the same broadcast address to sync to the master. Note that this feature assumes an ethernet-like low-latency network connection. Your mileage may vary on high latency networks.

Mentioned on issue #898.

Also on https://github.com/mpv-player/mpv/issues/252#issuecomment-24864050.

[ghost]

Porting this back to mpv shouldn't be too hard. But I don't like the way mplayer integrates this (it looks like it blocks the player on network to achieve sync), so it should be done differently. The branch udp_sync contains an earlier attempt.

[nixternal]

I believe this is the only thing blocking the adoption of mvp over mplayer for a client project that relies heavily on mplayers udp sync stuff. I wish I had the time & knowledge to help here because I can honestly say mpv > * as a media player. I now use it over VLC.

[ghost]

The most annoying thing is actually that it'd require raw network programming, which is painful/unportable if it should work on windows.

[artwait]

I'm using the udp sync feature in mplayer right now for a video wall, but I'm running multiple slaves on a single machine (rather than across a network). The challenge with this is that the UDP port is blocked by the first slave, thus subsequent slaves are not able to listen. I've got a hacky work-around to this, but it would be nice to also have a solution that uses a non-network approach for sending timestamps between instances on a single machine. Just a thought.

[ghost]

it would be nice to also have a solution that uses a non-network approach for sending timestamps between instances on a single machine.

Do you have a specific mechanism in mind?

[lionep]

:+1:

[mgcrea]

:+1:

Would leveraging an external dependency like http://zeromq.org make it easier? Otherwise, I would be fine with a linux-only support.

[gcoupelant]

:+1: This would really be awesome.

[smarek]

Hey guys, any progress on this feature please?

[ghost]

Nobody is working on it. It could now be implemented without mpv changes via one of the following mechanisms:

• as Lua script
• as a cplugin
• as a wrapper application using libmpv

[smarek]

@wm4 cool, so any network synchronized protocol and wrapper around mpv should suffice? Maybe using --input-ipc-server and any daemon that converts network issued command into mpv json command (or just dumb pass-through) could work.

Also I've found this, but theres no documentation. https://github.com/derlaft/synco Anybody tried to use that?

[rossy]

@smarek You could also look at http://syncplay.pl/

I don't know if it's accurate enough to use as a video wall (like MPlayer's UDP-sync apparently was,) but it definitely can sync multiple mpv instances over a network.

[Collocalini]

I've recently did just what Smarek offered - used --input-ipc-server mode to synchronize mpv and osmcplayer from kodi on raspberry pi. Supports play/pause detection. I used it mostly to show video from raspberry pi but take audio from a better audio card on a separate computer. Not perfect but I find it watchable most of the time. mpvsync.zip

[bitbyt3r]

I've been trying to do this through libmpv and have had some success, however it isn't quite frame perfect. I'm using a loop similar to this: `

 mpv_observe_property(player0, 0, "time-pos", MPV_FORMAT_DOUBLE);

 while (1) {
    mpv_event *event = mpv_wait_event(ctx[0], 1000);

    if (event->event_id == MPV_EVENT_PROPERTY_CHANGE) {
        mpv_event_property *prop = (mpv_event_property *)event->data;
        if (strcmp(prop->name, "time-pos") == 0) {
            double pos = *(double*)prop->data;
            for (int i=1; i<instances; i++) {
                mpv_set_property(player1, "time-pos", MPV_FORMAT_DOUBLE, &pos);
            }
        }
    }

    if (event->event_id == MPV_EVENT_SHUTDOWN)
        break;
}

`

Basically I subscribe to changes of the "time-pos" property, then set the time-pos of every slave player instance. This gets really close to what I want, but sometimes doubles a frame or skips a single frame and I'll end up off by one for a bit before returning to the correct timing. Is there a better method that I could use here? The time-pos property appears to update once per frame, so it is a bit course for this.

[ghost]

A better method would be adjusting playback speed for small deviations. Also "time-pos" is somewhat asynchronously reported, so it can definitely happen that it's off by a frame or more.

[bitbyt3r]

Ok, I can try that and see how it works out. I saw in issue #5592 that there may be a delay in adjusting the playback speed. I could probably work around that, but is that normal behavior?

I'm also working on code to sync the playback time across the network accounting for network delay and some amount of jitter. I'm writing all this as a separate executable that calls libmpv, but I could try to merge it into mpv itself. Would that be something you'd be interested in merging, or should I keep it separate?

Thanks for the help!

[hansinator]

Hi, I need network sync support like in mplayer but with the ability sync the playlist position, i.e. the slaves should follow the master if the master plays another video in the playlist. I am currently using mplayer, but I would need to modify the mplayer network sync anyway (and make it incompatible to the upstream version), so I've decided it'd be a good opportunity to implement this whole thing in mpv. I am not convinced by a separate executable, plugin or script. I'd like to add support directly into mpv so that the feature is readily available via command line and for library users.

Do you have any suggestions or pointers into the right direction before I start? I have read that the blocking socket read behaviour on the slave side is undesired. Does anyone have an idea what a better solution might look like? However, I believe this is an implementation detail that can be changed at any time without affecting the network packet format or user experience. I've got a couple of days time at my disposal and a video wall with multiple slaves to test the whole thing.

Edit: I've taken a closer look at the mpv code and did some tests with the existing udp_sync branch (rebased onto the current master). First I've measured the latency using a master and separate slave with identical hardware on a local network and a video that shows the played frame number without audio at 60FPS. Recording the playback with a high-speed camera shows that mplayer has a constant lag of about 1 frame. mpv has a lag of 1-2 frames. That's better than I expected from code using events and separate threads.

Now I could be satisfied, but I see a problem when the slave gets slightly out of sync. Mplayer has a pretty elegant solution to this using the blocking udp read. It is not as stupid as it might look first. The video playback loop essentially ceases to use it's own timing and relies on the operating system to block the thread until a UDP packet arrives. This ensures that the slave does not play the video faster then the master. It also has a very low latency, lower than propagating events via threads. If the slave lags behind the master then UDP packets will queue up, so the code won't block and the slave playback loop iterates as fast as the UDP queue can be emptied while still decoding video frames. This will make the slave playback speed faster to catch up. So essentially the way the network code is implemented (with all its ugly blocking ) automagically does the playback speed adjusting that wm4 suggested.

The question now is, how to implement that in mpv. I fear it'll be a lot more complex and messy when not using a blocking socket.

Edit2: One thing about the playback speed adjustment that makes me uncomfortable is that when the video FPS matches the output FPS then faster playback is just dropping frames unless you can vary the output FPS quickly. Slowing down by blocking in turn will make one video frame be shown for multiple output frames and then just continue normally. It all results in stuttering motion which would be quite noticeable depending on the content.

Edit3: @bitbyt3r have you had a look at PTP / IEEE1588 for syncing time while accounting for network delays? The linuxptp implementation has lots of code doing this with µs accuracy. One could use linuxptp to sync the time of multiple playback devices in a local network and then synchronized playback is as easy as the master transmitting the exact wall-clock time it plans to show the next frame. The slaves would just need to sleep until that moment and show the frame. Works well with software ptp for video applications, i.e. no phy/mac support of hardware timers necessary. The existing UDP sync with it's 1-2 frame latency however is more than good enough (given it does not seek) unless you have one continuous display surface where e.g. the left and right parts are connected to different playback devices, in which case frame-perfect sync is a necessity. In any case, it seems to me like implementing accounting for network delays yourself is overkill. If you don't want to sync the system clock, then ptp can just provide synced software/hardware timers. There is this phc2sys daemon which syncs the system clock to the hardware/software timers complete with servos for syncing gradually in small steps (no big step-change). It is a perfect starting point: in the code paths where it actually adjusts the system clock time, subtract an offset from the time and just call your mpv_set_property(player1, "time-pos", MPV_FORMAT_DOUBLE, &pos); instead and it is done on the slave side.

[nixternal]

I built mpv with the network sync stuff, but I am clueless on how to use it properly. Any help there will be greatly appreciated. Also, anyone done any further work since last year? mplayer is great and all, but when utilizing Intel graphics & libvdpau-va-gl anything other than h264 is a no go.

[Collocalini]

2nixternal: use { "command": ["get_property", "speed"], "request_id": "asc" } to get playback speed { "command": ["get_property", "time-pos"], "request_id": "gtp" } to get current playback position

The request_id could be arbitrary.

{ "command": ["set_property", "speed", 1.0]} to set speed { "command": ["set_property", "time-pos", 0.0]} to set playback position

Commands should be sent through a socket. Run mpv like this: mpv --input-ipc-server=/tmp/mpvsocket file.mp4

for more see section JSON IPC from https://mpv.io/manual/master/

[EternityForest]

Regular NTP can be pretty darn good on a wired network. A really standard, interoperable way could be to just use the system time. It's probably never going to be worse than your ping time, and WiFi almost always let's a few 2ms or so packets through. Daemons like Chrony have this stuff all figured out, why reinvent? Even Windows can apparently do 1ms sync?

Using a --wallclock-start=TIMESTAMP, you could tell MPV that you want to play synced to wallclock, with the start of the video at exactly that time. If the command needs time to process and it can't hit the time you specified, or it's in the future, you skip or delay as needed so that the current position is always what it would be if you started exact then.

With 1 second of video locked to 1 second of real time, everything is perfect as long as they get the same time. You can even start them at different times. No communication is actually needed here, you could just have GPS.

Network issues will not cause stuttering as long as they are brief enough that the clocks stat synced.

This would be troublesome manually, but from there it's easy to add a tiny bit of automation with 2 really simple packets. The first one gets a --sync-group=name argument, and the second one gets a --sync-join argument.

The second sends a multicast packet asking for the group master's start time, putting the name in the UDP. Once it gets that, no communication is needed.

[pyfave]

There is a big need for this and osc frame search.

the omxplayer-sync approach is quite good, but omxplayer is deprecated, and standards exist : midi time code from jack or others , OSC protocol on a local network ,

[guillaumekh]

To anyone considering giving this a go: don't forget about buffering & latencies introduced by the underlying layers.

I'm not familiar w/ video renderers, but all audio backends (e.g. PulseAudio, Pipewire, Core Audio, etc) will introduce some latency, and even a tiny amount of drift can lead to problems. For example, a drift of just a 10ms between 2 speakers playing concurrently will produce a noticeable "murky" effect. It's easy to reproduce on one's system. Just pick some track, downmix it to mono, upmix it back to stereo, then add a delay to one channel.

And we haven't even mentioned phase issues.

[Frenzie]

I suspect most users used mplayer -nosound -udp-slave etc to prevent some minor cycle waste regardless of potential drift. But ftr I think you're overestimating UDP latency by at least an order of magnitude.

[nixternal]

Yeah, I have machines in the field that have been running for years with mplayer -udp-master & -udp-slave, all running sound. Only time we have ever noticed any sort of drift is poorly made videos. When we keyint=1 so every frame is keyed, the only way you would detect any drift is with a picture or record a video & go frame-by-frame.

[guillaumekh]

@nixternal that's interesting. Do you do anything to control audio backend latency — i.e. anything that would ensure that the audio backend on each machine running mpv is within the same tight range? Come to think of it, using identical machines running same OS would probably qualify.

@Frenzie i wasn't really commenting on UDP latency. My point was only aimed at the latency that audio backends introduce — e.g. PulseAudio can easily introduce >100ms latency since low latency was never one of its design goals.

[nixternal]

@guillaumekh we don't do anything at all in regards to audio latency. Our machines are built upon the Intel NUC so there is a chance they are all the same, but usually just similar on the rental side. All running some LTS version of Ubuntu. mPlayer isn't bad, but with Intel you can't do h.265 or anything bigger than 4k. The mplayer-vaapi project is long gone & dead, and the whole vdpau-vaapi stuff is a mess. We are now using Pulse as our backend, but used Alsa prior to using 18.04 iirc.

[guillaumekh]

Thanks for taking the time to share this.

I'm still mulling this over but maybe the factor here is that in most environments, sound will take many different paths from speakers to listeners, each with its own small delay, which would help hiding sync drift between sources.

Anyway, I came to this problem with some knowledge of how this specific subject of audio sync is often handled these days in professional audio systems (e.g. concert venues, malls, hotels, stadiums, etc). See Dante/AES67/SMPTE2110, all of which use PTP to ensure precise sync. It's quite interesting to hear you're getting away without all this (expensive) stuff :)

[nixternal]

Yeah, we looked at all of the Dante stuff as well, and it still confuses me about as much as DMX does, but they UDP sync with mplayer is so good, when you keyint=1 every frame. If you don't, yeah there "could" be a little noticeable drift.

ffmpeg -i input.mp4 -c:v libx264 -x264opts keyint=1 -crf 15 -c:a copy output.mp4

We run this quick little script on every video & they have been flawless. Like I tell my clients, garbage in, garbage out which has been the case.