Feature Request: Serve up a stream with a mosaic of streams

[yuuzhan]

Use Case: Serving up multiple cameras to devices that can only accept one stream at a time (My use case is Kodi powering my TV).

Ive done some digging and found that ffmpeg has the ability to do this:

ffmpeg -rtsp_transport tcp \
    -i rtsp://smartiptv:PASSWORD@192.168.2.3:6911/Streaming/Channels/101 \
    -i rtsp://smartiptv:PASSWORD@192.168.2.3:6911/Streaming/Channels/201 \
    -i rtsp://smartiptv:PASSWORD@192.168.2.3:6911/Streaming/Channels/301 \
    -i rtsp://smartiptv:PASSWORD@192.168.2.3:6911/Streaming/Channels/401 \
    -filter_complex "
        nullsrc=size=1920x1080 [base];
        [0:v] setpts=PTS-STARTPTS, scale=960x540 [upperleft];
        [1:v] setpts=PTS-STARTPTS, scale=960x540 [upperright];
        [2:v] setpts=PTS-STARTPTS, scale=960x540 [lowerleft];
        [3:v] setpts=PTS-STARTPTS, scale=960x540 [lowerright];
        [base][upperleft] overlay=shortest=1 [tmp1];
        [tmp1][upperright] overlay=shortest=1:x=960 [tmp2];
        [tmp2][lowerleft] overlay=shortest=1:y=540 [tmp3];
        [tmp3][lowerright] overlay=shortest=1:x=960:y=540
    " \
    -c:v libx264 -preset superfast -crf 18 -f matroska - | ffplay -

Source: https://gist.github.com/mikaelz/12b394334e3226d67a37399d40751d8f which is based on https://trac.ffmpeg.org/wiki/Create%20a%20mosaic%20out%20of%20several%20input%20videos

a "gold plated" part of the request would also be an option to configure recording of the mosaic.

Thank you for all the hard work you have done on this project!

[mrlt8]

Will look into this! Could potentially be done in rtsp-simple-server, but it will probably eat tons of processing power.

[yuuzhan]

Thank you! And yes, I would guess that it would take quite a bit of CPU - but as long as the feature comes with a warning I dont know of any way to avoid that :P

I suppose a mosaic recording could be scheduled for a specific time at night when the processing power is not needed as much for live viewings and such.

[mrlt8]

Hey @yuuzhan, does something like this work for you? (replace all the cam1, cam2 , etc)

- RTSP_PATHS_MULTI_RUNONINIT=ffmpeg -rtsp_transport tcp -thread_queue_size 512 -r 15 -i rtsp://0.0.0.0:8554/cam1 -rtsp_transport tcp -thread_queue_size 512 -r 10 -i rtsp://0.0.0.0:8554/cam2 -rtsp_transport tcp -thread_queue_size 512 -r 10 -i rtsp://0.0.0.0:8554/cam3 -rtsp_transport tcp -thread_queue_size 512 -r 15-i rtsp://0.0.0.0:8554/cam4 -filter_complex "[0:v][1:v][2:v][3:v]xstack=inputs=4:layout=0_0|w0_0|0_h0|w0_h0,format=yuv420p,scale=1920:-1[v]" -map "[v]" -c:v libx264 -preset ultrafast -b:v 5000k -maxrate 5000k -bufsize 10000k -force_key_frames expr:gte(t,n_forced*2) -f rtsp rtsp://0.0.0.0:8554/multi
- RTSP_PATHS_MULTI_RUNONINITRESTART=yes

You should then be able to access the mosaic on /multi.

[yuuzhan]

Thank you for that @mrlt8 ! There was a typo in your first line ( 15-i rtsp://0.0.0.0:8554/cam4 needed to have a space after the 15) but I was able to figure that out by examining the logs. the RSTP Stream can be viewed.

After a few minutes two of the streams seem to have frozen (time stamp on them is not moving). Those two streams did seem to have a time stamp about 1 minute behind the other two to start with and it got worse until they kind of stopped.

When viewing the offending streams individually they seem to be working with the usual lag.

Edit: I did a grep on the logs for multi with the following results:

[h264 @ 0x55e63c14f480] error while decoding MB 30 31, bytestream -326966 drop=0 speed=1.33x
[h264 @ 0x55e63c14f480] concealing 4459 DC, 4459 AC, 4459 MV errors in I frame
[h264 @ 0x55e63af76e00] error while decoding MB 23 20, bytestream -366984 drop=0 speed=1.33x
[h264 @ 0x55e63af76e00] concealing 5786 DC, 5786 AC, 5786 MV errors in I frame
rtsp://0.0.0.0:8554/backdoor: corrupt decoded frame in stream 0
rtsp://0.0.0.0:8554/basement: corrupt decoded frame in stream 0A dup=6992 drop=0 speed=1.33x
[h264 @ 0x55e63c2ee040] concealing 1966 DC, 1966 AC, 1966 MV errors in I frame=0 speed=1.33x
[h264 @ 0x55e63afddc00] error while decoding MB 107 38, bytestream -48243 drop=0 speed=1.33x
[h264 @ 0x55e63afddc00] concealing 3542 DC, 3542 AC, 3542 MV errors in I frame
More than 10000 frames duplicated/A time=00:15:08.26 bitrate=N/A dup=9995 drop=0 speed=1.33x

more extensive logs here: https://pastebin.com/F3djdtiD