[FR] Slow Video Playback

[SiftingSands]

At first, I tried loading in 500+ videos with some videos over 1GB, and video playback from the webapp was unresponsive. Then I rolled back to a few (10-20) short duration videos; 1.5 minute, 50MB, mp4 & h264 codec, 1080p. However, it still takes a significant amount of time to buffer a video, but it gets a bit better if I leave the webapp alone for several minutes.

I'm not really noticing much resource utilization on the host machine, so I wanted to ask if there was any advice on improving video playback response. I did see https://github.com/voxel51/fiftyone/pull/1134#pullrequestreview-723891149, which is conflicting with my experience. Maybe the disparity is just a different video dataset?

Thanks!

[brimoor]

Are you running the latest version of FiftyOne? fiftyone --version. The latest is 0.14.0. You can upgarde via

pip install --upgrade fiftyone

Also, do your videos have frame-level labels or is this just an unlabeled video datasets?

[SiftingSands]

Yes, I'm running the latest version of FiftyOne (source build). I was just testing unlabeled video datasets at first.

EDIT: Just for clarity, here's the very simple session script that I'm using

import fiftyone as fo
temp_dir = <placeholder>
dataset = fo.Dataset.from_videos_dir(temp_dir)

session = fo.launch_app(dataset, port=13337, remote=True)
session.wait()

[jasoncorso]

Hi @SiftingSands:
@benjaminpkane may be able to provide some more help here. But, some very mundane questions relate to the core way videos are played in FiftyOne which just uses the browser's HTML5 video playing functionality, which also relies on the server's ability to quickly send chunks of the video at a time. So, important information here is if your network connection is fast enough to transfer such large videos quickly (I noticed you have remote=True). AFAIK, the app will invoke an HTML5 video player for each video in the scrolling pane, which can be expensive for a large set of remote videos. We can think about ways of reducing that (by, for example, limiting about of prebuffering in the HTML5 video player; I forget the exact amount, but I know there is a generous default prebuffer).

[SiftingSands]

I think that my network connection is decently fast between the client and server machines. I remember using CVAT on a similar network configuration (different server but same network), and video playback/scrubbing was much faster. However, I understand that FiftyOne has additional complexities. I can experiment with running everything locally to isolate that variable, and I'll report my findings tomorrow.

[benjaminpkane]

I'll test with a long video and report back. 0.13.3 may be faster, in which case I have a regression to fix.

There are two data streams, the native video stream and the label stream, which can both block playback. How much or little the HTML5 <video> element buffers is outside our control, save for the preload attribute which we set to metadata. But the buffering of label data may need to be improved. Currently unlabeled videos still have a (empty) label stream, which is unnecessary.

[benjaminpkane]

Ok, yes. I believe I'm seeing the same slowness. Working through it.

[SiftingSands]

My findings this morning on ~10 videos:

1. Chrome (Version 94.0.4606.81 (Official Build) (64-bit)) is much faster than Firefox (93.0 (64-bit)) at video playback
2. Having everything ran locally (no port forwarding through SSH), is about 3x faster at video buffering (eyeballing it here)