Poor streaming performance of videos

[mzur]

I noticed that the current implementation of a streamed video response and range request support is not very efficient. It looks like both the Swift and S3 adapters download the whole object when readStream is called, so the stream becomes seekable. If they would only pass on the Guzzle HTTP stream, it would not be seekable. This means that the whole object is downloaded for each request which gets very slow for larger videos. I spent quite some time to debug this and it seems that there is no way to get a seekable stream from the filesystem adapters without having to download the whole object. The alternatives so far are:

• Use a temporary URL. This is only supported by some filesystem adapters ,e.g. S3 but not (yet) Swift. It's also hard to choose a lifetime for the temporary URL because the video annotation tool would break once the URL gets invalid. The URL also exposes the full path to the object storage. But this is really fast.

• Download the file with the file cache and create a streamed seekable response from there. This becomes impractical when really huge videos should be handled, which overflow the maximum cache size. Also the request might time out when a very large video is initially loaded into the cache.

[mzur]

The download solution could be accomplished with the file cache. The file cache can be called inside the callback function of a streamed response so the cache does not inadvertedly delete the file while it is downloaded.

[mzur]

I implemented the download solution with the file cache for now. This can't be a permanent solution, though. We probably need public/temporary URLs at some point of time.

[mzur]

Maybe a composite solution can be achieved, too. The app can create a temporary URL that is valid for, say, one hour and stores it in the cache. Whenever a new request from a video element is received, the controller authenticates the request and then proxies it (no redirect) to the temporary public URL (if it is still valid) or creates a new public URL. This way the public URL is not exposed. How to handle expiration of the public URL while the request is being processed? Maybe use a generous margin and create a temporary URL that is valid for two hours but recreate it after only one hour?

[mzur]

Alternatively it should be possible to generate temoprary public URLs for file storage adapters like S3. BIIGLE could detect if this is the case and switch from the download option to the public URL option automatically.

[mzur]

Temporary URLs might be available for Swift, too. Usage of these URLs should be configured either in the storage disk configuration (if possible) or in the configuration of this module.

[mzur]

I think temporary URLs should be the way to go, with the current download solution as fallback (or maybe the proxy solution outlined above). The URLs could be valid for longer periods of time, too (e.g. 24 h).

This should also be implemented for the Elements adapter ( #207 ).

[mzur]

Note on temporary URLs for our OpenStack instance: The instance uses radosgw which only supports SHA1 and account level keys for temporary URLs. Set the key with swift post -m "Temp-URL-Key:xxx".

[mzur]

Support for temporary URLs is now fully implemented and active at biigle.de.