bedeho commented 2 years ago

Background

Having distinct resolutions for the same video is very useful, in particular to support different screen sizes or levels of connectivity across users. We will also need this as part of youtube-synch, where Youtube offers various levels of encoding of the same media.

Proposal

Consumer

Baseline

Basically allow the viewer to both see the resolution of the media currently being viewed, but also discover whether others exist, and the opportunity to change during play.

Stretch

Taking this one step further, it would be useful if the application could have a built in propensity to select a resolution best aligned with current level of connectivity, and switch between versions automatically. This could also be done to lower latency on starting playback of new videos, and is a very standard pattern. I would consider everything in this bucket as a stretch goal, and if it is not decided as in scope, just create some new fresh issues capturing this as possible enhancements for the future.

Studio

Allow a publisher to provide more than one media file, both during original publishing, or at a later time. Great the product can detect resolution information from file, so user does not need to specify manually. The ability to have more than one resolution should not be a major choice, instead something a creator can choose to engage with if they need it.

In the future Orion likely will have a transcoding integration, but that may be quite far ahead, and even then it may still be sensible to allow user to specify their own pretranscoded media.

TODO

I think the query node already supports all the required metadata representations, so the work here is really in doing some design on the studio side, perhaps a bit on the consumer side, and then mostly application engineering work.

Timeline

I don't believe this feature is a precondition for launching youtube-synch, or even mainnet, but it seems like a sensible feature enhancement, and some of the exploratory engineering RnD can be done as a filler task at any time.

kdembler commented 2 years ago

I'm not sure if this is worth doing separately from trans-coding (or trans-sizing, but on server-side).

I've already did a bit of research in this area. There are widely-used streaming protocols that support adaptive streaming (mostly HLS and MPEG-DASH). Both of those define primitives for defining multiple video tracks, have libraries supporting what you described as automatic switching and many more. However, based on my research, creating manifests (file that describe content/sources that these protocols are based on) will not be possible client-side, without some kind of video parsing. That means if we wanted to do manual trans-sizing on client-side, we would need to do our own custom implementation, which would become obsolete once we adopt some standard.

I'm also not sure how valuable this is to creators if they need to do it manually and if they would be willing to actually use this. I think at the moment, as far as creators are concerned, most trouble comes from the fact that they are unsure whether the content they're uploading would be playable by their viewers.

I think the query node already supports all the required metadata representations, so the work here is really in doing some design on the studio side, perhaps a bit on the consumer side, and then mostly application engineering work.

I don't think that's true. While runtime should be agnostic, we would need to update protobufs and QN, but I don't think that would be a lot of work.

Maybe instead of doing this client-side we could figure out how to enable server-side support with minimal work? I think we could even do this separately from Orion, as a completely separate service that would later be integrated into Orion. Those are my thoughts, WDYT?