search

RenderHeads / UnityPlugin-AVProVideo

AVPro Video is a multi-platform Unity plugin for advanced video playback
https://www.renderheads.com/products/avpro-video/
238 stars 28 forks source link

IRandomAccessStream support for non-Windows #1408

Closed drew-512 closed 1 year ago

drew-512 commented 1 year ago

Hi Morris and friends,

Our system here features an all-platform Go-based runtime that pushes state to our Unity runtime (that embeds the Go runtime). This scheme offers many powerful advantages, one of which is that we can pipe all kinds of data and AV to the Unity runtime without having to manage slow or unnecessary C# code.

This means that we already have a random access pipe open and ready to stream an AV stream, making it ideal to build an implementation of IRandomAccessStream around (or something similar). The goal is to use the data transport that is already established (rather than burn overhead on additional network connections). The (painful) alternative is the Go runtime hosts a local http port that serves assets and when it's time to play an AV asset, the Go runtime sends a (local) URL to the Unity side, and then we tell AVPro to play that.

Perhaps we make a lean and mean knock-off of IRandomAccessStream? Or we could work with any random access stream interface that AVPro exposes. The goal of course being that we can hand AVPro this IRandomAccessStream-like instance and it can go to town on it. We could also stick whatever hint accessors you like on it since our system will have its MIME type and the first 1024 bytes.

Best, Drew

MorrisRH commented 1 year ago

The HTTP route would be your best bet for now as we only support reading from buffers on windows (although work is being done on bringing this over to Android).

Reading from arbitrary buffers/streams on Apple platforms is not supported and wouldn't be trivial to implement. Our own internal research suggested the simplest route would be to embed an HTTP server for local playback of HLS streams or mp4's where the download/buffering was handled externally to the OS.

drew-512 commented 1 year ago

That makes sense.

I forgot about how limiting Apple can be with their codecs and AV processing. In making Tunr many moons ago now, there was much pain and workarounds with audio playback on iOS. Sad that DRM's legacy basically amounts to making AV devs lives painful and high friction. At least the lawyers made out?

Ok, so it sounds like we should go all-in on a localhost streaming from our kernel.

Looking forward to showing y'all this internal prototype once we get through these key decisions.