search

johanholmerin / browsercast

Cast local files to Chromecast directly from the browser
https://johanholmerin.github.io/browsercast/
MIT License
5 stars 4 forks source link

MediaStream support #3

Open ThaUnknown opened 3 years ago

ThaUnknown commented 3 years ago

Right now this implementation only works with formats supported natively by chromecast, would it be possible to achieve the same behavior with MediaStreams?

This could for example allow us to easily convert say: HEVC videos to MediaStreams using canvases allowing for wider codec support.

Additionally you could burn-in advanced subtitle formats like SSA on the browser side, and cast those to the chromecast. To my knowledge there aren't any solutions which allow SSA support on chromecasts without transcoding the entire video so this would be a first.

johanholmerin commented 3 years ago

No, not without a complete change of how Browsercast works, and a major increase in complexity. However, if those are your specific use cases, there might be other solutions:

  1. Newer Chromecasts support HEVC natively. For those that don't, converting on the sender wouldn't work since Chrome doesn't support HEVC.

  2. I'm not familiar with SSA, but a much more efficient solution than re-encoding the entire video would be to render them on the receiver in real-time.

    Would that work for you?

ThaUnknown commented 3 years ago

No, not without a complete change of how Browsercast works, and a major increase in complexity. However, if those are your specific use cases, there might be other solutions:

  1. Newer Chromecasts support HEVC natively. For those that don't, converting on the sender wouldn't work since Chrome doesn't support HEVC.
  2. I'm not familiar with SSA, but a much more efficient solution than re-encoding the entire video would be to render them on the receiver in real-time.

Would that work for you?

yeah, I'm aware of the re-write, it most likely would also change how it functions a bit, new edge supports HEVC so that would be a plus, HEVC was just an example, I think the full list is 3g2, 3gp, m2ts, m4v, mov, mkv, av1, vp9, theora

the thing about SSA, is it's an insanely complex format, it can do as much as for example CSS3, it supports 3d positioning, translations, transitions, any font etc, which means it's immensely resource intensive to render it real-time in the browser. I've already re-wrote most of the JS on the library you linked to make it perform MUCH, MUCH better, but it's still nowhere close to be runnable on chromecast, which is why I run it on the client, so I don't have to do any transcoding, and still have flawless streaming support

johanholmerin commented 3 years ago

In that case I would recommend creating a new project - Browsercast works for what it does, there is no need to change it. Using MediaStream instead would be such a major change that it would be easier to start from scratch anyway. But it's not something I'm interested in doing.

johanholmerin commented 3 years ago

I continued thinking about this, and decided to create a prototype that supports MediaStreams. WebRTC has built-in support for MediaStreams, but that is for live video-call, and is unsuitable for other kinds of video, where quality and not missing content is a priority. Instead I used MediaRecorder to capture the output and then stream that as ArrayBuffers to the receiver, which uses Media Source Extensions to display it. The prototype uses a plain video element for decoding, but you could try replacing it with a canvas element. Alternatively, if you want to add support for codecs that isn't available in the browser, it should be possible to replace the MediaStream/MediaRecorder with ffmpeg.wasm.

The code is available in the mse branch and the app is available at https://browsercast-beta.vercel.app/. Let me know if this would work for you. If it does I can look at polishing it up and we could discuss what transcoders/renderers you would be interested in.

ThaUnknown commented 3 years ago

I continued thinking about this, and decided to create a prototype that supports MediaStreams. WebRTC has built-in support for MediaStreams, but that is for live video-call, and is unsuitable for other kinds of video, where quality and not missing content is a priority. Instead I used MediaRecorder to capture the output and then stream that as ArrayBuffers to the receiver, which uses Media Source Extensions to display it. The prototype uses a plain video element for decoding, but you could try replacing it with a canvas element. Alternatively, if you want to add support for codecs that isn't available in the browser, it should be possible to replace the MediaStream/MediaRecorder with ffmpeg.wasm.

The code is available in the mse branch and the app is available at https://browsercast-beta.vercel.app/. Let me know if this would work for you. If it does I can look at polishing it up and we could discuss what transcoders/renderers you would be interested in.

Yeah, I already solved this, but decided to opt for the solution which used MediaStreams instead of sending raw data chunks, solely because of it's simplicity, the quality drop is unfortunate in this situation indeed, but I couldn't come up with anything that was performant enough, this already has a few issues, you can see my implementation here: https://github.com/ThaUnknown/webtorrent-player/commit/b3124c54b5fc200500798952a438abea3e8b3e06#diff-e727e4bdf3657fd1d798edcd6b099d6e092f8573cba266154583a746bba0f346 the main reason I used this was again performance, I'm already running a P2P torrent client, http server, 3d subtitle renderer and now a transcoder, all inside a single browser window, so performance is quickly becoming an issue and I need to cut corners, but your implementation is probably miles better for quality.

ThaUnknown commented 3 years ago

adding a playback delay seems to have marginally improved the quality of the stream, but it's still not too great, it goes from a RAW to something you'd see on netflix, additionally the UX is kinda poor