search

danielpaulus / quicktime_video_hack

Record iOS device audio and video
MIT License
520 stars 110 forks source link

Improve qvh by adding a more stable video integration either using mkv + tcp or ffmpeg integration #147

Open danielpaulus opened 1 year ago

danielpaulus commented 1 year ago

QVH is a rock solid tool, except for the flaky gstreamer integration. I can see two options to improve this:

  1. Integrate with ffmpeg
  2. Implement the mkv container format and export a stream via tcp so any media framework can be integrated easily

Some pointers: We get a series of CMSampleBuf objects from the device (https://github.com/danielpaulus/quicktime_video_hack/blob/main/screencapture/coremedia/cmsamplebuf.go) which I decoded to a Golang struct. Those contain either raw h264 NaLus or PCM audio. Sometimes with a format descriptor that contains PPS Nalus needed to start a stream.

To understand how to work with this, look at the current implementations of CmSampleBufConsumer (https://github.com/danielpaulus/quicktime_video_hack/blob/d81396e2e7758d98c2a594853b64f98b54a8a871/screencapture/interfaces.go#L6)

To solve this, we either need a MKVConsumer, that receives the above CMSampleBufs and turns them into an MKV stream or a FFMPEGConsumer that can use FFMPEG directly somehow to turn CMSampleBufs into a useful format.

If we have an MKVConsumer, we/me can hook it up easily to a local tCP server like f.ex. here: screencapture/gstadapter/tcp_server_adapter.go and stream it really fast to ffmpeg or gstreamer running in external processes which make everything much more stable.

nnnpa31 commented 1 year ago

Have you seen gomedia? It can easily wrap NALU and PCM data into fmp4. It is possible to stream fmp4 using an external player, only a web server needs to be provided (or a socket connection would work, I'm not sure).

I can help with this work if I can :) I have previous experience working on wrapping an h264 raw stream to fmp4 and mp4, it works fine and can be played using ffplay.

danielpaulus commented 1 year ago

I have not tried it no, do you have some example code you can share? @nnnpa31 If I know how to use nalus and pcm samples, I can probably do the rest pretty easily

nnnpa31 commented 1 year ago

Here's an example of using gomedia to mux h264 streams to mp4 files: https://github.com/yapingcat/gomedia/blob/main/example/example_mux_mp4_memory_io.go I'm currently working with it, just replacing the mp4 in it with fmp4 and overriding the Write function of the memory io.

danielpaulus commented 1 year ago

@nnnpa31 I checked the code and it does not support raw PCM 16 bit / wave audio.

nnnpa31 commented 1 year ago

It looks like it was my amateurism that led to the misunderstanding of the content. It seems to me that everything is back to square one and I have to reconsider the option of enabling ffmpeg instead of a pure go implementation.