Capture audio alongside video

[clearlysid]

There are 2 approaches to achieve this:

1. use a cross-platform solution like cpal or potentially even just FFmpeg.
2. use OS native APIs to capture audio (eg: ScreenCaptureKit, AVFoundation, DirectShow)m then feed audio samples to encoder.

Would love to have some inputs on which one is better. I'm currently leaning towards 2 since I don't believe there are significant performance gains to be unlocked wrt audio capture when it comes native APIs.

[clearlysid]

Affects both #9 and #10

[clearlysid]

@ZohebMOPO had made some progress on this using cpal. Added his snippet and made the playground of sorts on the branch test-cpal 🙌

[anubhavitis]

@clearlysid, check this: https://github.com/clearlysid/cypher/pull/14

[anubhavitis]

I have tested this in local, it's woking. However, I think in cypher's use case, audio and video encoding has to be done together.

[clearlysid]

There's an option to explore recording just audio with FFmpeg and encode both video and audio together, but I've not gotten FFmpeg to work with anything in cypher at all so don't really know the best way forward at the moment 🥲

We have both FFmpeg and Gstreamer as options to explore.

[1313]

Should be fairly simple to add audio capture to screencapturekit-rs There is a config option captures_audio that needs to be enabled, and then you can filter for audio CMSampleBuffers in the outputhandler.

I'll fix the configuration part.

So just match on the SCStreamOutputType where you implement the trait and pass captures_audio as true to the SCStreamConfiguration.

    struct SomeErrorHandler {}
    struct SomeOutputWrapper {
        pub audio_tx: SyncSender<CMSampleBuffer>,
        pub video_tx: SyncSender<CMSampleBuffer>,
    }
    impl StreamErrorHandler for SomeErrorHandler {
        fn on_error(&self) {}
    }
    impl StreamOutput for SomeOutputWrapper {
        fn did_output_sample_buffer(&self, sample: CMSampleBuffer, of_type: SCStreamOutputType) {
            match of_type {
                SCStreamOutputType::Screen => self.video_tx.send(sample),
                SCStreamOutputType::Audio => self.audio_tx.send(sample),
            }
            .unwrap()
        }
    }
    #[ignore]
    #[test]
    fn test_output_wrapper() {
        let mut content = SCShareableContent::current();
        let display = content.displays.pop().unwrap();
        let width = display.width;
        let height = display.height;
        let filter = SCContentFilter::new(Display(display));
        let config = SCStreamConfiguration {
            width,
            height,
            captures_audio: true,
            ..Default::default()
        };
        let (audio_tx, audio_rx) = sync_channel(1);
        let (video_tx, video_rx) = sync_channel(1);
        let mut stream = SCStream::new(filter, config, SomeErrorHandler {});
        let w = SomeOutputWrapper { video_tx, audio_tx };
        stream.add_output(w);
        stream.start_capture();
        println!("{:?}", audio_rx.recv().unwrap());
    }

[1313]

Ah need to add code for getting the audio data out of a CMSampleBuffer though, but feel free to PR that.

[clearlysid]

@anubhavitis thanks for your contribution. I can verify that the CPAL implementation is working smooth across Windows and macOS. Closing this issue for now, but will happily take another look at this if you end up making the PR mentioned above^