multi monitor + multi audio device reconnection

[EzraEllette]

---

name: device reconnection about: submit changes to the project title: "[pr] " labels: '' assignees: ''

---

description

This PR adds functionality to handle disconnection and reconnection of recording devices.

related issue: #694

type of change

• [ ] bug fix
• [x] new feature
• [ ] breaking change
• [ ] documentation update

how to test

• Audio
  1. Start screenpipe, selecting an audio device that is not plugged in.
  2. Plug the device in once screenpipe has started.
  3. Test Transcription.
  4. Disconnect device.
  5. Reconnect device.
  6. Test Transcription again.
• Screen
  1. Start screenpipe, selecting a monitor that is not connected.
  2. Repeat the same steps as audio, just with your monitor.

if relevant add screenshots or screen captures to prove that this PR works to save us time.

checklist

• [x] MOST IMPORTANT: this PR will require less than 30 min to review, merge, and release to production and not crash in the hand of thousands of users
• [x] i have read the CONTRIBUTING.md file
• [x] i have updated the documentation if necessary
• [x] my changes generate no new warnings
• [x] i have added tests that prove my fix is effective or that my feature works

additional notes

any other relevant information about the pr.

/claim #694

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
screenpipe	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Nov 19, 2024 10:50pm

[EzraEllette]

...
2024-11-19T06:06:46.673975Z  INFO screenpipe_audio::core: starting continuous recording for MacBook Pro Microphone (input) (10s segments)
...
2024-11-19T06:06:46.713398Z  WARN screenpipe_server::core: Audio device not found: Zone Wired    
...
2024-11-19T06:06:55.809971Z  INFO screenpipe_audio::core: starting continuous recording for Zone Wired (input) (10s segments)    
...
2024-11-19T06:07:10.117651Z  INFO screenpipe_server::core: device Zone Wired (input) received transcription Some(" Yeah, so I'm just going to subscribe one more time. I really don't even need to see if it listens to me. I just want to check some stuff.\n")    
...
2024-11-19T06:07:24.620032Z  WARN screenpipe_audio::core: audio device Zone Wired (input) disconnected. stopping recording.
...
2024-11-19T06:07:53.691219Z  INFO screenpipe_audio::core: starting continuous recording for Zone Wired (input) (10s segments)
...  
2024-11-19T06:08:07.237800Z  INFO screenpipe_server::core: device Zone Wired (input) received transcription Some(" Alright, let's see what's going on. Am I recording\n")
...
2024-11-19T06:08:16.727032Z  WARN screenpipe_audio::core: audio device Zone Wired (input) disconnected. stopping recording.    

[louis030195]

idk how to test, basically transcription does not work with my Sony WH-1000XM4 headphone

i tried:

• use the input and output

showing "preparing segment" but no transcription for some reason

few problems i noticed on the way unrelated to this PR:

iphone recording but does not work until you unlock the iphone

when selecting headphone as audio input it will disable noise cancellation (Sony WH-1000XM4) also had this on airpod

[EzraEllette]

I will fix the logging. It's likely that you didn't meet the speech ratio threshold, but I will make sure.

[EzraEllette]

Ok I'm going to test with AirPods in about an hour

[EzraEllette]

2024-11-19T22:51:25.105371Z  INFO screenpipe_audio::stt: device: iPhone Microphone (input), speech ratio: 0.9402985, min_speech_ratio: 0.2, audio_frames: 107200, speech_frames: 63    
2024-11-19T22:51:26.407882Z  INFO screenpipe_audio::multilingual: detected language: "en"    
2024-11-19T22:51:28.257151Z  INFO screenpipe_audio::whisper:   0.0s-0.0s:     
2024-11-19T22:51:28.257377Z  INFO screenpipe_audio::whisper:   0.0s-6.6s:  It seems like somebody has to update their permissions because my iPhone does not need me to unlock it before I record.    
2024-11-19T22:51:28.470457Z  INFO screenpipe_server::core: device iPhone Microphone (input) received transcription Some(" It seems like somebody has to update their permissions because my iPhone does not need me to unlock it before I record.\n")   

[EzraEllette]

I am unable to replicate any of the issues correlated with this pr. I don't think they are related to this PR. I updates logging to show speech frames and ratio. I also removed logging for preparing segments and resampling.