Spatial Microphone Toy

[technobotanist]

What would this gadget do?

This gadget would serve as the VR equivalent of a tape deck recorder toy, rendering captured sounds from underlying applications down to stereo audio snippets that represent the sound as perceived from the location of the microphone (user's hand) at the time of recording.

Reach goals could deal with snippet playback environment and the play that emerges from these 'sound pearls'. The ability to locate these pearls around a space and record that soundscape with the same microphone could be a fun form of emergent play (precedent: the spatial sound recording in AnimVR).

Who would use this gadget?

Sound designers/engineers, musicians, streamers

Assuming that you're on the team, what other skillsets would you need to make this project happen over a couple days of hacking?

I'd need collaboration around everything that isn't sound design/ graphic design (I'm a programming novice with zero React experience). I don't even know enough to know what I don't know, in fact. I'd need enthusiastic and patient programming partners in order to attempt this project.

What will be the toughest part of building this gadget?

Possibly the rendering out of spatially specific stereo (or stereo+) audio at the hand location rather than at the head position, potentially necessitating the toggling between those audio vantage points or the simultaneous tracking of them? The matter of playback and UI for managing the experience of creating these 'sound pearls' could become pretty involved pretty quickly.

[BOLL7708]

This does sound like it would bump into some walls, audio would be simulated for the head position so yes if we could black out the headset and move it to the audio record position we would get the right audio from whatever scene application is running, like a game.

Not sure this would be comfortable though, probably needs driver hooks like Input Emulator and not sure Chromium has access to record other desktop audio either 😅 It might be possible, just sounds like it would need some implementation in Aardvark directly to work at all 🤔

[JoeLudwig]

I'm not sure about Chromium support for capturing the audio from the system itself. I'd likely have to plumb the permission popups into Aardvark, but that's on the long-term TODO list anyway. Using the head position as the capture position would work with all apps if we can figure out how to capture audio from the web. Maybe the handheld part of this isn't necessary?

Another variant that would definitely work would be a dictation gadget. Let the user record their own mic to take notes or whatever.

[technobotanist]

Yeah, the handheld part is likely a level of abstraction that would serve as convoluted UI anyway. 'True to head position' sound capture is likely a lot more intuitive.

[kahunamoore]

lolz, I was just working on an MRE for BRCvr (in AltSpaceVR) to make a "Where Thoughts Go" type of experience for the virtual burning man earlier this month. Ran into the fact that MREs can't access the microphone.

Failing that, I started down the path of having the user record audio via a web page that would forward/match w/ the avatar/MRE in AltSpaceVR. Seemed like a totally lame workaround but I thought I'd try anyway. Unfortunately I got pulled off that effort to help model some of the playa art pieces so never completed it.

Also, I have several paid & royalty-free/CC0 tape deck models I downloaded from one of the 3d model sites available if needed.

I'd be down trying to get something working. Near complete lack of art skills offset by an ok bag of dev skills. Somewhat time constrained but willing to do what I can.

[kahunamoore]

I'm not sure about Chromium support for capturing the audio from the system itself. I'd likely have to plumb the permission popups into Aardvark, but that's on the long-term TODO list anyway.

This sounds like an interesting problem to work on. I'll take a look at the Chromium apis.

[JoeLudwig]

Assuming Javascript provides APIs to capture desktop audio this should just be the permission dialogs in Aardvark.