First effort on #7. Adds portaudio and configures it for stereo input of audio signals. Realtime audio is only supported on realtime rendering mode. Also found and fixed a race condition in the StageManager and added some commenting to try and find other Vulkan threading issues by adding thread IDs in the logs. Also the texture mapped has a hard-coded imageID of 1 right now, will add OSC commands to map streams to imageIDs later.
Implementation
Adds Portaudio in third_party. It needs to be a git submodule, as opposed to a cmake ExternalProject, because we rely on a not-normally-exported part of Portaudio, the lockless thread-safe circular queue.
Audio is added by the callback from portaudio into the queue. Then there's a new Compositor function to read one frame's worth of audio (assuming a hard-coded 60Hz frame for now) into a host-accessible buffer, and then call StageManager to issue a copy command to the GPU to transfer the latest buffer update to a device-only texture with optimal layout.
Right now the audio samples come in channel-interleaved format, so get uploaded to the video card as vec2 if stereo, and could be vec4 if quadraphonic. Packed triples of signed floats have limited GPU support and so are officially not supported. This is uploaded to the GPU as a texture image of n = (sample rate / frame rate) texel wide and 1 texel high.
If there's a desire to support an arbitrary number of channels it could be possible to instead upload un-interleaved single floats into an image of x texels wide and a height of 1 texel per channel, so stereo uploads would be n by 2, and quad would be n by 4, 7.1 surround could be n by 8, etc. This is saved for subsequent work, perhaps to be revisited when we add support for uploading audio to the GPU from previously recorded audio and video samples.
Purpose and motivation
First effort on #7. Adds portaudio and configures it for stereo input of audio signals. Realtime audio is only supported on realtime rendering mode. Also found and fixed a race condition in the StageManager and added some commenting to try and find other Vulkan threading issues by adding thread IDs in the logs. Also the texture mapped has a hard-coded imageID of 1 right now, will add OSC commands to map streams to imageIDs later.
Implementation
Adds Portaudio in third_party. It needs to be a git submodule, as opposed to a cmake ExternalProject, because we rely on a not-normally-exported part of Portaudio, the lockless thread-safe circular queue.
Audio is added by the callback from portaudio into the queue. Then there's a new Compositor function to read one frame's worth of audio (assuming a hard-coded 60Hz frame for now) into a host-accessible buffer, and then call StageManager to issue a copy command to the GPU to transfer the latest buffer update to a device-only texture with optimal layout.
Right now the audio samples come in channel-interleaved format, so get uploaded to the video card as
vec2if stereo, and could bevec4if quadraphonic. Packed triples of signed floats have limited GPU support and so are officially not supported. This is uploaded to the GPU as a texture image of n = (sample rate / frame rate) texel wide and 1 texel high.If there's a desire to support an arbitrary number of channels it could be possible to instead upload un-interleaved single floats into an image of x texels wide and a height of 1 texel per channel, so stereo uploads would be n by 2, and quad would be n by 4, 7.1 surround could be n by 8, etc. This is saved for subsequent work, perhaps to be revisited when we add support for uploading audio to the GPU from previously recorded audio and video samples.
Types of changes
Status