Possible to pass in Audio Data?

[Andersama]

Clearly this is the part where variables are being updated, it looks as if the structure is arbitrary, thought it might be interesting...

static void shader_filter_tick(void *data, float seconds)
{
    struct shader_filter_data *filter = data;
    obs_source_t *target = obs_filter_get_target(filter->context);

    // Determine offsets from expansion values.
    int base_width = obs_source_get_base_width(target);
    int base_height = obs_source_get_base_height(target);

    filter->total_width = filter->expand_left
        + base_width
        + filter->expand_right;
    filter->total_height = filter->expand_top
        + base_height
        + filter->expand_bottom;

    filter->uv_scale.x = (float)filter->total_width / base_width;
    filter->uv_scale.y = (float)filter->total_height / base_height;

    filter->uv_offset.x = (float)(-filter->expand_left) / base_width;
    filter->uv_offset.y = (float)(-filter->expand_top) / base_height;

    filter->uv_pixel_interval.x = 1.0f / base_width;
    filter->uv_pixel_interval.y = 1.0f / base_height;

    filter->elapsed_time += seconds;
        //pass along pcm data?
    //filter->pcm = // ?
}

[nleseul]

It's a great idea, but the implementation would take some thought.

First question would be what format the audio data would need to be in. Could be either a raw waveform, or a frequency graph. Most use cases would probably want a frequency graph, I think. Probably shouldn't rely on the shader to do an FFT on a waveform. Possibly something that can be toggled by the user?

Second, how does the shader code see this data? HLSL obviously doesn't have any native support for audio data. It would probably want to be a texture1d, I guess? Might need to use something like HLSL annotations to tag a particular uniform texture1d as being intended to be populated with audio, if possible.

Third, which audio data should be used? OBS has a fairly complicated mixer pathway, where each audio source can be sent to 6 different channels, and each of the 6 channels can be selected to go to the stream, the output file, or neither. Somehow you'd have to decide which bits of that process should be captured and sent into the shader. I also haven't looked into how well OBS's API supports accessing all those audio buffers from a plugin, so there might need to be some API work in OBS to expose that.

Excellent idea, but rather complex to implement.

[nleseul]

Ooh, I do see that you added annotation support to OBS a few months ago! That will definitely help with things like this once it makes it into an OBS release. obsproject/obs-studio@af6708691269066782388748c568802fb3704d52

[Andersama]

@nleseul I've actually been hoping to talk to you, yeah I've rewritten your plugin and I've already started trying to make some neat features w/ annotations. Feel free to check it out.

(it can handle audio)

https://github.com/andersama/obs-studio/tree/shader-filter-cpp

[Andersama]

Oh, and I guess to clarify, the annotation determines the audio's transformation, whether it's raw audio (the waveform), and fft etc what window function to use. And is uploaded as a texture2d, where the x dimension represents the array of audio data and the y represents the channel.

Though looking at this now I think I should probably give some control over the channels used, just to maybe save on some processing.