FFT Texture for shaders

[strangerattractor]

Acceptance Criteria

• [x] Waveform of all channels can be obtained as a texture in realtime in the game mode
• [x] Spectrum of all channels can be obtained as a texture in realtime in the game mode
• [x] Spectrogram of all channels can be obtained as a texture in realtime in the game mode
• [x] target channel is selectable in the inspector and can be changed in the game mode
• [x] the information is stored in the red channel only texture ( TextureFormat.R16 ) since it contains just one channel of information.

Another very useful tool I found in Cables was the FFT Texture. this texture could be applied to shaders as different types of texture. f.e. Normal map. Displacement Map etc.

I did a small demo before we started here: https://cables.gl/p/XEtTUN

do you see a chance of using different styles of audio visualizations, so that they can be used as textures in shaders?

Starting with FFT

but also sound wave

and maybe also frequency spectrum. Although I don't have an idea how to make that look cool ;-)

[chikashimiyama]

Ok, you want to have FFT spectrum and waveform in the form of texture? if so, it's possible. I can implement that in 4-8 hours.

[strangerattractor]

yes. Best case scenario would be, that I can use the texture in shader graph Do it please :-)

[chikashimiyama]

@strangerattractor The one you mention in the beginning is not FFT. It's a spectrogram. I'm going to make three subtasks out of this task

[chikashimiyama]

@strangerattractor

Actually most of the implementation can be done much faster in a shader. To draw texture I have to fill 512 * 512 = 262144 pixels 60 times per seconds = 15728640 pixels. This is a lot of data and CPU should not do this.

What I want to do is to pass the content of the audio buffer (512 samples) or 1D texture with 512 pixels to the GPU and let GPU generate Texture for us. This is almost a must otherwise we lose a lot of resources for this.

so now shader graph...

[chikashimiyama]

hmm... I'm not sure if the shader graph is a good idea for this kind of generative shader. I need to create a 2D texture out of 1D texture programatically. UV coordinate is determined by the intensity of the input pixel... I can code that easily but in shader graph...

[chikashimiyama]

Custom node is very buggy... will try this. https://docs.unity3d.com/Packages/com.unity.shadergraph@5.7/manual/Custom-Nodes-With-CodeFunctionNode.html

It doesn't make sense if I write code in Shader Graph.

[chikashimiyama]

Jees. This way is deprecated.

https://forum.unity.com/threads/unable-to-create-custom-shader-graph-node-due-to-inaccessibility-of-customfunctionnode-class.586876/?_ga=2.223159067.1886793707.1569929867-1408144593.1543075298

[chikashimiyama]

It's really hard to write custom shader function.. there is not so much info on the internet.

@strangerattractor

Can I assign this task to you? otherwise I waste a lot of time. It's about Shader Graph. In HDRP, it's really hard to code with text. Everything has to be a node but I have no idea how I should map the intensity of Texture1D to the UV coordinate of Texture2D.

Basically I provide 1 array of float [512]. You generate a waveform texture based on it. We have to do this in GPU not CPU. but then we have to do that in Shader Graph in HDRP.

[strangerattractor]

ok, where should I start... I'll try my best.

[chikashimiyama]

I can give the Shader graph a texture 2D ( 512 x 512). The first raw of your texture contains the waveform info as the color value. based on that you render actual waveform in another texture.

[chikashimiyama]

@strangerattractor

Spectrogram is almost no loss with CPU solution. so I will do that first.

[chikashimiyama]

@strangerattractor

The spectrogram is implemented I would go waveform with CPU because I have to refresh only 1024 pixels per frame. That's doable with CPU.

The spectrum visualizer requires GPU because the number of pixels that I have to refresh is dynamic

[strangerattractor]

right now the spectrum is not logarithmic, correct? Could you add an option to make it logarithmic, also could you add an option to focus on a specific range of the spectrum in more detail?

• [ ] option for logarithmic, non logarithmic view

• [ ] input-option for specific frequency range to be looked at

[chikashimiyama]

@strangerattractor Could you make another issue if you want to add more features? We should keep the lifetime of one issue as short as possible to simplify each task.

[chikashimiyama]

Made #80 Please close this issue, if you are fine with the original ACs