Implement frequency spectrum calculation

[CANA-Dan]

im wondering because im interested in making a spectral analyzer in blueprints. I know blueprints are slower, but i could probably make a spectral analyzer in a few days in blueprints rather than a few months in C++. i also intend to multi thread this, so even if blueprints are slower it shouldn't be as much of an issue. while i do intend to learn C++ in the future, right not im a bit crunched for time and need to get my current project finished in the next month or so. in addition, once im done my project, i fully intend on making a video tutorial showing the coding process from scratch. there are very little FFT resources/tutorials online, so i think this could be very helpful.

[gtreshchev]

The plugin right now can do spectral analysis. The plugin has a parameter called MagnitudeSpectrum that might be useful to you (it is calculated as "sqrt(re^2 + im^2)", where "re" is the Real FFT part and "im" is the Imaginary FFT part)

You can get this value in Blueprints using the "GetMagnitudeSpectrum" function. You can also use functions like "GetSpectralDifference", "GetSpectralDifferenceHWR" and "GetComplexSpectralDifference" which can also be useful.

See the documentation for more details (it's not ready yet as some of the plugin's features are still under development, but it will give you a basic idea of ​​how the plugin works)

[CANA-Dan]

hmm. looking through the documentation, its not exactly what im looking for, unless im understanding the documentation wrong. also i should apologize; when i wrote spectral analyzer, i meant spectrogram. if your plugin is able to give a db value of a specific frequency at a specific time, then that would be the final result of what im looking for. the main reason i was wondering about samples being open in blueprints, is because iv found some spectrogram code in C that i can read to a decent degree that i could code into blueprints. the only thing i would be missing are samples.

[gtreshchev]

There is no such functionality now, but it's a good idea to implement. In the future, I will implement this when I have time.

[CANA-Dan]

alright cool! thank you for reading over the message. im excited to be able to mess around with audio at a more fundamental level outside of directly using C or C++. i do want to learn a low level language of some kind (looking at rust actually, but the C Family of languages are much more common), but i dont have the time currently.

[gtreshchev]

No problem. I definitely recommend learning some low-level programming language like C/C++ or Rust, as understanding it will give you a huge range of possibilities for implementing anything.

[CANA-Dan]

i was looking through the docs, and the On Generate PCM Data delegate seems to give samples as the song plays. is there any way to get that sample data all at once? or if i cant, do you think there would be any loss in quality if i were to fast forwards the song extremely fast and try to get the data that way (ie, like 10x speed or something)? im assuming PCM is analogous to Sample in this instance.

[gtreshchev]

You can't get all PCM audio data from Blueprints at once, but only in C++. But in the future I will make it possible.

There will be no problems in quality, but there will be optimization. So it’s better not to do this, but to get data instantly.

[CANA-Dan]

hmm. well, lets say i was to use the On Generate PCM Data delegate normally (ie, just real time and not trying to speed up anything), i would get the current audio samples to mess around with right? just to be sure. because in that case, thats still pretty usable for my specific use case. it may not be fast, but it would work for now.

[gtreshchev]

Yes, they are generated as the sound wave is played (they are passed in the same way to the audio renderer)

[gtreshchev]

These are not really samples, but frames. So this is not data from any one channel, but from all channels. But I think you already understand the idea.

[CANA-Dan]

frames? is that whatever is in the main audio buffer thats being sent to the DAC to render? ie a chunk of audio a 1024-2048 samples long or something?

[gtreshchev]

Yes, but the engine usually uses 4096 by default

[CANA-Dan]

i see

[CANA-Dan]

status on the implementation of sending raw sample data to a blueprint (the entire song in one go)?

any possible things i could maybe help you research or find for its implementation? i may not know my C++ but i have a decent bit of confidence in my raw logic ability.

[CANA-Dan]

so im messing around with this now and im kind of curious. now that you have the raw wave buffer, do you think youd be able to add in the ability to grab samples as needed? previously i was wondering about getting the whole song all at once, but thats really not needed at all. im working on figuring out all the FFT/spectrogram math now and from what iv seen, its actually not too bad. im using the On Generate PCMData node to get a very small (128 sample) buffer to test with and iv had some middling success. DFT (which is the slow big brother to fft) is extremely slow to try to calculate, so thats why the small buffer size. using On Generate PCMData isnt really optimal tho as i would want to do this all faster than real time if possible. the raw wave buffer sound like it would work because it cleanly has all the samples in it right? just a guess based on the name and how it interacts with the built in spectrogram function.

i think the function get audio frame from sound wave by time custom is what im looking for actually, but i couldn't get it to work. im assuming im doing something wrong on my side.

[gtreshchev]

The "GetAudioFrameFromSoundWaveByTimeCustom" function is needed to get audio frames in the specified time range. You just need to specify the start and end time and you will get an array of the retrieved PCM audio data.

What exactly doesn't work for you?

[CANA-Dan]

i was getting really odd sample data out so i thought it was broken

[CANA-Dan]

right. what happened was that i couldn't get samples from time zero. 0.0001 is fine, but not 0. if i tried the return value would be false for the bool. heres the audio file below.

one thing i did notice when analyzing the samples of other audio (audio with actual sound) was the fact that the 0th index was never a value of 0, which is unexpected. a sin wave is always supposed to start on 0 right? actual question, not 100% sure). this leads me to believe that a sample or two is being dropped somewhere in the conversion chain. this could possibly explain the clicking you get from playing audio; your speakers may be expecting a more pure sin wave but instead go from nothing to some high positive value a single sample later. untitled.zip

on a separate note, the compress sound wave node isnt working the same for the raw wave buffer? it also doesnt provide options for compression method. is this intentional/expected behavior?

[CANA-Dan]

possibly the issue above (the 0th sample making get audio frame node break) is one and the same issue as the first sample not being on 0

[gtreshchev]

This was a small bug that the analysis could not pass if the specified time was 0. Should be fixed.

But the problem with snapping at the beginning and at the end of the audio is not related to this. The bug was related to the RAW format transcoding process and was fixed in Apr 29. Briefly, the bug was in the RAW audio data conversion that their 32-bit float range was "0:1" instead of "-1:1" and the audio data was not at 0, but closer to 0.5 at the beginning and at the end. https://github.com/gtreshchev/RuntimeAudioImporter/issues/19

[CANA-Dan]

ah i see that makes sense with the values weren't between the correct values.

iv also got some progress on the fft stuff. sorta.

i managed to get samples to dft conversion working, but only under certain conditions (following a tutorial basically) and im not sure how to get it working better. so im sorta giving up for now lol. its very complicated and Every. Single. Youtube. Video tells me JACK about how to actually implement it using real numbers, which is extremely frustrating because its not all that complicated overall. unreal engine just doesnt have complex numbers however and i have no idea how to convert that math into real number logic. i do know someone who could help me, but they are in uni and i dont want to bother them too much as they are making their thesis rn

that being said, i did find some working FFT code in C++ thats really easy to follow. its also very fast it seems. im going to tweak it to better suit our situation tho and add some more features.

ill convert convert my mesh generation stuff to C++ so you can have a node that allow for direct audio to 3d spectrogram conversion. it will have high degree of color and detail control as well as chunk loading. that wont take long to do as its just blueprints to C++, so easy peasy. in addition im also going to allow for curves to be added so you can specify how you want the spectrogram to crush/expand on certain frequency ranges. ie, if you want to showcase the mids, but not the lows or highs for example.

let me know if you have any other ideas for special spectrogram settings. shouldn't be too hard too add them in.

[gtreshchev]

There is a kissfft library that takes care of all the FFT calculations. I have implemented an FFT analyzer in AudioAnalysisTools based on it: FFTAnalyzer.

I didn't delve into the details of FFT much and basically just took a small part of the code and made it work optimized with Unreal Engine.

You can see how the calculation of the freqency spectrum (USoundVisualizationStatics::CalculateFrequencySpectrum) is implemented in the engine and implement something similar based on the AudioAnalysisTools plugin, which can handle it. I don't have time to implement it now, so I'll probably do it in the future.

[CANA-Dan]

wait, fft analyzer exists? what does it output? or is it jut used for other parts of the plugin and doesnt exist in blueprints? if it just outputs an array of fft values, then thats all you really need. it should basically just take in some samples and output an array of fft values. and you're kinda done. atleast for a basic version.

[CANA-Dan]

if it isnt that simple and its alot more work, ill keep on working on the spectrogram code on my side. i should have had a working spectrogram about 3 months ago, but iv been pushing it off because i thought it was more intimidating than it actually was. this new code iv been working on tho gives me hope that ill actually be able to have a spectrogram in the next few days. there isnt alot to go through to get it done; convert the blueprints that i have for the mesh stuff, and get look ahead/bands done, both of which are just a couple of for loops.

[CANA-Dan]

right, so im having some issues with getting visual studio to work with me.

i dont know enough about visual studio to understand what i should or should not be doing so im kinda at a loss. more or less im just trying to get a function showing up in blueprints and nothing else. yesterday when i tried to build the solution, i got literally 9k+ errors as visual studio seemed to had forgotten how C++ worked... or something. idk lol.

iv also decided to make a plugin. that way i can reuse this code on other projects more easily, or share it if someone has a need for the functions contained inside. this is currently what i have and i suspect im missing something basic and important

also, ill try to get my telegram account setup so i can better contact you. stuff like this is better relegated to a private chat

[CANA-Dan]

i managed to get it working. poggers! im using a function library for now, but i would like to move everything over to a plugin at a later date if possible. that way i can share this around. more easily. for aesthetic purposes, how would i go about adding tooltips like you have in all your nodes? i really like how much info they give and want to do the same if possible.

also, how do you give multiple outputs for a single function? its definitely possible, I'm just not sure how. looking around online did give me some methods, but im not sure they are correct (use a struct, array, class, ect). the most promising idea was to use pointers, but im still unsure.

[gtreshchev]

Hi. Please email me at gtreshchev@gmail.com with questions, as this doesn't seem appropriate here. I will answer here for now, but next time only by email.

The GENERATED_UCLASS_BODY macro is legacy, use GENERATED_BODY instead.

The plugin's source code is open, so you can see how it's done here. In a nutshell, there are comment fields like "@param", "@return" which are the traditional multi-line style used by the engine.

To return multiple parameters you can use non-const references, more information is here. I would highly recommend that you learn C++ before doing development.