search

liamappelbe / fftea

A simple and efficient FFT implementation for Dart
Apache License 2.0
68 stars 1 forks source link

Real-time collection of rhythm points #52

Open tang158152 opened 1 month ago

tang158152 commented 1 month ago

I use the microphone to pick up the audio raw data of a song, and return 4100 sample points every 0.1 seconds, how should I get the rhythm point perceived by the human ear of this song, because we need to upload the rhythm point in real time with hardware, I have tried to use the sample point to convert RMS amplitude, or FFT, but there will always be rhythm points that cannot be collected, is there any solution to provide, I am a novice in audio, I can't understand many concepts of audio, I hope to provide an idea, thank you

liamappelbe commented 1 month ago

Not really sure what you mean by rhythm points, but I'll assume you're asking about BPM detection. I said some stuff about BPM detection in this bug: https://github.com/liamappelbe/fftea/issues/36

Always try to break the problem down into simpler steps. This real time BPM detection on a streaming sound source is a pretty challenging problem, so you should start simpler and build up to that:

  1. A nice thing about Dart is you can write your program as a command line tool, and then just copy/paste the code into your Flutter app afterwards. So that's a good place to start to keep things simple. Write a command line tool that loads a WAV file, processes it, and tells you the average BPM.
    • This is easier to test because you can construct artificially easy audio to verify it's working (eg a click track at a particular BPM).
    • Then you can test it on a real song with a known BPM
    • Maybe your question is about finding the timestamps of the beats? If so, implement that here.
    • Only move on to the next step once you have this working reliably.
  2. Adapt that code so that instead of processing the whole file at once, it streams it in small chunks, and tells you the current BPM, or the timestamps of the beats.
  3. Now in your app, test that your microphone streaming library is working by saving the chunks to a file, downloading that file to your PC, and making sure it sounds correct. Some of the Flutter mic libraries actually don't work very well, and the audio can have artifacts that will break your BPM detector. So verifying the audio is a critical step.
  4. Once you've done all that, you're ready to stick your BPM detection algorithm in your app.

This is a hard problem, so no skipping steps. Build up to complex things bit by bit.

Ok, now for the actual beat detection algorithm in step 1, you might not actually need FFT at all. Try running the audio through a low pass filter, then calculate the moving average RMS volume, then look for peaks in that average. As I said in the other bug, there will probably be some noise in the beats you get, so you'll have to experiment with tuning the algorithm and debounce those beats.