ddf / Minim

A Java audio library, designed to be used with Processing.
http://code.compartmental.net/tools/minim
GNU Lesser General Public License v3.0
665 stars 135 forks source link

High latency (up to half minute) when doing fft on live audio from mixer #89

Open hamoid opened 5 years ago

hamoid commented 5 years ago

I'm trying to do FFT on live audio from the mixer, so I can feed data into sketches based on whatever music I happen to be playing. The issue I'm having is latency: between 2 and 20 seconds. And it seems to drift, so I believe it increases while the program runs.

Could it be something related to mismatching bitrates or buffer sizes?

import ddf.minim.*;
import ddf.minim.analysis.*;
import javax.sound.sampled.*;

Minim minim;
AudioInput audio;
FFT fft;

// This affects latency, and in theory lower = less latency
int bufferSize = 512; // try 128 256 512 1024

void setup() {
  size(800, 100, P2D);
  // here you configure your audio device. Run it once and look in the console
  // to see available names. In my system one of the lines looks like
  // [2] default [default], version 4.18.0-12-generic
  // so I grab the part between [num] and the comma.
  startAudioInput("default [default]");
  noStroke();
}

void draw() {
  background(0);
  if (fft != null) {
    fft.forward(audio.mix);
    int bands = fft.avgSize();
    for (int i = 0; i < bands; i++) {
      float val = fft.getAvg(i);
      float x = map(i, 0, bands, 0, width);
      rect( x, height, 2, -val);
    }
  }
}

private void startAudioInput(String deviceName) {
  minim = new Minim(this);

  Mixer.Info[] mixerInfo = AudioSystem.getMixerInfo();

  for (Mixer.Info info : mixerInfo) {
    if (info.getName().equals(deviceName)) {
      minim.setInputMixer(AudioSystem.getMixer(info));
      audio = minim.getLineIn(Minim.STEREO, bufferSize);

      fft = new FFT(audio.bufferSize(), audio.sampleRate());
      fft.logAverages(22, 3);
      return;
    }
  }
  println(deviceName, "not found! Use one of these:");
  printArray(mixerInfo);
}

I also asked about it on the forum.

ddf commented 5 years ago

I haven't run the program yet, but my guess would be that the audio from the mixer is somehow not being read fast enough so it backs up over time.

It's also possible this is just a problem on Linux in particular. I just dug around in the code again to remind myself how the AudioInput works:

This should work OK, but I came across a comment I left for myself in the code that gets the SourceDataLine about latency just being bad in Linux: https://github.com/ddf/Minim/blob/d5497221c4f55d7202d9166f2c3e77b7d148e956/src/main/java/ddf/minim/javasound/JSMinim.java#L818

All that said, I worked on a project over the summer that also incorporated monitoring the current system audio in order to create reactive visuals. We were working on Windows, essentially through FMOD, and also ran into latency issues there. It wasn't as long as 20 seconds, but it was long enough to look incorrect. Basically I think the only way around this is to somehow mute the audio that is going directly from the browser or wherever and enable monitoring on your AudioInput so that you hear exactly what your program is reading.

You might also try to use a different Linux audio system? I'm not up on what's currently shipping with Linux, but I think I've read in the past that Pulse Audio generally has better latency than ALSA?

Another thing you could try, is to use getInputStream instead of getLineIn. This will give you an AudioStream (https://github.com/ddf/Minim/blob/master/src/main/java/ddf/minim/spi/AudioStream.java) object that you must call read on to get data out of it. It won't automatically pull. So in draw you'd read from the AudioStream until it read fewer samples than your bufferSize. Best would be to create MultiChannelBuffer (https://github.com/ddf/Minim/blob/master/src/main/java/ddf/minim/MultiChannelBuffer.java) with the same bufferSize that you use for the AudioStream and use the read method that takes a MultiChannelBuffer and returns number of samples read. You'd have to decide whether to run the FFT on every buffer or accumulate it somehow or whatever. Definitely it's more complex code and is why AudioInput exists in the first place.

This might help you determine whether the latency is coming from the read loop of AudioInput or if it is latency in putting audio data into the Java's TargetDataLine on the Linux side.

neilcsmith-net commented 5 years ago

Came through from the Processing forum post on this. In my experience Linux has by far the best latency performance of all the OS JavaSound implementations. I'm intrigued why we get such different results. Will try and take a look sometime.

hamoid commented 5 years ago

Thank you for the detailed answer!!! I'll do tests and report back :)

ddf commented 5 years ago

@hamoid Curious if you have anything to report about all this.

rvorias commented 4 years ago

Hey guys, I had the same problem on Elementary Juno (Ubuntu 18.04 LTS), Processing 3.5.3. I implemented @ddf's recommendation and reading from an inputStream seems to do the trick for me. This is my QaD-code:

import ddf.minim.*;
import ddf.minim.spi.*;

Minim minim;
AudioStream in;

MultiChannelBuffer sampleBuffer;

void setup()
{
  size(512, 200, P3D);

  minim = new Minim(this);

  in = minim.getInputStream(2,1024,44100.0,16);
  in.open();
  sampleBuffer = new MultiChannelBuffer(1024, 2);
}

void draw()
{
  background(0);
  stroke(255);

  // draw the waveforms so we can see what we are monitoring

  in.read(sampleBuffer);
  for(int i = 0; i < 1023; i++)
  {
    line( i, 50 + sampleBuffer.getSample(0,i)*50, i+1, 50 + sampleBuffer.getSample(0,i+1)*50 );
    line( i, 150 + sampleBuffer.getSample(1,i)*50, i+1, 150 + sampleBuffer.getSample(1,i+1)*50 );
  }
}

void stop() {
  in.close();
}

Could someone verify this on their system? update: it didn't go as well as I expected. Findings:

Right now I reverted to using an audiosplitter (hardware) and rewire one output to the mic input and use that for audio analysis. It's noisy on the low-end spectrum but at least it's fast.

hamoid commented 2 years ago

I just added a Minim example to OPENRNDR and remembered this issue.

I tried my code above and also the one from @rvorias but I couldn't start diagnosing things because I'm getting the microphone input instead of my system audio, which was my original plan. I tried the different devices returned by AudioSystem.getMixerInfo(); (on Ubuntu).

At least the microphone input seems to have low latency :)