Sound Wave refactor

[nighca]

• Get rid of wavesurfer, which now delivers little value & brings limits
• Fix TODOs introduced in #589
  • Avoid wave differences between recording & non-recording case
  • Smooth wave for long audio

[ComfyFluffy]

To refactor, we need to:

• Create a Vue component that renders waveform using canvas from given points (loadness). The points should be already smoothed and be in range of [0, 1]. We can potentially use a visualization library like d3 to achieve a better result.
• Come up with a new approach to compute the loudness data from microphone stream in realtime.
• Fine-tune the algorithm of rendering and/or data processing to achieve a more stable FPS.

For a regular audio file, we can get PCM data of channel directly with AudioBuffer.getChannelData().

For microphone streams, we can fetch PCM data in AudioWorklet and do processings in a separate thread:

const source = audioContext.createMediaStreamSource(stream)
await audioContext.audioWorklet.addModule('/logger-procesor.js')
const audioWorkletNode = new AudioWorkletNode(audioContext, 'logger-processor')
source.connect(audioWorkletNode)
...

// In logger-procesor.js:
class LoggerProcessor extends AudioWorkletProcessor {
  process(inputs, outputs) {
    const input = inputs[0]
    const output = outputs[0]

    const inputData = input[0]
    const outputData = output[0]

    console.log(inputData.length)

    for (let i = 0; i < inputData.length; ++i) {
      outputData[i] = inputData[i]
      // We can for example average the absolute value of data
      // in a channel and send the result to the main thread.
    }

    return true
  }
}

registerProcessor('logger-processor', LoggerProcessor)

Previously we use analyser.getFloatTimeDomainData() to get FFT transformed data of the waveform, which is not consistent with PCM-based rendering in wavesurfer.js.

[nighca]

We can potentially use a visualization library like d3 to achieve a better result.

It is probably unnecessary to involve another library like d3. The logic for wave-rendering is not complex with raw canvas 2D-context API.

Come up with a new approach to compute the loudness data from microphone stream in realtime.

If we've converted both audio file and microphone stream to the same audio data structure (e.g. PCM data as mentioned above), it should be possible to use the same method to get loudness data from them.