Support for audio files

[mbpictures]

Hi!

First of all thanks for this awesome package! I just wanted to ask whether it's possible to add support for prerecorded audio files? So e.g. the method PitchDetector.detectPitches accepts a file path (or a Float32Array containing the audio data) and then returns an array with the frequency or null and the time relative to the start of the audio file.

Thanks in advance and all the best!

[mbpictures]

As an implementation example, I'm currently using this class, which works without any native code and relies on ffmpeg (transcoding audio to desired sample rate), the pitchfinder package, and the audio-decode package:

import RNFS from "react-native-fs";
import {FFmpegKit, ReturnCode} from "ffmpeg-kit-react-native";
import {Buffer} from "buffer";
import decode from "audio-decode";
import Pitchfinder from "pitchfinder";
import {frequencyToNoteIndex} from "../../util/notes";
import {Frequency} from "./Detector";

interface Frequency {
    time: number;
    frequency: number | null;
    noteIndex: number;
}

export class YIN {
    private readonly _samplingRate: number;
    private readonly _bufferSize: number;
    private readonly detector: PitchDetector;
    constructor(samplingRate:number = 22500, bufferSize: number = 1024) {
        this._samplingRate = samplingRate;
        this._bufferSize = bufferSize;
        this.detector = Pitchfinder.YIN({
            sampleRate: samplingRate
        });
    }

    private async convertAudioForModel(file: string): Promise<Float32Array> {
        const tempFile = RNFS.TemporaryDirectoryPath + "/resampling/sample.wav";
        if (!(await RNFS.exists(RNFS.TemporaryDirectoryPath + "/resampling"))) {
            await RNFS.mkdir(RNFS.TemporaryDirectoryPath + "/resampling");
        }

        const command = `-i "${file}" -ac 1 -ar ${this._samplingRate} -q:a 0 "${tempFile}"`;
        const fFmpegSession = await FFmpegKit.execute(command);
        if (!ReturnCode.isSuccess(await fFmpegSession.getReturnCode())) {
            throw new Error("Error while resampling");
        }
        const fileContent = await RNFS.readFile(tempFile, "base64");
        const buffer = Buffer.from(fileContent, "base64");
        const result = await decode(buffer);

        await RNFS.unlink(tempFile);
        return result
            .getChannelData(0)
            .map((a: number) => a) as Float32Array; // without this mapping, the function never returns. No idea why
    }

    private getPitches(audio: Float32Array) {
        const pitches = [];
        const max = audio.length - this._bufferSize;
        for (let i = 0; i <= max; i += this._bufferSize) {
            pitches.push(this.detector(audio.slice(i, i + this._bufferSize))); // slice audio into chunks with size of buffersize and apply pitch detection algorithm
        }
        return pitches;
    }

    private convertToFrequencies(pitches: (number | null)[]): Array<Frequency> {
        const timePerPitch = (1 / this._samplingRate) * this._bufferSize * 1000;
        return pitches.map((p, i) => ({
            time: i * timePerPitch,
            frequency: p,
            noteIndex: p !== null ? frequencyToNoteIndex(p) : 0
        }));
    }

    async process(file: string) {
        const audio = await this.convertAudioForModel(file);
        const pitches = this.getPitches(audio);
        const frequencies = this.convertToFrequencies(pitches);
        return this.convertToFrequencies(pitches);
    }
}

During testing I encountered, that it works, but doesn't have the same accuracy as the real-time pitches from your package, unfortunately.

[marksyzm]

I did something like this without having to use ffmpeg - the wavefile package:

      import {WaveFile} from 'wavefile';

      // get base 64 from wave file
      const waveFileBase64 = await RNFetchBlob.fs.readFile(
        waveFilePath,
        'base64',
      );
      await deleteFile(waveFilePath);
      const waveFile = new WaveFile();
      waveFile.fromDataURI(waveFileBase64);

      // get wav samples
      const audioFileSamples = waveFile.getSamples(
        false,
        sampleArrayTypes[bitsPerChannel],
      );

I imagine ffmpeg would be more efficient, making use of the native environment and/or C

[marksyzm]

OT: WIth pitchfinder, there's a bug on the Yin profile where it won't detect below a certain frequency, but if you use the McLeod profile then this should solve it. I use the pitchy library for the same purpose that you mention.

[mbpictures]

@marksyzm I'm using ffmpeg to convert other files than wav (e.g. mp3, ogg, ...) and to resample with the required sample rate. Thanks for mentioning pitchy, sounds really promising!

[marksyzm]

Of course, mine wasn't a good example - just another route. You still need that wave buffer data of course, regardless of the file format - I was just suggesting this in case it's not that efficient or granular enough.

I'm hoping to fix a couple of things with this package soon, such as issues with the buffer size on android not correlating to packet performance, and similarly with iOS. Feel free to chat direct with me on linkedin as I'm working on something at the moment too: https://www.linkedin.com/in/markelphinstone/