Example combining node-hotworddetector with node-audiorecorder for recording wav file?

[b-g]

Hi @RedKenrok,

I'm trying to use node-hotworddetector to trigger recording of a wav file via node-audiorecorder (I would like to send the wav file later to the google-cloud/speech api).

I manged to to get it running with the two independent scripts. But I'm having trouble to combine the two.

Any hints on how to combine the two node modules in an elegant way to record a wav file?

(I'm aware of https://github.com/RedKenrok/Electron-VoiceInterfaceBoilerplate/blob/master/app/scripts/input.js ... but would prefer a node only solution)

snowboy hotword detection

const HotwordDetector = require('node-hotworddetector');

const detectorData = {
  resource: './node_modules/snowboy/resources/common.res'
};
const modelData = [{
  file: './node_modules/snowboy/resources/snowboy.umdl',
  hotwords : 'snowboy'
}];

let hotwordDetector = new HotwordDetector(detectorData, modelData, {audioGain: 2}, console);

hotwordDetector.on('error', function(error) {
  console.error('Error: ' + error);
});

hotwordDetector.on('hotword', function(index, hotword, buffer) {
  console.log('Hotword: ' + hotword);
});

hotwordDetector.on('silence', function() {
  console.log('Silence');
});

hotwordDetector.on('sound', function(buffer) {
  console.log('Sound');
});

hotwordDetector.start();

console.log('Active hotwords 🎤 :');
for (model of modelData) {
  console.log(model.hotwords);
}

record wav file

const fs = require('fs');
const AudioRecorder = require('node-audiorecorder');

let audioRecorder = new AudioRecorder({
  program: 'rec',
  channels: 1,
  bits: 16,
  encoding: 'signed-integer',
  silence: 0,
  rate: 16000,
  type: 'wav'
});
const fileStream = fs.createWriteStream('test.wav', { encoding: 'binary' });
audioRecorder.start().stream().pipe(fileStream);

audioRecorder.stream().on('close', function(code) {
  console.warn('Recording closed. Exit code: ', code);
});
audioRecorder.stream().on('end', function() {
  console.warn('Recording ended.');
});
audioRecorder.stream().on('error', function() {
  console.warn('Recording error.');
});

[RedKenrok]

Hello,

With send the wav file later do need to write it to file first or is that just your temporary solution and will be keeping it in active memory be good as well?

Kind regards, Ron

[RedKenrok]

I do not have a machine to test on at the moment so I can not guarantee that they work, but I have two scripts for you to test out. The first one (1) writes to file and the other (2) sends the audio stream to the Google Cloud services via the '@google-cloud/speech' module.

If you want to get this code cleaner perhaps write the function that gets triggered when a hotword has been detected separately and on its own so you can reference it.

Check them out and let me know whether this answers your question.

1:

// Node modules.
const fs = require(`fs`);
// Dependency modules.
const AudioRecorder = require(`node-audiorecorder`),
    HotwordDetector = require(`node-hotworddetector`);

// Options audio recorder.
const recorderOptions = {};
// Options hotword detector.
const detectorData = {
    resource: `./node_modules/snowboy/resources/common.res`
};
const modelData = [{
    file: `./node_modules/snowboy/resources/snowboy.umdl`,
    hotwords : `snowboy`
}];
const recorderData = {
    audioGain: 2
};

// Initialize hotword detector.
let hotwordDetector = new HotwordDetector(detectorData, modelData, recorderData, console);
hotwordDetector.on(`error`, function(error) {
    throw error;
});
hotwordDetector.on(`hotword`, function(index, hotword, buffer) {
    console.log(`Hotword: ` + hotword);

    // Record audio.
    let audioRecorder = new AudioRecorder(recorderOptions);
    // Start recording.
    audioRecorder.start();
    // Listen to events.
    audioRecorder.stream().on(`error`, function() {
        throw error;
    });
    audioRecorder.on(`close`, function(exitCode) {
        console.log(`audio stream closed.`);
        audioRecorder.stop();
    });
    audioRecorder.stream().on(`end`, function() {
        console.log(`Audio recording ended.`);

        // Send file to `google-cloud/speech api`.
        console.log(`TODO: start sending file to 'google-cloud/speech api'.`);
    });

    // Create file stream and transfer audio to it.
    const fileStream = fs.createWriteStream(`test.wav`, { encoding: `binary` });
    audioRecorder.stream().pipe(fileStream);
});

// Start detection.
hotwordDetector.start();

2:

// Dependency modules.
const AudioRecorder = require(`node-audiorecorder`),
    HotwordDetector = require(`node-hotworddetector`);
const googleSpeech = require(`@google-cloud/speech`);

// Options audio recorder.
const recorderOptions = {};
// Options hotword detector.
const detectorData = {
    resource: `./node_modules/snowboy/resources/common.res`
};
const modelData = [{
    file: `./node_modules/snowboy/resources/snowboy.umdl`,
    hotwords : `snowboy`
}];
const recorderData = {
    audioGain: 2
};
// Options Google-Cloud Speech API.
const speech = new googleSpeech.SpeechClient({
    keyFilename: `KEYPATH_GOOGLECLOUD`
}); // TODO: Add path to file.
const speechRequest = {
    config: {
        encoding: 'LINEAR16',
        sampleRateHertz: 16000,
        languageCode: 'en-GB'
    },
    interimResults: false
};

// Initialize hotword detector.
let hotwordDetector = new HotwordDetector(detectorData, modelData, recorderData, console);
hotwordDetector.on(`error`, function(error) {
    throw error;
});
hotwordDetector.on(`hotword`, function(index, hotword, buffer) {
    console.log(`Hotword detected: '${hotword}'.`);

    // Record audio.
    let audioRecorder = new AudioRecorder(recorderOptions);
    // Start recording.
    audioRecorder.start();
    // Listen to events.
    audioRecorder.stream().on(`error`, function() {
        throw error;
    });
    audioRecorder.on(`close`, function(exitCode) {
        console.log(`Audio stream closed.`);
        // Stop audio recorder.
        audioRecorder.stop();

        // Re-start detection so cycle continues.
        hotwordDetector.start();
    });

    // Start Google-Cloud Speech API web stream.
    let stream = speech.streamingRecognize(speechRequest)
        .on('error', console.error)
        .on('data', function(data) {
            console.log(`Transcript: '${data.results[0].alternatives[0].transcript}'.`);
        });

    // Start streaming audio to web stream.
    audioRecorder.stream().pipe(stream);
});

// Start detection.
hotwordDetector.start();

Kind regards, Ron

[b-g]

Hi @RedKenrok, Many thanks for this! 🙏 Yes adding these examples to the repo would be super helpful! Exactly what I was hoping for.

I gave both quicky a try ... unfortunately I get AudioRecorder: Exit code: '1'..

1

$ node example1.js 
AudioRecorder command: rec -V0 -q -L -c 1 -r 16000 -t wav -b 16 -e signed-integer silence 1 0.1 0% 1 2 0% -
HotwordDetector initialized.
AudioRecorder: Started recording.
HotwordDetector: Started detecting.
AudioRecorder: Exit code: '1'.

2 I've added the keyfile ... and google speech is working.

$ node example2.js 
AudioRecorder command: rec -V0 -q -L -c 1 -r 16000 -t wav -b 16 -e signed-integer silence 1 0.1 0% 1 2 0% -
HotwordDetector initialized.
AudioRecorder: Started recording.
HotwordDetector: Started detecting.
AudioRecorder: Exit code: '1'.

3 Also I noticed that recording is just working on my machine if silence: 0.

const fs = require('fs');
const AudioRecorder = require('node-audiorecorder');

let audioRecorder = new AudioRecorder({
  program: 'rec',
  silence: 0,
  // silence: 2,
  // threshold: 0.35,
  type: 'wav'
});
const fileStream = fs.createWriteStream('test.wav', { encoding: 'binary' });
audioRecorder.start().stream().pipe(fileStream);

Otherwise the output is simply something like this ... not sure whether this related though.

$ node example3.js 
Recording ended.
Recording closed. Exit code:  false

Could you give it a try on your machine and check whether you have the same issues at some point?

Running:

• "@google-cloud/speech": "^2.1.1",
• "node-audiorecorder": "^1.1.5",
• "node-hotworddetector": "^1.2.1"
• I have sox "stable 14.4.2" installed via homebrew

[RedKenrok]

The issues seems to be the order of the sox/rec command options are in. The - needs to be added before the silence option not afterwards. If will release a patch for node-audiorecorder later today, and I will make sure to update this module as well.

[RedKenrok]

I've updated the recorder and detector to the following versions:

node-audiorecorder: v1.1.6
node-hotworddetector: v1.3.0-beta.0

Do let me know if this works for you.

[b-g]

Hi @RedKenrok, Many thanks for this! We are getting closer! :)

I've updated to:

node-audiorecorder: v1.1.6
node-hotworddetector: v1.3.0-beta.0

1 (see inline coments)

$ node example1.js 
AudioRecorder command: rec -V0 -q -L -c 1 -r 16000 -t wav -b 16 -e signed-integer - silence 1 0.1 0% 1 2 0%
HotwordDetector initialized.
AudioRecorder: Started recording.
HotwordDetector: Started detecting.
// If I say "snowboy" this triggers exactly once the wav recording ...
HotwordDetector; hotword detected; index: 1; hotword: snowboy.
Hotword: snowboy
AudioRecorder: Exit code: '0'.
// But saying a second time "snowboy" doesn't do anything

2 ✅ Works + super! Would be a great example for the repo!

$ node example2.js 
AudioRecorder command: rec -V0 -q -L -c 1 -r 16000 -t wav -b 16 -e signed-integer - silence 1 0.1 0% 1 2 0%
HotwordDetector initialized.
AudioRecorder: Started recording.
HotwordDetector: Started detecting.
HotwordDetector; hotword detected; index: 1; hotword: snowboy.
Hotword detected: 'snowboy'.
AudioRecorder: Exit code: '0'.
Transcript: 'hello Google this is a test'.
Transcript: ' hello wrong many thanks for the code example'.
Transcript: ' this is great'.
Transcript: ' it is actually working'.
Transcript: ' how cool is this'.
Transcript: ' many thanks!'.

3 Is recording a file now, even if silence != 0. However with the setting below I always get a file with a duration of 1 sec. I would have expected that the recording only stops if there was 2 secs of silence detected, or do I missed something?

const fs = require('fs');
const AudioRecorder = require('node-audiorecorder');

let audioRecorder = new AudioRecorder({
  program: 'rec',
  silence: 2,
  threshold: 0.5,
  type: 'wav'
});
const fileStream = fs.createWriteStream('test.wav', { encoding: 'binary' });
audioRecorder.start().stream().pipe(fileStream);

[RedKenrok]

1) It is working as intended, since it does not reactivate the hotword detector after it has finished recording. See the audioRecorder.on('close', function(exitCode) {... function where it only stops the recorder. Use hotworddetector.stop() and hotworddetector.start() to control this.

2) Awesome!

3) It should not start recording until it has gone above the threshold of 0.5 has been reached and only stop after the volume has not been above 0.5 for two seconds. It sounds like this is not working as it should, therefore I'll look into it later today.

[RedKenrok]

I've updated the recorder and detector to the following versions:

node-audiorecorder: v1.2.0
node-hotworddetector: v1.3.0-beta.1

The issue regarding silence (3) should now be resolved. Do note I have removed the threshold option in favour of the thresholdStart and thresholdStop options.

[b-g]

Hi @RedKenrok,

Great + many thanks! 🙏

I tweaked the examples a bit for my students audience ... see https://github.com/hfg-gmuend/1819-designing-systems-using-voice/tree/master/examples/snowboy-hotword-detection

• 1-hotword-detection.js
• 2-hotword-record-audio.js
• 3-hotword-google-speech-to-text.js

PS. In the README.mk of node-audiorecorder ... I would remove "overrides threshold" in the comment.

thresholdStart: 0.5, // Silence threshold to start recording, overrides threshold. thresholdStop: 0.5, // Silence threshold to stop recording, overrides threshold.

[RedKenrok]

Great + no problem love to help out + thank you for helping me improve the module!

PS: I have fixed the comments of the audio recorder library.