Sample of use rustpotter-java

[krusalovorg]

Hello! Do you have an example project using rustpotter-java? The code which you attached in an example does not react in any way to wakeword. I'd like to examine the full code to make sure I'm doing everything right. Thank!

[GiviMAD]

This one is the only project I have done with it https://github.com/openhab/openhab-addons/blob/main/bundles/org.openhab.voice.rustpotterks/src/main/java/org/openhab/voice/rustpotterks/internal/RustpotterKSService.java

Yes the example is probably not correct, instead of discard the incomplete buffers the correct thing to do is store them in a circular buffer until you have what rustpotter needs.

Let my know if you need more help, hope the sample is enough clear. It uses directly the stream bytes instead of sampling them like in the example.

If you can PR a better example it will be awesome :)

[krusalovorg]

My code:

    private fun processAudioStream(
        rustpotter: Rustpotter,
        audioStream: InputStream,
    ) {
        var numBytesRead: Int
        val frameSize = rustpotter.getSamplesPerFrame().toInt()
        val bufferSize = frameSize * 2
        val captureBuffer: ByteBuffer = ByteBuffer.allocate(bufferSize)
        captureBuffer.order(ByteOrder.LITTLE_ENDIAN)
        val audioBuffer = ShortArray(frameSize)

        while (true) {
            try {
                // read a buffer of audio
                numBytesRead = audioStream.read(captureBuffer.array(), 0, captureBuffer.capacity())
                if (numBytesRead != bufferSize) {
                    // do not pass incomplete buffers
                    Thread.sleep(100)
                    continue
                }
                captureBuffer.asShortBuffer().get(audioBuffer)
                val result: Optional<RustpotterDetection>? = rustpotter.processSort(audioBuffer)
                if (result != null) {
                    if (result.isPresent) {
                        println("OFGSGSDGSDGSD")
                        val detection = result?.get();
                        val word = detection?.name;
                        println("keyword '${detection}' detected ${word} with score!")
//                    listener.accept(word.toString())
                    }
                }
            } catch (e: IOException) {
                val errorMessage = e.message
                throw e // another consumer for the errors makes more sense as this function is intended to run on a separate thread.
            } catch (e: InterruptedException) {
                val errorMessage = e.message
                throw e
            }
        }
        rustpotter.delete()
        println("rustpotter stopped")
    }
    @FXML
    private fun onHelloButtonClick() {
        Rustpotter.loadLibrary();
        val rustpotterBuilder = RustpotterBuilder()
        rustpotterBuilder.setThreshold(0.1f)
        rustpotterBuilder.setAveragedThreshold(0.1f)
        rustpotterBuilder.setSampleRate(16000)
        rustpotterBuilder.setChannels(2)
        val rustpotter = rustpotterBuilder.build()
        rustpotterBuilder.delete()
        rustpotter.addWakewordModelFile("rust_model/elie.rpw")

        Thread {
            val mixerInfos = AudioSystem.getMixerInfo()
            val mixerInfo = mixerInfos[17]
            println(mixerInfo.name)
            val mixer = AudioSystem.getMixer(mixerInfo)

            val targetLineInfo = mixer.targetLineInfo[0]
            val microphone = mixer.getLine(targetLineInfo) as TargetDataLine
            microphone.start()
            val audioStream = AudioInputStream(microphone)
            processAudioStream(
                rustpotter,
                audioStream
            )
        }.start()
    }`
``

    Why doesn't rustpotter seem to hear me?

[GiviMAD]

I meant something like this:

    final AtomicBoolean stopped = ...

    private fun processAudioStream(
        rustpotter: Rustpotter,
        audioStream: InputStream,
    ) {
         int numBytesRead;
        var bufferSize = (int) rustpotter.getBytesPerFrame();
        byte[] audioBuffer = new byte[bufferSize];
        int remaining = bufferSize;

        while (!stopped.get()) {
            try {
                // read bytes into the audio buffer
                numBytesRead = audioStream.read(audioBuffer, bufferSize - remaining, remaining);
                if (stopped.get() || numBytesRead == -1) {
                    break;
                }
                if (numBytesRead != remaining) {
                    // continue reading remaining bytes until buffer is full
                    remaining = remaining - numBytesRead;
                    continue;
                }
                // reset remaining so next iteration starts filling the buffer at the beginning
                remaining = bufferSize;
                // feed the buffer into rustpotter processBytes method
                var result = rustpotter.processBytes(audioBuffer);
                // process response if any
                if (result.isPresent()) {
                    var detection = result.get();
                    if (logger.isDebugEnabled()) {
                        ArrayList<String> scores = new ArrayList<>();
                        var scoreNames = detection.getScoreNames().split("\\|\\|");
                        var scoreValues = detection.getScores();
                        for (var i = 0; i < Integer.min(scoreNames.length, scoreValues.length); i++) {
                            scores.add("'" + scoreNames[i] + "': " + scoreValues[i]);
                        }
                        logger.debug("Detected '{}' with: Score: {}, AvgScore: {}, Count: {}, Gain: {}, Scores: {}",
                                detection.getName(), detection.getScore(), detection.getAvgScore(),
                                detection.getCounter(), detection.getGain(), String.join(", ", scores));
                    }
                    detection.delete();
                }
            } catch (e: IOException) {
                val errorMessage = e.message
                throw e // another consumer for the errors makes more sense as this function is intended to run on a separate thread.
            } catch (e: InterruptedException) {
                val errorMessage = e.message
                throw e
            }
        }
        rustpotter.delete()
        println("rustpotter stopped")
    }

[GiviMAD]

Also need to set this:

rustpotterBuilder.setSampleFormat(SampleFormat.INT);
rustpotterBuilder.setEndianness(Endianness.LITTLE);

Are you sure the audio uses 2 channels?

And you can set rustpotterBuilder.setMinScores(0) and rustpotterBuilder.setAveragedThreshold(0.0) while testing if it works.

[krusalovorg]

Hello! Sorry, but that didn't help. Rustpotter doesn't recognize the phrase. Do you have a telegram for live communication?

[GiviMAD]

Do you have a telegram for live communication?

I was planning on open a matrix room to enable live communication, but I couldn't find the time for it.

I have added a test in the v3 about consuming a java AudioStream, maybe it can help to detect your problem: https://github.com/GiviMAD/rustpotter-java/blob/v3/src/test/java/io/github/givimad/rustpotter_java/RustpotterTest.java