wakeword alexa (Porcupine v1.9) on DIY client (<= 1GB RAM)

[royrogermcfreely]

hey,

got the diy client working with ww "hey sepia" and also the leds are working

but when i try to change the ww to "alexa", i get the broadcast event in the control hub:

Broadcaster event: {"broadcast":{"client":"alexa_chrome_app_v0.24.1","deviceId":"alexa","sepia-wake-word":{"state":"inactive","keywords":["Alexa"]}}}

this is the device section from the settings.js in clexi/www/sepia

"wakeWordSensitivity": "0.6",
"wakeWordNames": "Alexa",
"wakeWordVersion": "1.9",
"wakeWordRemoteUrl": "http://192.168.0.64/files/wake-word/porcupine/1.9/",
"voiceEngine": "sepia",
"voiceCustomServerURI": "",
"en-voice": "",
"de-voice": "de-DE pico f",
"asrEngine": "sepia",
"asrServerURI": "http://192.168.0.64:20741",
 "asrServerUser": "any",
"asrServerToken": "test1234",
 "en-asrModel": "",
 "de-asrModel": "my/vosk-model-de",
"big-screen-mode": true,

i tried with wwRemoteUrl: http://192.168.0.64:20721/files/wake-word/porcupine/ and http://192.168.0.64/files/wake-word/porcupine/

this is my complete settings.js file:

 //Settings primarily for headless mode and setup (URL parameter: 'isHeadless=true')
//Some examples: https://github.com/SEPIA-Framework/sepia-html-client-app/blob/master/Settings.md
SepiaFW.settings = {
        headless: {
                device: {
                        "host-name": "192.168.0.64",
                        "deviceId": "alexa",
                        "deviceLocalSiteData": {
                                "location": "home",
                                "type": "room",
                                "name": "unassigned",
                                "index": ""
                        },
                        "mediaDevices": {
                                "mic": {},
                                "player": {},
                                "tts": {},
                                "fx": {}
                        },
                        "microphoneSettings": {
                                "gain": 15
                        },
                        "deviceSounds": {
                                "micConfirm": "sounds/coin.mp3",
                                "alarm": "sounds/alarm.mp3"
                        },
                        "wakeWordSensitivity": "0.6",
                        "wakeWordNames": "Alexa",
                        "wakeWordVersion": "1.9",
                        "wakeWordRemoteUrl": "http://192.168.0.64/files/wake-word/porcupine/1.9/",
                        "voiceEngine": "sepia",
                        "voiceCustomServerURI": "",
                        "en-voice": "",
                        "de-voice": "de-DE pico f",
                        "asrEngine": "sepia",
                        "asrServerURI": "http://192.168.0.64:20741",
                        "asrServerUser": "any",
                        "asrServerToken": "test1234",
                        "en-asrModel": "",
                        "de-asrModel": "my/vosk-model-de",
                        "big-screen-mode": true,
                        "clexiGpioInterface": {
                                "items": [{

                        "wakeWordVersion": "1.9",
                        "wakeWordRemoteUrl": "http://192.168.0.64/files/wake-word/porcupine/1.9/",
                        "voiceEngine": "sepia",
                        "voiceCustomServerURI": "",
                        "en-voice": "",
                        "de-voice": "de-DE pico f",
                        "asrEngine": "sepia",
                        "asrServerURI": "http://192.168.0.64:20741",
                        "asrServerUser": "any",
                        "asrServerToken": "test1234",
                        "en-asrModel": "",
                        "de-asrModel": "my/vosk-model-de",
                        "big-screen-mode": true,
                        "clexiGpioInterface": {
                                "items": [{
                                        "id": "led-array",
                                        "file": "rpi-respeaker-mic-hat-leds",
                                        "options": {
                                                "model": "4mic"
                                        },
                                        "modes": {
                                                "idle": [{"ledIndex": 1, "red": 0, "green": 0, "blue": 0},{"ledIndex": 7, "red": 0, "green": 0, "blue": 0}],
                                                "listening": [{"ledIndex": 1, "red": 150, "green": 0, "blue": 0},{"ledIndex": 7, "red": 150, "green": 0, "blue": 0}],
                                                "speaking": [{"ledIndex": 1, "red": 0, "green": 0, "blue": 150},{"ledIndex": 7, "red": 0, "green": 0, "blue": 150}],
                                                "awaitDialog": [{"ledIndex": 1, "red": 100, "green": 100, "blue": 0},{"ledIndex": 7, "red": 100, "green": 100, "blue": 0}],
                                                "loading": [{"ledIndex": 1, "red": 10, "green": 10, "blue": 10},{"ledIndex": 7, "red": 10, "green": 10, "blue": 10}],
                                                "wakeWordActive": [{"ledIndex": 4, "red": 10, "green": 0, "blue": 0},{"ledIndex": 10, "red": 10, "green": 0, "blue": 0}],
                                                "wakeWordInactive": [{"ledIndex": 4, "red": 0, "green": 0, "blue": 0},{"ledIndex": 10, "red": 0, "green": 0, "blue": 0}]
                                        }
                                }]
                        }
                },
                user: {
                        "clexiSocketURI": "ws://localhost:8080",
                        "clexiServerId": "clexi-123",
                        "clexiConnect": true,
                        "useRemoteCmdl": true,
                        "useGamepads": true,
                        "useBluetoothBeacons": true,
                        "useBluetoothBeaconsInAoModeOnly": false,
                        "knownBluetoothBeaconAddresses": [],
                        "useWakeWord": true,
                        "autoloadWakeWord": true,
                        "allowWakeWordDuringStream": true,
                        "activeSkin": "2",
                        "activeAvatar": "0",
                        "proactiveNotes": false,
                        "autoGPS": false,
                        "embeddedPlayerSettings": {
                                "canEmbedYouTube": true,
                                "canEmbedSpotify": false,
                                "canEmbedAppleMusic": false
                        }
                },
                location: {
                        "latitude": "",
                        "longitude": ""
                },
                broadcast: {
                        "state": true,
                        "login": true,
                        "clientError": true,
                        "accountError": true,
                        "speech": true,
                        "wakeWord": true,
                        "audioPlayer": true,
                        "alarm": true,
                        "info": false
                }
        }
};

[fquirin]

Broadcaster event: {"broadcast":{"client":"alexa_chrome_app_v0.24.1","deviceId":"alexa","sepia-wake-word":{"state":"inactive","keywords":["Alexa"]}}}

This is not necessarily an error, it just says that the wake-word is currently off which can happen for various reasons. It should turn on after the next interaction with SEPIA though. Are there any other 'wake-word' errors shown?

i tried with wwRemoteUrl: http://192.168.0.64:20721/files/wake-word/porcupine/ and http://192.168.0.64/files/wake-word/porcupine/

The default URL (<assist_server>/files/wake-words/porcupine/) or no URL at all should work since the files are on the server and if you can login you can see the files as well (unless you've removed them deliberately of cause). If you really need to change the URL I think http://192.168.0.64:20721/files/wake-word/porcupine/ is the right one since the version determines the rest.

[royrogermcfreely]

it doesnt matter if i set the url to /files/wake..... or nothing.

i allways get the state "inactive" and no matter what i do, the client wouldnt reconize the wakeword.

i dont get it, cause on my phone/tablet/laptop the wakeword "alexa" is working.

do you have any other idea what i can try?

and one more question: is it possible to let the leds spin around when for example the state is "listening"? or is too complex at the state of sepia right now?

/roy

[fquirin]

Could you post your full CLEXI log after you press the client "reload" button. It should look similar to this:

Broadcaster event: {"broadcast":{"client":"o1_chrome_app_v0.24.1","deviceId":"o1","sepia-wake-word":{"state":"active"}}}
GPIO-Interface event: {"gpio":{"type":"itemSet","msgId":"SEPIA-o1-5","id":"led-array"}}
Broadcaster event: {"broadcast":{"client":"o1_chrome_app_v0.24.1","deviceId":"o1","event":{"state":"active","user":"uid1007"}}}
GPIO-Interface event: {"gpio":{"type":"getAll","msgId":"SEPIA-o1-2","buttons":[],"leds":[],"items":[{"id":"led-array","file":"respeaker-usb-array-v2"}]}}
Broadcaster event: {"broadcast":{"client":"o1_chrome_app_v0.24.1","deviceId":"o1","msg":"Reloading client."}}

I just double-checked to make sure "Alexa" with version "1.9" is actually still working ^^. Oh and btw I'm not sure if its a copy-paster error but I just saw that your settings.js that you've posted at the top has multiple same entries and repeats itself after "clexiGpioInterface"!

If you don't see any errors can you try to run the remote mic test via CLEXI command call mictest play recording (don't forget to choose message type 'SEPIA Client' and set the correct deviceId).

and one more question: is it possible to let the leds spin around when for example the state is "listening"? or is too complex at the state of sepia right now?

You are using the ReSpeaker 4mic HAT right? It's certainly possible but not without a modification of the CLEXI GPIO item file because currently you can only set one static LED configuration per state change. One would have to introduce something like a "listening" script that rotates LED states until it is stopped by a new state command. The USB mic array file works similar to this with the difference that the animation itself is hardcoded on the hardware and simply switched on and off.

[royrogermcfreely]

thats my clexi log when i call reload client:

Broadcaster event: {"broadcast":{"client":"alexa_chrome_app_v0.24.1","deviceId":"alexa","event":{"state":"active","user":"uid1007"}}}
Broadcaster event: {"broadcast":{"client":"alexa_chrome_app_v0.24.1","deviceId":"alexa","event":{"state":"active","user":"uid1007"}}}
GPIO-Interface event: {"gpio":{"type":"getAll","msgId":"SEPIA-alexa-2","buttons":[],"leds":[],"items":[{"id":"led-array","file":"rpi-respeaker-mic-hat-leds"}]}}
GPIO-Interface event: {"gpio":{"type":"getAll","msgId":"SEPIA-alexa-2","buttons":[],"leds":[],"items":[{"id":"led-array","file":"rpi-respeaker-mic-hat-leds"}]}}
Broadcaster event: {"broadcast":{"client":"alexa_chrome_app_v0.24.1","deviceId":"alexa","msg":"Reloading client."}}
Broadcaster event: {"broadcast":{"client":"alexa_chrome_app_v0.24.1","deviceId":"alexa","msg":"Reloading client."}}

when i call mictest play recording, the client is recording and speaks my words after that correctly.

ah yes the settings.js was a copy paste error.

[fquirin]

:thinking: it looks like you have 2 clients with the same 'user' and same 'deviceId', is that intentionally? If so could you close one of them (it shouldn't really matter but who knows ^^) and reboot the device once.

[bpl34567]

I have the same issue, if I use another wakeWordVersion then 1.4 - the wakeword state is inactive "state":"inactive".

When I use the default (Hey SEPIA) and Version 1.4 the state is active and I get an response if I say "Hey SEPIA", if the wakeword is inactive I don't get the acoustic response. Are there further logs to check to find a reason for that?

I am running the sepia-client on a Rasperry Pi 3 B+.

• Version 1.9

Broadcaster event: {"broadcast":{"client":"o1_chrome_app_v0.24.1","deviceId":"o1","sepia-wake-word":{"state":"inactive","keywords":["Server: Jarvis"]}}}

• Version 1.4

Broadcaster event: {"broadcast":{"client":"o1_chrome_app_v0.24.1","deviceId":"o1","sepia-wake-word":{"state":"active"}}}

[fquirin]

Ok so I think we need to use the heavy guns ^^. There is a new, experimental remote debugging mode. If you run bash setup.sh from ~/sepia-client/ you should see an option called ENABLE remote debugging mode. If you use this you will see some instructions how to proceed and you should be able to open the SEPIA client from your Desktop Chrome/Chromium browser. You can then open settings -> Hey SEPIA enable expert mode and debugging and check the error messages ... if any.

@bpl34567 when you use v1.9 don't use "Server: Jarvis" but only "Jarvis" as name, the file is already available in the default configuration.

[EDIT] I just tested again to see what happens when I use a wrong wake-word. You should see an error message like this:

Broadcaster event: {"broadcast":{"client":"o1_chrome_app_v0.24.1","deviceId":"o1","sepia-wake-word":{"state":"error","msg":"Uncaught NetworkError: Failed to execute 'importScripts' on 'WorkerGlobalScope': The script at 'http://localhost:8080/sepia/audio-modules/picovoice/porcupine-keywords/jarvos_wasm_19.js' failed to load."}}}

So I think the ww is actually loading :thinking:

[bpl34567]

@fquirin Thank you for the fast reply.

I tried around with Server: Jarvis and without. Even with the ww Dragonfly which is version 1.5 the ww is inactive.

• v 1.5

Broadcaster event: {"broadcast":{"client":"o1_chrome_app_v0.24.1","deviceId":"o1","sepia-wake-word":{"state":"inactive","keywords":["Dragonfly"]}}}

I will try the remote debugging mode and post the logs.

[bpl34567]

Remote-Debugging is working now, need a moment to realize that I have to restart the sepia-client. The Remote-Debugging is a great feature!!!

It seems that the ww listener can not be started, see the attached screenshot. I enabled the debug-log, if the debug log in the settings page is enabled is there an additional log in the filesystem (text based) or "just" the log as shown in the screenshot?

Please, let me know if I can provide you more information to fix that issue!

So, I was testing around for a while. I got the following error.

Steps:

• Edit the ww Version to 1.9 and ww to Jarvis
• Engine --> Reset
• Load

[fquirin]

Maybe this is actually really timing out 🤔. Do you feel like the client in general is a bit laggy when answering questions?

There are two things you can try. First you can reboot your SEPIA server just to make sure that the server itself isn't performing bad. Then on your client go to ~/clexi/www/sepia/www/xtensions/picovoice run the WASM download script. This will copy the files to your client local folder. Adjust your wake-word file as mentioned in the script and restart the client.

[bpl34567]

With version 1.4 it's ok. The android client is a little bit faster, but it seems to be ok for me. The response of the actions which can be used in SEPIA-Hub -> Client connections are ok.

• I downloaded the files and set 'SepiaFW.wakeTriggers.porcupineVersionsDownloaded = true;' wakeWords.js
• reboot the pi
• open ssh tunnel
• connect via Remote-Debugger (chrome) - the following error occurs:

Memory usage of the system seems to be ok, I restarted the engine manually and checked the RAM usage, it's around 25% used.

Using ww Version 1.4 and Hey SEPIA still works.

[fquirin]

This error is unfortunately a bug in web assembly and Chromium that doesn't free the reserved memory in time :-/ Something with the garbage collector. I get this now and then when I do too many mic tests switching and releasing the wake-word. The only thing that helps in this case is closing Chromium, a simple page reload is not enough ☹️. After that it should remain stable if you don't change settings.

[bpl34567]

Sorry to ask - how can I close chromium if the error occurs during the start of the sepia-client? The error occurs reproducible each time I start the engine with ww of version 1.9.

[fquirin]

Oh 😬 You are using a Rpi 3 right? What version of Chromium and Debian is this? I'll try to reproduce the problem on one of my Rpi3s

[bpl34567]

• Pi

Raspberry Pi 3 Model B Rev 1.2

• OS

PRETTY_NAME="Raspbian GNU/Linux 11 (bullseye)"
NAME="Raspbian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"

I downgraded the Chromium browser, to avoid issue with missing ASR / STT support. I used the provided script install_chromium_92_from_archive.sh.

Can I provide further information for you? Thank you again for your help.

[fquirin]

Can I provide further information for you?

Could you add the output of uname -a as well plz?

This problem is getting more and more confusing :raised_eyebrow: . Honestly I have no idea why the old 1.4 "Hey SEPIA" works and none of the other versions in your case since the 1.5 and 1.6 files are almost identical and 1.9 is even smaller. On my Pi4 I usually had an "almost" reliable way of reproducing the memory error which involved starting the wake-word then switching to microphone test and back 2-3 times ... currently this runs without ever triggering the error :man_facepalming: .

I'll flash a new SD card now and test on my RPi3.

[bpl34567]

@fquirin thank you very much for effort to try to reproduce the error.

uname -a
Linux jarvisTwo 5.10.92-v7+ #1514 SMP Mon Jan 17 17:36:39 GMT 2022 armv7l GNU/Linux

[fquirin]

A bit of progress:

• I can reproduce the problem on my Rpi3 with same OS
• In my setup v1.4, v1.5 and v1.6 are actually working
• v1.9 crashes every time with the same error :-|

Chromium 95.0.4638.78 btw, so we can rule out the browser at least. I'll experiment a bit with this setup and if I can't change anything I'll try to downgrade to RPi OS Buster (aka "Legacy").

[fquirin]

Ok, so here is what I found out. Porcupine v1.4-1.6 need around 16MB WebAssembly memory, v2.0 should start with ~60MB and may grow to ~120MB and v1.9 ... ... requires 256MB :grimacing: . This is unfortunately hard-coded during WASM-file build and throws an error if you try to reduce it via Javascript. I've created an issue in the Porcupine repo and asked them if they'd be willing to updated the old file with new settings but chances are probably low :-/.

The problem with the 1GB RPi3 (and likely even the 1GB RPi4) is that in my case it has space for around 360MB of WebAssembly memory left (probably set by Chromium in regard to free memory) and that is just a little to small to instantiate the Speex resampler + Porcupine v1.9 (and eventually the VAD). If you deactivate the resampler via "tryNativeResampling": true in microphone settings you might see one successful creation of the wake-word but it will crash on the next load.

There is some room for improvements in the SEPIA client code, but without a change of the v1.9 WASM file by Picovoice it will most likely remain broken on devices <= 1GB RAM.

I haven't tested it because my Picovoice keys are always running into the free tier activation limit :sweat_smile: (I build too many test clients ^^) but v2.0 should work.

[cap-blackbeard]

I receive the same message with my Rpi3b+. It would be great if v2.0 worked.

[fquirin]

It would be great if v2.0 worked.

Do you mean you've tried an it didn't work?

[cap-blackbeard]

Yes. But I suspect that my newly installed client cannot connect to my old server version!?

[fquirin]

It will complain a bit about missing features but should connect and work. What will fail is loading the 2.0 Porcupine files unless you change the wake-word URL to <sepia_website>/files/porcupine/ to load them from 'sepia-framework.github.io' instead (or any other URL inside your network).

[bpl34567]

@fquirin Thank you very much for your time! I will try 2.0 and let you know whether it works or not.

Nevertheless, a Raspi 4. with more RAM sounds good, too. Maybe I can use then Mary TTS, too.

[cap-blackbeard]

v2.0 works very well on the Rpi3b+.