search

Dimeiza / Assistants

Apache License 2.0
25 stars 11 forks source link

readme

AssistantControl

Tools to assist when run Alexa and Google Assistant at the same time on Raspberry Pi(with Respeaker 2mic/4mic/6mic).

Notice

This tool is an eternal PoC(Proof of Concept). You can only use this tool at your own risk. And I strongly recommended that this tool only use a power user. If you are a newbie of raspberry pi and linux, I recommend that you don't use this tool, and you use both assistants only according to official procedures.

This prototype only written for me. Therefore, you can't expect support and guarantee from me. But if you still hope to use this prototype, you have to read official document of both assistant carefully.

https://github.com/alexa/avs-device-sdk

https://github.com/googlesamples/assistant-sdk-python

I expect a person that use this prototype have enough knowledge of both Assistant.

If you have a trouble to execute this prototype, At first, you have to confirm that this problem is occured even if you follow official installation procedure. If so, You should send a issue to official team directly. It is earlier than sending issue me to resolve your problem.

In principle, I don't receive a question how to execute Assistants in your environment. But if you find a point of improvement, you can share it in issue in this repository. I can't promise to improve, but I will read your issue.

Description

This tool/scripts features are:

Demo

Rapsberry Pi Zero with ReSpeaker 2-Mics Pi HAT.

https://www.youtube.com/watch?v=P66bfCi45wE

Raspberry Pi 3B with ReSpeaker 4-Mic Array

https://www.youtube.com/watch?v=WsatqjlLYls

Raspberry Pi 3A+ with ReSpeaker 4-Mic Array

https://www.youtube.com/watch?v=wH6oOzjj26o

Raspberry Pi 3B With ReSpeaker 6-Mic Circular Array kit for Raspberry Pi

https://www.youtube.com/watch?v=ja3NCrgnNSk

VS.

If you only want to run both assistant simultaneously and you don't have ReSpeaker, you can use AssistantPi instead of this tool.

Requirement

You need:

  1. A Raspberry Pi. You can choose one in following.

  1. SD card with a flesh install of Raspbian Stretch

  2. ReSpeaker. You can choose one in following.

  3. USB Sound Output(I recommended: if you use Respeaker 4-Mic).

Install

Alexa setup

I recommend that you refer avs-device-sdk installation guide carefully. Following procedure is similar to it.

  1. Boot with fresh Raspbian Stretch.
  2. Clone this repository.
git clone https://github.com/Dimeiza/Assistants.git

  1. Complete Alexa developer registration.

  2. Into Assistants folder, and run this command.

cd Assistants

  1. Get your config.json from Amazon Developer Portal(see https://github.com/alexa/avs-device-sdk/wiki/Create-Security-Profile#create-a-security-profile) and copy it to the "Assistants" folder.

  2. Run the setup script.

bash setupAlexa.sh
  1. Wait to finish script.

Google Assistant setup

  1. Complete Google Assistant registration.

  2. Run this command.

bash setupGoogleAssistant.sh
  1. Follow the prompt instructions to input registered information.
Your credentiail JSON filename:
Your Google Cloud Console Project-Id:
Device Model ID of your device:
  1. Wait to finish script.

Respeaker setup

  1. Run this command
bash setupRespeaker.sh
  1. After script finished, configure SPI to be enable.
1. Run "sudo raspi-config".
2. Select "5 Interfacing Options".
3. Select "P4 SPI".
4. Select "Yes".
  1. set .asoundrc.

set .asoundrc in your home folder according to the Respeaker you have.

No copy needed. check that .asoundrc doesn't exist in your home folder.

ls -la ~/

Execute this command in Assistants folder.

cp misc/asoundrc.4mic ~/.asoundrc

check that .asoundrc exists in your home folder.

ls -la ~/

And update a hardware that is used pcm.dmixer in .asoundrc to match your sound device(USB Sound Output).

pcm.dmixer {
        type dmix
        ipc_key 353289
        slave {
                pcm "hw:2,0" # update this line to match your sound device(USB Sound Output)
                period_time 0
                period_size 1024
                buffer_size 4096
        }
        bindings {
                0 0
                1 1
        }
}

Execute this command in Assistants folder.

cp misc/asoundrc.6mic ~/.asoundrc

check that .asoundrc exists in your home folder.

ls -la ~/
  1. Reboot.

AsssitantControl setup

  1. Run This command.
bash setupAssistantControl.sh
  1. Modify AssistantControl/AssistantControl.py as following to match your ReSpeaker.
from mic_hat import pixels
from mic_hat import alexa_led_pattern
from mic_hat import google_home_led_pattern
from mic_hat_4 import pixels
from mic_hat_4 import alexa_led_pattern
from mic_hat_4 import google_home_led_pattern

Usage

At first, I recommend that you use this tool with terminals.

  1. Open Terminal and run these commands.
cd Assistants
./startsample.sh

After several seconds, Alexa is booted. At first time boot, Alexa may appears message for authentication(URL and pass code). You have to finish authentication before you use Alexa.

This Alexa can control with keyboard. If you needed, you can change a language Alexa uses.

  1. Open 2nd Terminal and these commands.
cd Assistants
./startGoogleAssistant.sh

After several seconds, Google Assistant is booted. But this Google Assistant can't control standalone. If you need a language Google Assistant uses, You can change it with using Google Assistant App with your smartphone.

  1. Open 3rd Terminal and these commands.
cd Assistants
./startAssistantControl.sh

  1. After several seconds, message stop.Then, say "Alexa" to raspberry pi. If LEDs light up, tell Alexa your instruction. After response of Alexa, LEDs light down.

    • (In this version, first several responses to "Alexa" may not work well.)

  2. Next, say "Snowboy" to raspberry pi. If LEDs light up, tell Google Assistant your instruction. After response of Google Assistant, LEDs light down.

    • The default wake word of the Google Assistant for this tool is "Snowboy". According to the procedure described below, you can change it.

wake word changing procedure

  1. Make wake word

This tool uses Snowboy. You can make original wake word in Snowboy official site. Read the official Snowboy website for how to make wakewords.

If you make a wake word, You can download wake word file(.pmdl)

  1. set wake word

wake word setting of this tool is in startAssistantControl.sh.

python AssistantControl.py resources/alexa.umdl resources/snowboy.umdl

wake word files are set as arguments of AssistantControl.py. First argument is wake word of Alexa, and second argument is wake word of Google Assistant. You can update these arguments to change both wake word.

For example, you want to change GoogleAssistant wake word and you already make wake word file as "HeyGoogle.pmdl".

You should copy "HeyGoogle.pmdl" to Assitant/AssistantControl/resources. And You should change startAssistantControl.sh as following.

python AssistantControl.py resources/alexa.umdl resources/HeyGoogle.pmdl

Restart startAssistantControl.sh and try to say new wake word.

Customize assistants behaviour

　You can customize:

　These configuration is set in AssistantControl/AssistantControl.yaml.

Assistants:
  Alexa:
    Model: resources/alexa.umdl
    Sensitibity: 0.6
    Callback: alexa_callback
  GoogleAssistant:
    Model: resources/snowboy.umdl
    Sensitibity: 0.6
    Callback: google_callback

for example

　If you want to use this prototype Alexa only, simply remove an entry of Google Assistant.

Assistants:
  Alexa:
    Model: resources/alexa.umdl
    Sensitibity: 0.6
    Callback: alexa_callback

　If you want to decrease a wakeword sencitivity, update value of "Sensitivity".

Assistants:
  Alexa:
    Model: resources/alexa.umdl
    Sensitibity: 0.5
    Callback: alexa_callback
  GoogleAssistant:
    Model: resources/snowboy.umdl
    Sensitibity: 0.4
    Callback: google_callback

　Don't update "Callback" entry, It is stlongly related an internal behaviour of script.

Install as a service / Uninstall

If you success that you run this tool with terminals, you can install this tool as a service.

sudo bash InstallService.sh

After finishing script, Reboot.

If you want to uninstall service, run following.

sudo bash UninstallService.sh

(Experimental) SmartDisplay using MagicMirror (Raspberry Pi 4 Recommanded)

Now, Assistant Control outputs HTML that assistants(Alexa or Google Assistant) returns.

So if you install MagicMirror in Pi and configure it to show these HTML, you can get a PoC like a smartdisplay powered by MagicMirror.

  1. Install MagicMirror.

https://magicmirror.builders/

Installation & Usage

https://docs.magicmirror.builders/getting-started/installation.html

Follow above documentation to perform "Manual installation" and test MagicMirror.

  1. Install MagicMirror modules.

This PoC needs following modules of MagicMirror.

https://github.com/AgP42/MMM-SmartWebDisplay

https://github.com/AgP42/MMM-Remote-Control

https://github.com/juzim/MMM-Api

Follow installation procedure in above sites to install modules.

  1. Create symbolic link to HTML that Assistants outputs.

Execute this command in MagicMirror/modules/MMM-SmartWebDisplay/ folder.

mkdir public
cd public
ln -s /var/tmp/Assistants.html Assistants.html

These commands create a link between the HTML output by the assistant and the HTML displayed by MagicMirror.

  1. Add modules to config.js.
var config = {
        ...
    modules: [
            ...
        {
            module: 'MMM-Api'
        },
        {
            module: 'MMM-SmartWebDisplay',
            position: 'middle_center',  // This can be any of the regions.
            config: {
                // See 'Configuration options' for more information.
                logDebug: false, //set to true to get detailed debug logs. To see them : "Ctrl+Shift+i"
                height:"100%", //hauteur du cadre en pixel ou %
                width:"100%", //largeur
                updateInterval: 0, //in min. Set it to 0 for no refresh (for videos)
                NextURLInterval: 0.5, //in min, set it to 0 not to have automatic URL change. If only 1 URL given, it will be updated
                displayLastUpdate: true, //to display the last update of the URL
                displayLastUpdateFormat: 'ddd - HH:mm:ss', //format of the date and time to display
                url: ["http://magicmirror.builders/", "https://www.youtube.com/embed/Qwc2Eq6YXTQ?autoplay=1"], //source of the URL to be displayed
                scrolling: "no", // allow scrolling or not. html 4 only
                shutoffDelay: 10000 //delay in miliseconds to video shut-off while using together with MMM-PIR-Sensor 
            }
        },
        {
            module: 'MMM-Remote-Control'
        },
        ...
    ]
    ...
}
  1. Set "url" attribute in MMM-SmartWebDisplay config.

MMM-SmartDisplay can specify the page to display by setting the "url" attribute.

So, specify it following.

url: ["./modules/MMM-SmartWebDisplay/public/Assistants.html"], //source of the URL to be displayed
  1. Configure display position.

With default configuration, look and feel is not good for smart mirror.

You probably need to configure a position of MMM-SmartWeb Display. you can configure it by addition custom css as /css/custom.css.

You can refer custom.css and config.js that I used for these configuration(in misc folder).

  1. Start MagicMirror.

After these configuration, start MagicMirror.

npm run start
  1. Start Assistants.
./startsample.sh
./startGoogleAssistant.sh
./startAssistantControl.sh
  1. Call Assistants.

If you can succeeded these settings, this PoC beheviour as following.

SmartDisplay PoC

Known Issue

Build an Alexa with Raspberry Pi3 A+

When build Alexa with Raspberry Pi 3A+ , Build process sometimes stop in progress.

If you find it, You probably solve this issue by increasing swap memory.

https://www.bitpi.co/2015/02/11/how-to-change-raspberry-pis-swapfile-size-on-rasbian/

After increasing swap, try build again.

Issue of Google Assistant

There may be noise on Google Assistant's response when you use ReSpeaker 4-Mic + Raspberry Pi 3 audio output(without USB soundcard)

In this case, I recommend that you use PulseAudio with USB soundcard.

If you do so, check sound configuration before you run both assistants.

SoundDevice Check

  1. On Raspbian Desktop, Open menu and choice "Sound & Video" -> "PulseAudio Volume Control".

  2. Click "Configuration" Tab, and check that "seeed-4mic-voicecard" profile is "Multichannel input".

  3. If you use USB Sound card, check "bcm2835 ALSA" profile is "Off" and "USB Audio Device" profile is "Analog Stereo output".

Unsynchronized status between AssistantControl and both assistant

It may occur that a status of AssistantControl unsyncronize with Alexa or Google Assistant. (For example, Alexa finished speaking but AssistantControl may not sense next wakeword)

In this case, this problem may be solved if you execute this command.

sudo rm /dev/mqueue/*

Google Assistant in Pi Zero with Respeaker 2-Mic stops sometimes during speak response.

You may be able to cope with this problem by following procedure(But if you can't understand a mechanism of following procedure, I don't recommend it.You should try other method).

  1. Set .asoundrc.
    • You can refer asoundrc of this repository(misc/PiZeroRespeaker2) and edit your .asoundrc, or you can copy it to your home folder.
    • Be careful DeviceNo in .asoundrc. This file has been set it "hw:0,0". You have to check DeviceNo of your Respeaker 2-Mic in your PiZero by executing these command, and adjust it. 
      arecord -l
aplay -l
  2. Overclocking
    • set /boot/config.txt for overclocking. You can refer or copy config.txt of this repository(misc/PiZeroRespeaker2).
  3. Don't use VNC.
    • I recommend that you use ssh or directly to operate Pi Zero instead of using VNC.

In this condition, Try to run Google Assistant and AssistantController, You may can be stable Google Assistant.

Even if I investigate, I haven't found a root cause of this problem. There is a possibility that Google Assistant(or sound driver it uses) can't process response sounddata properly because Pi Zero don't have enough CPU power or network capacity.

dependency

This tool depends these software.

Author

dimeiza