S.A.R.A.H. is an OpenSource client/server framework to control Internet of Things using Voice, Gesture, Face, QRCode recognition. It is heavily bound to Kinect v1 and v2 SDK.
This project contains C# Client for SARAH. And will communicate with NodeJS Server for SARAH.
DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
Version 2, December 2004
Copyright (C) 2012 S.A.R.A.H. <sarah.project@encausse.net>
Everyone is permitted to copy and distribute verbatim or modified
copies of this license document, and changing it is allowed as long
as the name is changed.
DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
0. You just DO WHAT THE FUCK YOU WANT TO.
This program is free software. It comes without any warranty, to
the extent permitted by applicable law. You can redistribute it
and/or modify it under the terms of the Do What The Fuck You Want
To Public License, Version 2, as published by S.A.R.A.H. See
http://www.wtfpl.net/ for more details.
Be Warned: some dependencies may require fees for commercial uses. Kinect XBox 360 can ONLY be used for development purpose
SARAH v4.0 Client is built on an AddOns Architecture sharing common properties. Each AddOn:
{name}.enable
propertyClient's Addons are stored in AddOns folder and have no relation with SARAH plugins. The idea is to sandbox features and library into subprojects.
This page describe Client's Core/AddOns specification. SARAH developers should see :
UNDER CONSTRUCTION
This AddOn use EmguCV to Detect and Draw blobs in camera frame according to color, shape, features, ...
SARAH provides NBody: an abstract representation of Body / Skeleton implemented by Kinect 1, Kinect 2, and other computer vision tools.
This AddOn draw body parts, joints and relevant areas.
gesture
parameter to start/stop recognitionThis AddOn implements logging features to output relevant data about SARAH:
; Path to log file
log-file=AddOns/debug/${shortdate}.log
; Path to recognized command
dump=AddOns/debug/dump/${date}.wav
; Path to audio response
voice=AddOns/debug/voice/${date}.wav"
This AddOn use EmguCV to perform Face Detection and Face Recognition. In a Nutshell:
face
parameter to start/stop recognition Choose face detection and recognition algorithms. See also CodeProject article.
; Cascade for Face detection
HaarCascade=AddOns/face/haarcascades/haarcascade_frontalface_default.xml
; Algorithm confidence
Eigen_Threshold=0
Fisher_Threshold=0
LBP_Threshold=30
This AddOn use GoogleSpeech API to perform Speech2Text on speech recognitions.
Since 2014 Google narrow it API v2 to 50 req/day ! An API Key MUST be provided to the plugin (See G+ tutorial and explanations).
This plugin handle HTTP communication with SARAH Server. It provides to other AddOns the ability to send GET/POST/FILE request with a custom token.
[http.remote]
; Remote address of NodeJS server
server=http://127.0.0.1:8080
[http.local]
; Local HTTP server port
port=8888
temp=AddOns/http/temp/
UNDER CONSTRUCTION
This AddOn use EmguCV or nVLC to process frames of IP Camera
This AddOn use Kinect SDK v1.8 to start all connected Kinects v1 XBox360 or Windows.
The motion detection stops as much as possible stream when there is no motion. Properties allow control of streams: Audio stream only or custom fps.
This AddOn use Kinect SDK v2.0 to start one connected Kinects v2 for Windows.
The motion detection stops as much as possible stream when there is no motion. Properties allow audio stream only.
This AddOn use Microphone to start a SpeechEngine with given confidence.
; Confidence of recognition (from 0 to 1)
confidence=0.7
; Define the audio device index (0 is the default device, see logs)
device=0
This AddOn handle all OS specific actions
run
and runp
parameter to run a processactivate
parameter to foreground a processkeyText
parameter to simulate key presskeyUp
parameter to simulate key upkeyDown
parameter to simulate key downkeyPress
parameter to simulate key pressrecognize
parameter to regognize path This AddOn use the speech audio stream to compute voice pitch and update ProfileManagers.
This AddOn manage profile for people playing with SARAH.
This AddOn compute color frame of each device to draw most prominent color.
UNDER CONSTRUCTION
This AddOn start an RTP Client on a given Thread to listen to inbound stream
Can be called on RaspberryPi using ffmpeg
ffmpeg -ac 1 -f alsa -i hw:1,0 -ar 16000 -acodec pcm_s16le -f rtp rtp://192.168.0.8:7887
avconv -f alsa -ac 1 -i hw:0,0 -acodec mp2 -b 64k -f rtp rtp://{IP of your laptop}:1234
Or using Kinect on windows
ffmpeg -f dshow -i audio="Réseau de microphones (Kinect U" -ar 16000 -acodec pcm_s16le -f rtp rtp://127.0.0.1:7887
; Confidence of recognition (from 0 to 1)
confidence=0.6
; Define the RTP Port
port=7887
This AddOn manage speaker to play audio stream:
speaking
parameter write "speaking" if client currently speakingnotts
parameter stop speakingstop
parameter stop playing audioplay
parameter play given audio; The device index to play on (use -1 for default device)
device=-1
; Volume level of device
volume=50
; Timeout in seconds to stop the player
timeout=480
This AddOn build a Speech Engine for each provided audio stream (Kinect, Microphone, ...).
bot.name
must match an improved confidencebot.name
is optional with user is engagedlisten
parameter start/stop listeningcontext
parameter set the contextgrammar
parameter set the XML of a grammarsentences/tags
parameter for AskMeasknext
parameter to call tts on a ruleThis AddOn perform text to speech according to given actions:
tts
parameterUNDER CONSTRUCTION
This AddOn provide a window (and a menu item) to display other AddOns data.