search

SEPIA-Framework / sepia-docs

Documentation and Wiki for SEPIA. Please post your questions and bug-reports here in the issues section! Thank you :-)
https://sepia-framework.github.io/
238 stars 16 forks source link
chatbot chatbot-framework digital-assistant nlu raspberry-pi sepia speech-recognition virtual-assistant voice-assistant vui wiki

readme

Welcome to the documentation page for SEPIA

Here you will (hopefully) find everything you need to know to get started with SEPIA.

S.E.P.I.A. Logo

Overview of SEPIA ecosystem (note: some parts are still in the dev branches).
For image icon attributions please check the homepage

Downloads

Wiki, Blog & News

Checkout the Wiki for detailed descriptions: S.E.P.I.A. Framework Wiki
Visit Twitter or Mastodon for the latest news: S.E.P.I.A. Twitter Feed | S.E.P.I.A. Mastodon Feed
Visit the blog for summaries and guides: S.E.P.I.A. Blog

Intro

S.E.P.I.A. is an acronym for: self-hosted, extendable, personal, intelligent assistant. It is a modular, open-source framework equipped with all the required tools to build your own, full-fledged digital voice-assistant, including speech recognition (STT), wake-word detection, text-to-speech (TTS), natural-language-understanding, dialog-management, SDK(s), a cross-platform client app and much more.
The framework consists of several, highly customizable micro-services that work together seamlessly to form the SEPIA Open Assistant. It follows the client-server principle using a lightweight Java server and Elasticsearch DB as "brain" and a Javascript based client that works as smart-speaker, smart-display, mobile assistant app or whatever smart-device you come up with :smiley:. All components work on Linux, Windows and Mac and have been optimized to even run smoothly on a Raspberry Pi :relieved: :robot:.
Out-of-the-box SEPIA currently has smart-services for: news, music (radio), timers, alarms, reminders, to-do and shopping lists, smart home (e.g. using open-source tools like openHAB), navigation, places, weather, Wikipedia, web-search, soccer-results (Bundesliga), a bit of small-talk and more. To realize your own ideas you can use tools like the SEPIA SDK and the code editor integrated into the SEPIA Control HUB to build services or write custom HTML widgets :man_mechanic::woman_scientist:!

Architecture

The SEPIA Framework consists of 2 core parts: The SEPIA Client and the Assist-Server.

SEPIA Client: The user interface that handles voice, text or touch interactions and manages the "dialog" with the SEPIA server. Server responses can be presented as text (chat), graphical elements (cards, buttons) and/or sound including speech synthesis (text-to-speech) and music (media-player). The client usually takes care of the speech-recognition (on-device or via SEPIA STT server) to transform voice into text and can even listen to wake-words like Hey SEPIA (thanks to Porcupine by Picovoice). There are clients for the browser, Android, iOS and a DIY version that even works "headless" for example on a Raspberry Pi.

Assist-Server: The "brain" of SEPIA that receives requests from the client via the HTTP REST API and takes care of the natural-language-understanding (intent and NER), conversation flow, smart-service integration (like a to-do list or news service), user-accounts, Text-to-Speech (TTS) and more. The Assist-Server can run on it's own hardware for example on SBCs like a Raspberry Pi 3 or parallel to the client on more powerful systems (RPi4, desktop PC ect.).

Because speech-recognition is a very delicate topic for multiple reasons (privacy, accuracy, performance, control etc.) the SEPIA Framework includes another major component: The Speech-To-Text (STT) server.

SEPIA STT Server: An open-source server for real-time speech-recognition that runs on most systems (x86, ARM), including Raspberry Pi and supports custom, dynamic ASR models (thanks to great tools like Kaldi, Vosk or Zamia speech).

Other notable components of the SEPIA Framework are the Control HUB to manage server, "headless" clients, Smart Home and more, the WebSocket server for multi-channel chats and duplex data transfer, the Teach-Server to store custom commands and a Java SDK to create powerful custom services.

Languages

Currently SEPIA works in German and English with basic support to create custom commands in other common languages. Some services like news and soccer-results are optimized for German meaning you will get an answer in English but might still see a mix of English and German news outlets or soccer results for the Bundesliga. The smart-services are constantly improving though and you can easily edit the list of outlets yourself.

Quick-start (for users)

To use S.E.P.I.A. your personal, digital, open-source voice assistant you need 2 things:

  1. Access to a S.E.P.I.A. server. This can be your own one, running e.g. on a Raspberry Pi or your Windows/Mac PC (see below) or you can find an open one hosted by a friend or a company (Note: we are currently not hosting any public servers).
  2. One of S.E.P.I.A.'s client apps, e.g. the web version: https://sepia-framework.github.io/app/ (hosted on your server as well) or the official Android app

To connect to a custom server simply open the app, change the "hostname" in the log-in screen and restart the app. A typical hostname could be the IP of the server, "raspberrypi.local", "my-server.example.org/sepia" or simply keep "localhost" (for test-servers on the same machine).

Quick-start (for makers)

Basic steps to install the server:

Instructions and an (almost) automatic installation script for Raspberry Pi can be found -HERE-

Instructions for the installation of the S.E.P.I.A. server stack on Linux, Windows or Mac can be found -HERE-

Questions and bug-reports

If you have any questions, need help or want to report a bug please go here or start a discussion here.

API keys for services

Some services integrated in SEPIA require an API key to run properly (e.g. navigation/reverse geo-coding). Find out how to get them (for free) here.

Final notes

If you run your own server and decide to open it to the public or to your friends please make sure it is properly secured and inform the users about your data privacy policy since you are operating a database with potentially sensitive, personal information.