HoloBot

Take your bots beyond Skype, Slack, Microsoft Teams and Facebook and bring them into the real world with Mixed Reality. Why simply type-chat with a bot when you could actually look at them, talk to them and listen to their answers?

HoloBot is reusable Windows Mixed Reality Unity project for Microsoft HoloLens v1, v2 and immersive VR headsets. The app acts as a holographic client for a chatbot. This 3D app lets you interact with a floating robot head using speech recognition, getting answers spoken back by the "bot" using Text-to-Speech. The commands sent to the bot are powered by the Microsoft Bot Framework and LUIS from Microsoft Cognitive Services.

• Unity version: 2018.4.12f1
• Mixed Reality Toolkit (MRTK) version: v2.1.0

To get started with HoloLens & Windows Mixed Reality development, visit the Windows Mixed Reality Dev Center. The HoloLens Developer Kit is available for sale in several countries at http://hololens.com.

Project Update

2021: I am not actively working on this given a lack of free cycles, but I do plan on revisiting this soon enough to upgrade to the latest version of Unity, test with the latest Speech SDK, and test on the latest build of HoloLens 2. New features are in planning too. If there are pressing issues, you can post them here and I will do my best to respond, but I cannot offer any guarantees at this time.

This current version is technically still a work in progress as I was migrating HoloBot to Unity 2018.4, the latest version of MRTKv2 (at the time), HoloLens 1 & 2, and use the new unified Speech SDK from Microsoft Cognitive Services. Note that builds in this branch might sometimes cause errors or have missing features given how MRTK and the speech SDK are evolving fast.

Note for users of older versions of Unity: Now that HoloBot has been upgraded to Unity 2018.4, I have archived the previous versions in a separate dev branches. Note that these legacy versions of HoloBot are only provided for archival purposes and these branches will no longer be worked on or supported:

• Unity 5.6.3p2
• Unity 2017.4

Requirements

The following plugins/packages are dependencies that must be imported manually in this project:

• JSON.NET for Unity: Download the package from the Unity Asset Store here.
• Cognitive Services Speech SDK for Unity: Download it from here

Features

• Hovering bot head (aka HoloBot) with looping ambient sound.
• Tap & hold the HoloBot to drag it to a different location, drop the hold to place.
• Gaze at the HoloBot to trigger the speech recognizer, you will hear a "ping" sound.
• Speak "commands" to HoloBot using natural language. HoloBot has only been tested with English for now.
• The HoloBot speaks back to you using Speech Synthesis (aka Text-to-Speech, or TTS).
• All sounds and speech use spatial sound that originate from the HoloBot's location in the room.
• The "brain" of HoloBot can be any public bot built with the Microsoft Bot Framework. Build your Bot using C# or Node. See bot integration instructions below.

Video Demonstration

The bot demonstrated in this video is The Maker Show Bot, also found here on GitHub. Feel free to fork the code and plug HoloBot to your own chatbot.

Instructions / Implementation Notes

• The HoloBot model and sounds come from the Holographic Academy tutorial: [Holograms 240: Sharing Holograms].(https://developer.microsoft.com/en-us/windows/holographic/holograms_240).
• HoloBot has finally been upgraded to a recent version of the HoloToolkit for Unity. See above for the specific version.
• Uses the InputManager from MRTK for Unity (prefab) for Gaze & Gesture management.
• Now using the Hand Draggable script instead of my own custom Tap to Place.
• MicrophoneManager.cs now implements IFocusable for Gaze/Input Controller events (enter/leave), which triggers the speech recording.
• Uses Text to Speech Manager from MRTK for Unity (Utilities scripts).
• It is no longer needed to use your own Newtonsoft.Json.dll in the /Plugins folder of the HoloBot Unity project since the MRTK already includes it.

Connecting your Bot to HoloBot

• Create and register your bot as per the intructions at https://dev.botframework.com. Bots can be built with C# & ASP.NET WebAPI or Javascript & Node.js. Since HoloBot uses free natural language dictation, it is highly recommended that your bot support NLP via the Language Understanding Intelligent Service (LUIS) from Microsoft Cognitive Services.
• From the Bot Connector portal, enable the Direct Line channel on your bot, and enable version 3.0 of the Direct Line API.
• Generate and copy your Direct Line secret (aka API key)
• Open BotService.cs in the /Scripts folder of the HoloBot Unity project and paste your Direct Line secret in the _APIKEY private string

Acknowledgments

I want to offer special thanks to the following people who have helped in various ways over the years: Jared Bienz, Tobiah Zarlez, Etienne Margraff, Vanessa Arnauld, Sara Nagy, Phil Cherner, Kat Haris, and the original Microsoft Mixed Reality Academy team & mentors from 2016 - especially Pat - for their awesome training, resources, patience and help.

Follow Me

• Twitter: @ActiveNick
• SlideShare: http://www.slideshare.net/ActiveNick