Context management

[bodiroga]

Hi Microft Team!

Awesome library/tool! Until now I have been playing with api.ai, wit.ai, and this kind of online intent parsers, but my plan is to use an offline open source solution to speed the processing time, improve my privacy and don't require an internet connection: "Adapt" looks really great!

I have seen that you have a branch to develop "context" capabilities within Adapt, but there's only a single commit by @clusterfudge and development seems to be halted. Is anybody working on it right now? What kind of data structure did you plan to use? It would be awesome if @clusterfudge could write what his plan was ;)

Many thanks for all your work guys, keep up with the good job!

Aitor

[clusterfudge]

Hi, @bodiroga . I'm no longer a member of the Mycroft team, and as such am not contributing to this repository anymore. I will be continuing my context work on my fork of Adapt, and will contribute upstream as progress is made (barring divergence of the forks). My schedule does not currently allow for regular development.

@penrods has recently written about his plans for conversational context within Mycroft, and he may be able to share more on those developments.

[bodiroga]

Hi @clusterfudge!

Many thanks for your fast answer and I'm sorry to hear that you are not longer working on the Mycroft team, good luck with your new projects ;)

It would be great if @penrods could write a little bit about the plan for the "context" integration and what's the general idea for the implementation. My idea is to develop something around this concept, so knowing if there has been a discussing about this topic in the Mycroft team would be awesome!

Many thanks for all the work you guys have done and best regards,

Aitor

[penrods]

Hi, @bodiroga and Sean (@clusterfudge)! My thoughts around context in Mycroft are jelling as I get to know this codebase better and think about what I had implemented in Christopher (the technology I built before Mycroft). In Christopher, there were several types of context that I will be integrating:

• Recent history context Just knowing what Skills have been invoked or interacted with recently is a powerful thing. For example, the phrase "stop it" can easily be interpreted if you know that an Alarm skill triggered a buzzer a few seconds ago. My implementation was to track not just the immediately Skill history, but everything invoked in the previous 5 minutes. The Skills in that recent list got a preview of all the utterances from the speaker, allowing them to handle phrases that wouldn't make any sense without context. That allows sensible handling of overlapping vocabulary, also. For example, "how about Thursday" could be handled by either a Weather Forecast or a Calendar skill. The recent history context allows the correct Skill to be selected.
• Conversation context The concept of a "conversation" implies some memory of what was said or done before. In Christopher I allow system responses to also include a packet of "context" info that will be delivered to the Skill the next time it is invoked. The Skill writer can use that context to better understand the utterance. I used this technique to build a sort of enhanced phone-tree system, where you can define structures of interactions and branches based on the response. It is handy when working through a multi-step procedure. But you can also use the context a little more free-form, perhaps just recalling who you sent a texts to recently so that the phrase "Tell Joe I'm going to be late" could figure out on its own that "Joe" is "Joe Smith" who you texted a few minutes ago, not "Joe Jones" or "Joe Johnson" who you haven't interacted with in months. And that you were texting his work number, not his personal phone.
• Audible/Visual context Christopher allows control of an associated display (e.g. a nearby TV). What is showing on that TV is part of the context the person is experiencing in that room. So if I pull up a recipe on that TV and have it show the preparation steps, the simple phrase "next step" can be understood. I implemented this by giving the Skill that created the current onscreen display the same preview of utterances.
• Location context Along with the words spoken, we know about the context of the location where they were said. So, for example, saying "turn on the lights" would have two different results if said to the Mycroft unit with the name "Bedroom" than they would to the unit named "Kitchen". All Skill should have access to location info when handling utterances.
• User context There are also several techniques to distinguish registered speakers, so "what is on my agenda for today" would react differently for me than for my daughter.
• Changes in presence I have a few things in Christopher that I'll be pulling into Mycroft before long. One involves using personal Bluetooth devices (Fitbit or phone) to identify who is in the room without them even saying anything. Skills will be able to register to be notified of user arrival/departures. Christopher uses this to remind me of upcoming calendar events when I walk in the house, to turn room lights on when I walk in the room, and to turn lights off when I leave the house.

One tricky part in migrating this into Mycroft is how Adapt maps utterances to intent. I don't believe analyzing the words of the utterance alone can determine the intent. Instead I think it will be necessary to match likely intent Skill targets and give them all a chance to determine how confident they are that the utterance was directed at them. Then the system can either take the highest confidence Skill response or perhaps ask the user for clarification if there are two near confidence matches resulting in ambiguity. (I hadn't resolved the ambiguity issue in Christopher yet, I just took the highest confidence which worked really well on its own.)

This is completely doable using Adapt, but requires breaking some of the expectations of Skill writers and/or adding new capabilities. For example, Skills will need something like a handle_conversation(self, utterance) function that might be invoked when they are not the target of the conversation. E.g. the LightsSkill might receive "how about tomorrow" because the user turned on the light recently and got it in the Recent list. But the phrase was really directed at the WeatherSkill which the user had invoked right after turning on the lights.

I hope that helps, feel free to discuss and give feedback!

• Steve

[clusterfudge]

Context management (or the ability to incorporate context when parsing an utterance) has been incorporated into adapt. Managing context (ie providing/observing context, pushing it back into adapt at parse time) is the responsibility of users of adapt. A sample (in memory) context manager implementation has been provided, but can be replaced by anything that is api compatible.

https://github.com/MycroftAI/adapt/commit/01bf0e969beb4c197eea18c79786a0183620b5bc