Conversation / Dialog support

[imraansarwar]

**rasa NLU version** (e.g. `0.7.3`): **Support for Conversations or dialog and context for NLU use cases**: I know this question has been asked multiple times, but I would like to emphasis on its importance and impact. The uses-cases of NLU with respect of chatbots, etc., if not all, most of them do have a need for conversation/dialog and context support to work. Without having this functionality, Rasa NLU may not be able to suffice for likes of Wit.ai, Api.ai, etc. I think this should be taken up on priority to take-up on the competition. Thanks

[PHLF]

I don't think that should be part of RASA's goal IMO: NLU is by itself a very large and complex topic to handle. But you can use a specific framework for that like botkit.

Here's what you can do:

• use RASA NLU http server as a middleware in botkit to get the intents and entities
• handle the corresponding conversation with botkit

[imraansarwar]

I see it from two perspective, of course that's my point of view and you may differ from it.

1. Dialog agents are part of NLU
2. From competition point of view, or from Rasa-NLU's selling pitch (below). wit.ai, api.ai being the competitor in this space have conversational, dialog capabilities built into them. Rasa NLU: Open-source language understanding for bots. A drop-in replacement for popular NLP tools like wit.ai, api.ai or LUIS

I think this is one recipe for success in NLU in bot space.

P.S. I have used botkit and a great fan of it. There's already a bokit middleware for rasa https://github.com/howdyai/botkit-rasa

[PHLF]

From a pure design perspective you can build a conversational agent without any NLU framework: using pattern matching for instance. This emphasizes the fact that handling conversations and NLU are two different things. I understand that you want better integration of these two components like WiT or API.ai can propose. But this has a cost and you can't really compare Google or Facebook's engineering resources with Lastmile's ones. Moreover if you look at Microsoft' strategy in terms of chatbot development, LUIS is not meant to handle conversations: this is the role of the BotFramework where you can use LUIS as the NLU framework to enhance your bot (again these are two separate components). From this perspective, I think RASA is more comparable to LUIS than Wit or API.ai. Again, this is just a personal opinion 😉

[nhatkhoa]

From my opinion, in the fact, context changed lead to meaning changed. So context is a part of NLU.

[PHLF]

Define what you call "context" here, as the intent/entities model used by RASA NLU is not meant to handle the dialog context (if that's what you mean): inputs are processed independently. Now if you want to build a NLU model tracking dialog states, this is another topic well beyond the scope of RASA at the moment. Have a look at The dialog state tracking challenge 2 and 3

[juanvillegas]

I agree with @PHLF here. The sole responsibility of the library has to be taking a phrase and giving back it's meaning (intents, entities, etc). I believe the confusion comes from comparing it with services like api.ai, where they have closely integrated the NLU service with a nice interface to design dialogues. Still, even in those services the modules are separated.

[imraansarwar]

Right, the confusion comes when it is placed along with wit.ai/api.ai. Its all over the place on the home page of the site. Probably it should reflect the scope and domain of rasa correctly. IMHO NLU without context is as good as NLP. Can we not do it directly using Spacy or NLTK? What are the cons of doing it in Spacy/NLTK? Well, I guess you can stick to the one-shot parsing of the input, however this should be reflected clearly on website to avoid confusion. Perhaps a comparison showing benefits of using this over other similar frameworks except (being on-premise of course)

One particular endorsement from Andrew Magliozzi, CEO of AdmitHub add to the confusion too.

If you're serious about conversational software you can't rely on 3rd party APIs. Our use case is extremely demanding and rasa handles it well.

[PHLF]

IMHO NLU without context is as good as NLP

NLU is a subfield of NLP. With RASA when we talk about NLU, we talk about the Intent/Entities model. Intent/Entities is not meant to handle conversations: it allows you to build transactions from natural language (user's input is the request and the entities are the related parameters) which is great to build natural language interface for a service. It looks quite simple, yet it is tricky enough to make it work with great precision. It uses lower level NLP tools like PoS, words embedding... to build features for intents and entities classifier.

Can we not do it directly using Spacy or NLTK? What are the cons of doing it in Spacy/NLTK?

Spacy and NLTK are NLP framework, they stand at a lower level than NLU framework like RASA. They allow you to achieve a lot of different things, so before thinking in term of framework think in term of model: what would be the best way to model a conversation according to my use case? Then you could pick the tools that are the most relevant to build your NLU model. It looks like you are more interested in chatbots from a research perspective. You can have a look at Alan's blog for a great and simple introduction, Denny's Britz blog for an introduction on Deep Learning for chatbot or Maluuba's blog for advanced ideas about dialogue systems.

If you're serious about conversational software you can't rely on 3rd party APIs. Our use case is extremely demanding and RASA handles it well.

I think this sentence is more related to a business point a view: potentially all user inputs may be considered as private data and you don't want 3rd party API to handle that, whereas RASA can be deployed on-premise (as you stated) allowing you to have complete control over your data.

I hope I don't look like too pedantic, but I feel your issue emphasizes the fact that building conversational agents is a confusing subject.

[amn41]

Thanks everyone for the lively discussion. I think @imraansarwar raises some valid points that NLU isn't the only thing you need for building conversational software, but from our perspective that's outside the scope of rasa NLU. We have some new stuff in the pipeline though which I think you'll like a lot :) And of course you could use lower level tools to build your own NLU system if you prefer. I hope people who go that route get some inspiration from how rasa NLU works, and if they find out something cool, contribute it back to this project. A lot of excellent contributions to this project have already come from the community. Conversational AI is hard! It makes a lot of sense to collaborate on open source projects.

[imraansarwar]

Excited to use rasa NLU and will definitely look forward to contribute wherever I can. Keep up the good work.

[amn41]

FYI - email hi@rasa.ai if you'd like to try out the dialogue engine