Get rid of NLP in user dialogues

[lexaux]

Background

Instead of building forms, UI and everything else, we decided to go with text-based interface in the form of chatbot. It gives a few clear benefits:

1. No need for identity/authentication. This is on the platform. So we don’t need: login, logout, expiration, social connections, profile form, forgot password, entire user management on the admin side.
2. No need to actually spend time on frontend/UI. Telegram can already handle links, commands, messages, display preview, store history (so one can get back to it).
3. Added perks like group chat pilots. Within the group chat we get all the right people from client side and from freespeech, and observe how they use the system, make intrusions when needed, help, demonstrate, announce updates and so on.

All this with no efforts in programming for our side.

We decided to initially go with Natural Language Understanding as it seemed easier. The model is hosted in Azure. lib/chat.py around def intent and api/chat.py are good places to start looking at.

However, we learned that NLP in this place is not the best thing to do because:

1. Anyway people need to teach themselves commands and formats the chatbot receives (so we can do plain regex match for example)
2. Freetext as freetext - e.g. get some random person in the street and let them ask bot to translate something - does not work out of the box, since language is complex and people don’t know what to mention. So it boils to above.
3. Sometimes the NLP is inaccurate, and the way we have trained it (100% match needed, old reset intent) lead to unpredictable results and people trying their way around.
4. We have seen a few cases where Azure NLP engine failed the requests, aborting user’s command.

Scope

Scope of this ticket is to get rid of NLP in conversations with users. Instead, use one of two options:

1. Full match. Develop a simple command language following what we have, and expect a full match when people type that in. It can be, for example, a regex match. Command examples are available after /help in bot.
2. Telegram commands. Bots do support commands starting with /. However, since we sometimes need parameters, this becomes a dialogue potentially. So, I would see this as following:
   1. If the user types in the short command like /transcribe, the bot would need to ask further questions to understand source language and get video URL, before it can start. This is handy since it narrows decision field for user and lets us use buttons in Telegram UI.
   2. If the user types in full command like /transcribe https://youtube.com/aaabbb in English using Machine C then we have a full match and don’t need anything else.
   3. If we decide to implement telegram commands, then there is a state management (and storage) to allow dialogues to clarify missing parameters. Keep in mind that we have no shared state sofar, so we would need to introduce something in the cloud.
   4. Also, if we implement commands, we can get rid of the read-all privacy setting on the bot. Currently, we had to give it a permission to read all messages in all chats it is in, so that it can react to mentions. Vanilla telegram way is to use bot commands and they are passed well to bot in all chats. Read more about [Privacy Mode](https://teleme.io/articles/group_privacy_mode_of_telegram_bots#:~:text=A bot running in privacy,to the bot's own messages)

With commands it is easier in a way, but we need to keep the interface compatible with other chat platforms. For example, we thought of expanding to Whatsapp for Business.

Definition of Done

I would expect this done if we:

1. Get rid of Azure NLP dependency in chat
2. All the tests in tests/test_ask.py do pass without modification
3. Changes consolidated around lib/chat.py and api/chat.py
4. If we decide to do commands, then they work both in DM and group chats with the bot.

[astaff]

Looking good. Few thoughts.

1. Current NLU implementation makes it hard to add new commands or modify existing ones with adding new values for arguments. Even adding a new language would require re-training or a bit of hacking to try to fit new functionality into the dialogues that can be understood by the existing NLU model.
2. Running NLU API endpoint in azure costs us $50/mo.
3. I encourage to approach removing of NLU and adding support for commands and/or stateful conversations as two separate work items.

[lexaux]

Let's descope this then to a simpler thing - get rid of NLU, implement similar freetext interface. Commands -> separate task.