Closed neilnelson closed 8 years ago
tanay1337
commented
8 years ago We can definitely learn and adapt from Jasper's source code, considering the fact that the project is MIT licensed. Speaking abstractly, our goal is not to compete with Jasper/other VAs or merge with any other project, this is and will remain an independent project :D Afterall, our goal is to improve Melissa. Speaking of improvements, which modules of Jasper do you have in mind for implementing in Melissa?
neilnelson
commented
8 years ago There is a lot of material here but the idea is to be comprehensive enough to provide a credible adaptation route. Roughly, the order of effort may be:
populate.py looks like a useful addition that when run will ask the user to provide detail for Melissa's profile.yaml.
Create Melissa web-pages similar to those for Jasper. An easy, initial option is to copy the Jasper pages that seem useful (or all) and make changes that focus on Melissa's capabilities.
Create a Melissa forum, as has been done for Jasper, at Google Groups.
Replace Melissa's brain.py and related modules with Jasper's to allow dynamic module loading and immediate inclusion of Jasper's action modules.
This is a significant change but one that is not particularly hard to do.
This also allows the inclusion of Jasper's many current and possible future Third Party Modules.
Melissa's current action modules only slightly overlap those from Jasper and can be easily adapted for Jasper's dynamic loading.
Inclusion of Jasper's standard action modules may required additional software installation and setup. The Third Party Modules certainly will. However, additional modules, which improve and expand Melissa's capability will commonly require additional installation and setup. This is an area that will likely need further thought.
A threading discussion is added at the bottom. Threading is not immediately useful but it seems a likely path in the future.
A Melissa GUI, noted in an earlier issue, is gaining in importance. A very brief search brings up Kivy as a possibility.
These links show a way to create the pages on GitHub.
https://pages.github.com/
https://help.github.com/articles/creating-project-pages-manually/
http://jmcglone.com/guides/github-pages/For the first rough cut we might just copy the Jasper pages, they appear to all be here, and trim them to what makes sense for us. It may be easy to come up with our own art and layout that would then replace those components in the existing pages. The developers may have art and layout directions in mind.
populate.py looks like a useful addition that when run will ask the user to provide detail for Melissa's profile.yaml.
Some of the software additions on the Configuration page may be useful. The components of possible use could be prioritized for later work.
Melissa's usage page would be merged here. Perhaps we would have a demonstration section for each of Melissa's actions.
Spotify seems against Jasper's Contributing to Jasper page, Philosophies, Item 2. But it illustrates that services of any kind might be added and the trick to that would be to provide those options either with populate.py or on a configuration page where options could be selected when desired.
This is a very interesting page talking about the design of action modules. It looks like brain.py is dynamically loading all the modules and while doing so gets the key words, the words used to cause an action, from a
WORDSlist in each module.In the future we may be adding phrases and sentences going in the direction of Api.ai, and then action descriptions as I mentioned for ConceptNet. The idea here is that this variety of text for an action would likely become separate from the module and maintained using a screen that would update JSON or YAML files, or even a database such as SQLite. I just now see that Melissa is using an SQLite database, memory.db.
The
isValid(input)method of finding which module applies to the user's expression, though quite adequate for the small number of actions we currently have, will become inefficient when the number of actions becomes large, certainly at around 100 and likely much less.This area provides voice notifications of received emails in Gmail and messages in Facebook.
Notfications looks like a good idea and we might just copy that area over to Melissa.
We can likely cut down this page for now and make changes as time goes on.
It would be rather useful to be able to mix these modules into Melissa. We would then need a Jasper-Module interface. This also goes toward having a screen on which to select capabilities and modules as desired.
This will be a useful page as the number of users and developers increases. We can likely cut it down for the moment.
This page can be kept much the same with Jasper replaced by Melissa. Of course it will be better to give it an independent look but it will help to have an initial page.
Home, About and Documentation
These are two nicely designed pages but are more user impact than substance and could be avoided until we provide unique graphics for Melissa. We might also provide a point that differentiates Melissa from Jasper. Saying something like An app for Linux and Windows that gets away from the Raspberry Pi emphasis.
This link goes to a forum page under Google Groups. Looks like an easy option to set up. Much of what I write should likely go on a forum page instead of an issues page. Here is how to create a Google Group.
This link would go to the current Melissa home page.
alteration.py - "Manually adjust output text before it's translated into actual speech by the TTS system."
app_utils.py - Email helpers, URL compressor, negative and positive sentiment detectors.
brain.py - I realize this will be a significant departure from the current Melissa code but given that it allows dynamic loading of modules, the immediate addition of Jasper 3rd-party modules, and that no changes will be required to this version of brain.py when new modules are added, it may merit consideration.
conversation.py - This provides a loop for notifications and conversations (the processing of user expressions) and is another significant change to Melissa. This modules fits with brain.py. client/mic.py handles the actual audio to text processing that for Melissa is done in main.py.
mic.py - What Melissa is doing in main.py could replace this code. Whether or not we abstract the speech-to-text processing so that different speech-recognizers can be used without change to this program is something to think about but for now that seems like a low priority.
diagnose.py - This may be useful when tailored to Melissa but may be optional at the moment.
g2p.py - This appears to be a way to improve the text-to-speech quality. Could be a secondary priority though perhaps easily done.
jasperpath.py - This obtains the absolute path to Jasper's run directory and to a config file,
~/.jasper. This may not be immediately useful but an idea of how it might be used would be:A melissa.py (instead of a main.py) might be put, on Linux, in /usr/local/bin.
melissa.py would read a
~/.melissafile that identified the disk location of the Melissa installation.The remaining Melissa programs would then use the absolute paths provided by melissapath.py to reach the programs and data of the Melissa system.
local_mic.py - "A drop-in replacement for the Mic class that allows for all I/O to occur over the terminal." This could be useful when testing.
notifier.py - This code is called by conversation.py on each expression-response loop to handle possible notifications from email and others. It would seem this kind of thing would be put in a separate thread so as to minimize the expression-response loop cycle time. See more on threading below.
test_mic.py - "A drop-in replacement for the Mic class used during unit testing." We should likely develop some unit-tests in which this could be used. Marking this as secondary.
stt.py - This might be useful when additional speech-recognizers are added as shown on Jasper's installation page. Marking this as secondary.
tts.py - This may be useful when different text-to-speech options are added as shown on Jasper's installation page. Marking this as secondary.
vocabcompiler.py - This assembles a list of all the word lists in the action modules. The combined word list is used to improve the accuracy of the speech-to-text software by concentrating recognition on only the words in the list. Might be useful for PocketSphinx. Marking this as secondary.
modules directory - If Melissa's brain.py is modified for dynamic loading, all the modules will be included automatically. We would check here for modules overlapping Melissa actions such that either one or the other would be used, and actions Melissa has that Jasper does not suggesting that the Melissa action could be converted to a module.
These modules may require additional installs and setup.
Birthday.py - Provides an audio list of "Facebook friends with birthdays today."
Gmail.py - Responds "with a summary of the user's Gmail inbox, reporting on the number of unread emails in the inbox, as well as their senders."
HM.py - "Returns the top headlines from Hacker News."
joke.py - Tells a joke randomly selected from a file list.
Life.py - Responds with "the meaning of life."
MPDControl.py - Plays music from Spotify.
News.py - Gets news articles from http://news.ycombinator.com and will send the articles to your email address or speak the headlines and the articles.
Notifications.py - Responds with a count of the user's Facebook notifications.
Time.py - Responds with the current time.
Unclear.py - Responds by saying it could not understand the user's expression.
Weather.py - Responds with the weather for the location (zip-code) provided for populate.py.
copied directly from Jasper's Modules page.
These modules require additional installs and setup.
Google Calendar – The Google Calendar module lets you check your Google calendar today or tomorrow for events, and it lets you add an event to your calendar.
Twitter – The Twitter module lets you tweet, check whats trending and check for any twitter notifications which include: Direct messages, retweets or mentions.
Evernote – The Evernote module lets you store notes for you in your Evernote account.
WolframAlpha – The WolframAlpha module lets you ask question to WolframAlpha search engine.
Status – The Status module outputs system status information upon request.
Movies – The Movies module is used to retrieve movie information.
Define – The Define module is used to get definitions of words.
Wiki – The Wiki module is used to get a Wikipedia summary of a word or phrase.
Domoticz – The Domoticz home automation module is used to issue commands to a Domoticz server using the JSON API.
Find My iPhone – The Find My iPhone module is used to force your phone to ring using the apple service “find my iphone”
Yelp – The yelp module is used to find a good restaurant in a given area. The module can also answer various questions about the restaurant.
There is little overlap between the Melissa and Jasper modules. If Jasper's dynamic loading method is adopted the Melissa modules should be converted to that method.
business_news_reader.py - Obtains headlines and news articles from http://profit.ndtv.com/news/latest/.
connect_proxy.py - Connects to a proxy server.
define_subject.py - Responds with subject information from Wikipedia.
general_conversations.py - Responds to several user expressions.
imgur_handler.py - Responds with a terminal list of image locations on the user's computer.
lighting.py - Changes the color of the USB RGB LED notification light.
notes.py - Writes and responds with notes saved into an sqlite3 database (memory.db).
open_firefox.py - Opens FireFox.
play_music.py - Plays music files from the user's computer.
sleep.py - Exits Melissa.
tell_time.py - Responds with the current time.
weather.py - Uses the city_code from profile.yaml to report the weather from Weather.com.
twitter_interaction.oy - Posts the user's recognized audio to twitter.
tanay1337
commented
8 years ago Thanks for the detailed analysis @neilnelson! These are some really great suggestions. One thing that I wish to avoid at this point, is the use of Wit.ai for brain.py as I am working on some homebrew alternatives. Let us split these suggestions to independent issues and well, get on to it :D
tanay1337
commented
8 years ago Discussion and support forum has been created here: https://groups.google.com/forum/#!forum/melissa-support--discussion-forum/
tanay1337
commented
8 years ago I have added most of the high level suggestions as issues, let me know if something is missing, we can add that can an issue to be implemented.
neilnelson
commented
8 years ago Tanay,
I have continued this discussion on the forum. I was going to do the Jasper-Melissa merge but it looks like you have something else in mind you are working on. Obviously, you would have a variety of changes to my merge result and we could refine that a fair amount.
Review of AI projects - Jasper
Jasper
After looking through a few of Jasper's python modules and noticing the MIT license I suggest we get all we can from Jasper. We would understand their code and mix it with ours, taking all that they have that we do not. This also gets us to a question of whether or not we will be ahead by merging with the Jasper project or whether our focus is sufficiently different to run a parallel effort.
I see they are using a direct word-action association that though effective limits the greater variation in user expression's that Api.ai would provide.
At the moment I am thinking we get all we can from Jasper, use a better user-expression-to-action method along Api.ai or as I have noted under the ConceptNet discussion, and begin developing domains along the Api.ai line.