search

openassistant / oa-core

Open Assistant Core
http://openassistant.org
GNU General Public License v3.0
293 stars 77 forks source link

Verify that keyboard command input is working. #37

Open ev3rl0ng opened 3 years ago

ev3rl0ng commented 3 years ago

Bug report

Describe the Bug

Attempting to use keyboard input to issue commands beyond some of the basic items in the hub command registry doesn't seem to be working.

To Reproduce

  1. Run OA.
  2. Attempt to type a command instead of speaking it.

Expected Behavior

Typed command is run exactly as it would be if spoken.

System Stats

macOS Catalina 10.15.6 Python 3.8.5

Package                                           Version
------------------------------------------------- ---------
astroid                                           2.4.2
certifi                                           2020.6.20
cffi                                              1.14.2
chardet                                           3.0.4
feedparser                                        5.2.1
idna                                              2.10
isort                                             5.4.2
keyboard                                          0.13.5
lazy-object-proxy                                 1.4.3
mccabe                                            0.6.1
numpy                                             1.19.1
pip                                               20.2.2
playsound                                         1.2.2
pocketsphinx                                      0.1.15
psutil                                            5.7.2
pycparser                                         2.20
pylint                                            2.6.0
pyobjc                                            6.2.2
pyobjc-core                                       6.2.2
pyobjc-framework-Accounts                         6.2.2
pyobjc-framework-AddressBook                      6.2.2
pyobjc-framework-AdSupport                        6.2.2
pyobjc-framework-AppleScriptKit                   6.2.2
pyobjc-framework-AppleScriptObjC                  6.2.2
pyobjc-framework-ApplicationServices              6.2.2
pyobjc-framework-AuthenticationServices           6.2.2
pyobjc-framework-AutomaticAssessmentConfiguration 6.2.2
pyobjc-framework-Automator                        6.2.2
pyobjc-framework-AVFoundation                     6.2.2
pyobjc-framework-AVKit                            6.2.2
pyobjc-framework-BusinessChat                     6.2.2
pyobjc-framework-CalendarStore                    6.2.2
pyobjc-framework-CFNetwork                        6.2.2
pyobjc-framework-CloudKit                         6.2.2
pyobjc-framework-Cocoa                            6.2.2
pyobjc-framework-Collaboration                    6.2.2
pyobjc-framework-ColorSync                        6.2.2
pyobjc-framework-Contacts                         6.2.2
pyobjc-framework-ContactsUI                       6.2.2
pyobjc-framework-CoreAudio                        6.2.2
pyobjc-framework-CoreAudioKit                     6.2.2
pyobjc-framework-CoreBluetooth                    6.2.2
pyobjc-framework-CoreData                         6.2.2
pyobjc-framework-CoreHaptics                      6.2.2
pyobjc-framework-CoreLocation                     6.2.2
pyobjc-framework-CoreMedia                        6.2.2
pyobjc-framework-CoreMediaIO                      6.2.2
pyobjc-framework-CoreML                           6.2.2
pyobjc-framework-CoreMotion                       6.2.2
pyobjc-framework-CoreServices                     6.2.2
pyobjc-framework-CoreSpotlight                    6.2.2
pyobjc-framework-CoreText                         6.2.2
pyobjc-framework-CoreWLAN                         6.2.2
pyobjc-framework-CryptoTokenKit                   6.2.2
pyobjc-framework-DeviceCheck                      6.2.2
pyobjc-framework-DictionaryServices               6.2.2
pyobjc-framework-DiscRecording                    6.2.2
pyobjc-framework-DiscRecordingUI                  6.2.2
pyobjc-framework-DiskArbitration                  6.2.2
pyobjc-framework-DVDPlayback                      6.2.2
pyobjc-framework-EventKit                         6.2.2
pyobjc-framework-ExceptionHandling                6.2.2
pyobjc-framework-ExecutionPolicy                  6.2.2
pyobjc-framework-ExternalAccessory                6.2.2
pyobjc-framework-FileProvider                     6.2.2
pyobjc-framework-FileProviderUI                   6.2.2
pyobjc-framework-FinderSync                       6.2.2
pyobjc-framework-FSEvents                         6.2.2
pyobjc-framework-GameCenter                       6.2.2
pyobjc-framework-GameController                   6.2.2
pyobjc-framework-GameKit                          6.2.2
pyobjc-framework-GameplayKit                      6.2.2
pyobjc-framework-ImageCaptureCore                 6.2.2
pyobjc-framework-IMServicePlugIn                  6.2.2
pyobjc-framework-InputMethodKit                   6.2.2
pyobjc-framework-InstallerPlugins                 6.2.2
pyobjc-framework-InstantMessage                   6.2.2
pyobjc-framework-Intents                          6.2.2
pyobjc-framework-IOSurface                        6.2.2
pyobjc-framework-iTunesLibrary                    6.2.2
pyobjc-framework-LatentSemanticMapping            6.2.2
pyobjc-framework-LaunchServices                   6.2.2
pyobjc-framework-libdispatch                      6.2.2
pyobjc-framework-LinkPresentation                 6.2.2
pyobjc-framework-LocalAuthentication              6.2.2
pyobjc-framework-MapKit                           6.2.2
pyobjc-framework-MediaAccessibility               6.2.2
pyobjc-framework-MediaLibrary                     6.2.2
pyobjc-framework-MediaPlayer                      6.2.2
pyobjc-framework-MediaToolbox                     6.2.2
pyobjc-framework-Metal                            6.2.2
pyobjc-framework-MetalKit                         6.2.2
pyobjc-framework-ModelIO                          6.2.2
pyobjc-framework-MultipeerConnectivity            6.2.2
pyobjc-framework-NaturalLanguage                  6.2.2
pyobjc-framework-NetFS                            6.2.2
pyobjc-framework-Network                          6.2.2
pyobjc-framework-NetworkExtension                 6.2.2
pyobjc-framework-NotificationCenter               6.2.2
pyobjc-framework-OpenDirectory                    6.2.2
pyobjc-framework-OSAKit                           6.2.2
pyobjc-framework-OSLog                            6.2.2
pyobjc-framework-PencilKit                        6.2.2
pyobjc-framework-Photos                           6.2.2
pyobjc-framework-PhotosUI                         6.2.2
pyobjc-framework-PreferencePanes                  6.2.2
pyobjc-framework-PushKit                          6.2.2
pyobjc-framework-Quartz                           6.2.2
pyobjc-framework-QuickLookThumbnailing            6.2.2
pyobjc-framework-SafariServices                   6.2.2
pyobjc-framework-SceneKit                         6.2.2
pyobjc-framework-ScreenSaver                      6.2.2
pyobjc-framework-ScriptingBridge                  6.2.2
pyobjc-framework-SearchKit                        6.2.2
pyobjc-framework-Security                         6.2.2
pyobjc-framework-SecurityFoundation               6.2.2
pyobjc-framework-SecurityInterface                6.2.2
pyobjc-framework-ServiceManagement                6.2.2
pyobjc-framework-Social                           6.2.2
pyobjc-framework-SoundAnalysis                    6.2.2
pyobjc-framework-Speech                           6.2.2
pyobjc-framework-SpriteKit                        6.2.2
pyobjc-framework-StoreKit                         6.2.2
pyobjc-framework-SyncServices                     6.2.2
pyobjc-framework-SystemConfiguration              6.2.2
pyobjc-framework-SystemExtensions                 6.2.2
pyobjc-framework-UserNotifications                6.2.2
pyobjc-framework-VideoSubscriberAccount           6.2.2
pyobjc-framework-VideoToolbox                     6.2.2
pyobjc-framework-Vision                           6.2.2
pyobjc-framework-WebKit                           6.2.2
python-forecastio                                 1.4.0
pyttsx3                                           2.90
requests                                          2.24.0
responses                                         0.11.0
setuptools                                        49.6.0
six                                               1.15.0
sounddevice                                       0.4.0
toml                                              0.10.1
urllib3                                           1.25.10
wheel                                             0.35.1
wrapt                                             1.12.1
ev3rl0ng commented 3 years ago

Additional testing with debug logging on.

OA> list commands
[2020-09-08 10:20:14,651] DEBUG MainThread oa.util.repl: list <- commands
OA> help
Help Stuff
OA> root mind
[2020-09-08 10:20:47,758] DEBUG MainThread oa.util.repl: root <- mind
OA> boot mind
[2020-09-08 10:22:45,634] DEBUG MainThread oa.util.repl: boot <- mind
OA> stop listening
[2020-09-08 10:22:50,119] DEBUG MainThread oa.util.repl: stop <- listening
OA> help
Help Stuff
OA> h
Help Stuff
OA> test
Unrecognized command: test
OA> list command
[2020-09-08 10:24:30,554] DEBUG MainThread oa.util.repl: list <- command
OA> this is a test
[2020-09-08 10:24:38,257] DEBUG MainThread oa.util.repl: this <- is a test
OA> b list commands
[2020-09-08 10:24:47,420] DEBUG MainThread oa.util.repl: b <- list commands
OA> q
ev3rl0ng commented 3 years ago

First word by default appears to be the part that the command is intended for.

e.g. voice "I am the one who knocks." elicits a vocal response with the phrase, mind list commands elicits the vocal readout of commands.

That's great for extensibility, but general assumption without documentation I believe would be for the command to always be interpreted by the minds unless otherwise specified.

I propose a compromise of defaulting to sending all commands except for control commands to the minds.

q, quit would still stop OA. h, help, and ? would still trigger help. All other commands would be defaulted to send to mind... if one wanted to send a command directly to another module, we could have a designator like a !<mind> or <module>: at the beginning of the command that could easily be found with regex.

ev3rl0ng commented 3 years ago

https://github.com/openassistant/oa-core/pull/39 submitted for implementation of the proposition. Not too much trouble to change if we want to swap it up.

joshuashort commented 3 years ago

Interesting thread..this prompt was meant more for controlling the hub, not just a mind, and to maybe get to scriptability. One current problem is there aren't that many interesting things to call, so usage is extra unclear.

Just a few days ago I was looking at the OA> prompt and thinking of adding the attached hub's name. So here's a proposal compromise to the compromise: it's.. modal?

Similar to how psql and kin let you attach to a database or not. That'd be a path to 'remote' hubs and easier control of its abilities (could even default to the context being set to the boot mind). The prompt would give an indicator of what the command is for.

Also, logging tends to get in the way of using the console -- maybe there's a better way to implement this? -- for now, something like python3 -m oa -d -l oa.log (that's a lowercase 'L') and tail -f oa.log cleans that up.

ev3rl0ng commented 3 years ago

Ok, so stepping back a bit to make sure I follow the logic, because there's a lot to unpack there.

Multiple Hubs

If I'm understanding you correctly, you see the potential of multiple hubs. Some of them may be remote, some of them may be local(?).

Local hubs, I'm not sure I see the draw. I think we can handle everything from one and have the abilities tie in.

The topic of remote hubs gets into the idea of interconnected-ness and how we want to architect the possibility (if any possibility) of clustering hubs across multiple systems and treating them as one, or keeping them as multiple individuals and allowing one another to influence each other or act on their behalf.

Long term, is that a capability that is desired? Probably so. There's a few things I can think that we might want to address with that model at some point. Client server model vs distributed mode... whether to directly merge all hubs into one or treat them as extended capabilities with addressing... etc.

Short term, eh. I can see not painting ourselves into a corner, but I don't think this does that.

Prompt Modes

If I'm understanding correctly, instead of prefixing commands with the module name, you're suggesting changing the module being acted against, like: 

OA> mode mind
Changed to mind module.
OA:mind> list commands
OA:mind> mode voice
Changed to voice module.
OA:voice> list commands

But also allowing flat prompt context like:

OA> voice I am the one who knocks!
(Audio plays)

Does that track? Reminds me of Cisco iOS for their switches between the configuration modes. Not opposed to it, but I think my main line of thinking was that if someone with less experience with the other technologies that we are used to gets ahold of this and starts experimenting, we need to explicitly call out that syntax in the docs or at the prompt.

To be honest, as you probably picked up from earlier comments, it wasn't clear to me to begin with either.

I hear the call for flexibility, but I want to make sure to balance it with making it intuitive as well.

ev3rl0ng commented 3 years ago

Re: logging, yeah... I burned some time looking at that today as well. I was looking at something like curses to maybe organize the screen, but that starts to get iffy with cross platform. I think the logs are still valuable and have a purpose on screen... just maybe located differently.