search

openassistant / oa-core

Open Assistant Core
http://openassistant.org
GNU General Public License v3.0
291 stars 77 forks source link

Is this working? #31

Closed gefagan closed 4 years ago

gefagan commented 4 years ago

Howdy! This project looks cool but I'm having some trouble with it. I think I've installed all the requirements, and when I run the command, here's the output:

[code]gef@purplebox:~/oa-core> python3 -m oa [2020-05-02 00:36:05,760] INFO MainThread [main.py::102]: Start Open Assistant [2020-05-02 00:36:05,764] INFO MainThread [hub.py:_load_modules:37]: Loading Modules <- /home/gef/oa-core/oa/modules [2020-05-02 00:36:05,764] INFO MainThread [util.py:load_module:60]: sound <- /home/gef/oa-core/oa/modules/sound [2020-05-02 00:36:05,767] INFO MainThread [util.py:load_module:60]: mind <- /home/gef/oa-core/oa/modules/mind [2020-05-02 00:36:05,986] INFO MainThread [util.py:load_module:60]: voice <- /home/gef/oa-core/oa/modules/voice [2020-05-02 00:36:05,988] INFO MainThread [util.py:load_module:60]: speech_recognition <- /home/gef/oa-core/oa/modules/speech_recognition [2020-05-02 00:36:05,995] INFO MainThread [util.py:load_module:60]: ear <- /home/gef/oa-core/oa/modules/ear OA> [2020-05-02 00:36:06,223] INFO mind [init.py:load_minds:55]: Loading minds... [2020-05-02 00:36:06,225] INFO mind [init.py:load_minds:59]: <- root.py [2020-05-02 00:36:06,324] INFO mind [init.py:load_minds:59]: <- boot.py [2020-05-02 00:36:06,325] INFO mind [init.py:load_minds:62]: Minds loaded! [2020-05-02 00:36:06,325] INFO mind [init.py:set_mind:41]: Opening Mind: boot[/code]

And after that, nothing. If I say "boot mind" as the listening test, there's no response. I know my mic works because I can get some interaction with mycroft, although it's crap which is why I'm checking this out. Is the output above normal, or should I be able to tell from that what's wrong?

Thanks,

GEF

gefagan commented 4 years ago

PS: My distro is openSUSE Tumbleweed, if that makes a difference. -GEF

joshuashort commented 4 years ago

Looks pretty normal.. except for the OA> [2020-05-02 00:36:06,223] INFO mind [init.py:load_minds:55]: Loading minds... part.

Try with python3 -m oa -d?

gefagan commented 4 years ago

Wow, you're up late, thanks for taking a look. Same behavior, whih is to say no response. I verfied on dictation.io that my hardware works. Here's the output:

gef@purplebox:~/oa-core> python3 -m oa -d [2020-05-02 02:28:08,425] INFO MainThread [main.py::102]: Start Open Assistant [2020-05-02 02:28:08,430] INFO MainThread [hub.py:_load_modules:37]: Loading Modules <- /home/gef/oa-core/oa/modules [2020-05-02 02:28:08,430] INFO MainThread [util.py:load_module:60]: sound <- /home/gef/oa-core/oa/modules/sound [2020-05-02 02:28:08,432] INFO MainThread [util.py:load_module:60]: mind <- /home/gef/oa-core/oa/modules/mind [2020-05-02 02:28:08,669] INFO MainThread [util.py:load_module:60]: voice <- /home/gef/oa-core/oa/modules/voice [2020-05-02 02:28:08,671] INFO MainThread [util.py:load_module:60]: speech_recognition <- /home/gef/oa-core/oa/modules/speech_recognition [2020-05-02 02:28:08,678] INFO MainThread [util.py:load_module:60]: ear <- /home/gef/oa-core/oa/modules/ear

OA> [2020-05-02 02:28:08,894] INFO mind [init.py:load_minds:55]: Loading minds... [2020-05-02 02:28:08,895] INFO mind [init.py:load_minds:59]: <- root.py [2020-05-02 02:28:08,986] INFO mind [init.py:load_minds:59]: <- boot.py [2020-05-02 02:28:08,986] INFO mind [init.py:load_minds:62]: Minds loaded! [2020-05-02 02:28:08,987] INFO mind [init.py:set_mind:41]: Opening Mind: boot [2020-05-02 02:28:08,987] DEBUG mind [init.py:_in:70]: "boot" is now listening. Say "Boot Mind!" to see if it can hear you.

joshuashort commented 4 years ago

There are a few things that could be going wrong. After seeing this issue, I got motivated to do some cleanup.. I don't think there's going to be a fix in the next few hours, but this output is interesting.

It's not really documented, but using -l <FILE> writes all the logging stuff to a file instead of stdout. There's a kind of REPL that accepts commands like <MODULE> <ARG> (e.g. voice say something.

In addition to that OA> thing (from the REPL), our load/start orders seem different. That might matter, but it shouldn't..

It might be load order or threading issues; maybe something with the ear or speech_recognition modules (or their dependencies).

gefagan commented 4 years ago

It might be load order or threading issues

Interesting. My laptop has an A12 cpu, an odd duck from before AMD got good (which is why I got it cheap), I have another laptop that I don't normally use because it's damaged, but it has a 7th-gen i7 with the same OS, a near-identical installation. While I've never had this kind of problem with the A12 before, based on your comment I tried on my intel laptop and it went smoothly. So it looks like we can chalk this up to hardware and close the issue.

So if I understand correctly, now I have to figure out how to program voice commands that I'll find useful? Krunner is such a useful tool as-is that if it were voice-activated, it'd be a pretty good secretary.

gefagan commented 4 years ago

Um, it's not obvious to me were the commands live. Is there a config file that I'm over-looking because it's in plain sight?

joshuashort commented 4 years ago

It might be load order or threading issues

So it looks like we can chalk this up to hardware and close the issue.

Glad you got it working on a different system! Give it a try (on some later release in the future) if you're still interested, it might work. Either way, stuff's getting cleaned up.

So if I understand correctly, now I have to figure out how to program voice commands that I'll find useful? Krunner is such a useful tool as-is that if it were voice-activated, it'd be a pretty good secretary.

Yeah, that's a good place to start.

joshuashort commented 4 years ago

Um, it's not obvious to me were the commands live. Is there a config file that I'm over-looking because it's in plain sight?

If only..

The quickest way to a custom command right now is by customizing the 'root mind' -- https://github.com/openassistant/oa-core/blob/master/oa/modules/mind/minds/root.py. I think there are a few gotchas with trying add a new mind, so you can just hack away on this one.

Make your changes, restart OA, when 'boot mind' is active say "open assistant" (command to switch to root mind, same as typing mind open assistant, iirc).

The important thing to note is the @command decorator; put that on a function you want to call, and give it a phrase or a list of phrases; it doesn't matter what the function is named, @command is for registering a phrase to a function.

When OA starts, it calls a web service to get some files for speech recognition (it takes all the phrases registered, and gets a pronunciation dictionary and ngram). That's not great, but that's how it currently works.

That's the basic idea, as brief as I could make it. Let me know how it goes!

gefagan commented 4 years ago

Thanks Joshua. I'm just a linux end-user, which is to say more sophisticated than a typical windows user but still no coder. Hacking the root mind python file sounds beyond my skills. Populating a config file with things I can say and bash commands I want to execute when I say them is somthing I could manage, if smarter people ever implement the capability. I fgure there'd need to be a couple hundred commands, minimum, especially since I saw another issue that seemed to state commands can't take parameters, so "open firefox" and "open firefox to gmail" would have to be separate commands.

What would be really useful is to convert speech-to-text and then pass the text, frex "open argument" would pass "argument" to krunner (or similar app on desktops other than plasma). Do I understand that passing an argument is impossible right now?

joshuashort commented 4 years ago

Do I understand that passing an argument is impossible right now?

Yes. And end-user ease.. isn't

I'll close this issue, and do what I can on the underlying issues.