search

tmc / righthand

RightHand - A GPT4 powered assistive tool.
84 stars 8 forks source link

not executing commands #1

Open jmanhype opened 1 year ago

jmanhype commented 1 year ago

rileylovett@Rileys-Air ~ % righthand
righthand: initializing... righthand: using whisper model: base.en whisper_init_from_file_no_state: loading model from '/Users/rileylovett/Library/Caches/whisper.cpp/ggml-base.en.bin' whisper_model_load: loading model whisper_model_load: n_vocab = 51864 whisper_model_load: n_audio_ctx = 1500 whisper_model_load: n_audio_state = 512 whisper_model_load: n_audio_head = 8 whisper_model_load: n_audio_layer = 6 whisper_model_load: n_text_ctx = 448 whisper_model_load: n_text_state = 512 whisper_model_load: n_text_head = 8 whisper_model_load: n_text_layer = 6 whisper_model_load: n_mels = 80 whisper_model_load: ftype = 1 whisper_model_load: qntvr = 0 whisper_model_load: type = 2 whisper_model_load: mem required = 310.00 MB (+ 6.00 MB per decoder) whisper_model_load: adding 1607 extra tokens whisper_model_load: model ctx = 140.66 MB whisper_model_load: model size = 140.54 MB whisper_init_state: kv self size = 5.25 MB whisper_init_state: kv cross size = 17.58 MB righthand: ready listening... transcribing... progress: 0% progress: 85% transcribed: "Can you hear me?Hello.Transcribe my words.[BLANK_AUDIO]" in 3.200136917s active app: Google Chrome righthand: using 1 few-shot examples for Google Chrome response: Can you hear me? Hello. Transcribe my words. righthand: typing remainder of text: Can you hear me? Hello. Transcribe my words. listening... transcribing... progress: 0% progress: 85% progress: 95% progress: 95% transcribed: "Go to Facebook.com.You may stop listening now.[MUSIC PLAYING]Bye!" in 7.577990334s active app: Terminal righthand: using 0 few-shot examples for Terminal response: The current active program is Terminal, which does not support browsing websites. To open a webpage, you would need to use a program such as a web browser. However, you can still use Terminal to ping the website.

To do this for Facebook.com, use the following command:

ping facebook.com

righthand: typing remainder of text: The current active program is Terminal, which does not support browsing websites. To open a webpage, you would need to use a program such as a web browser. However, you can still use Terminal to ping the website.

To do this for Facebook.com, use the following command:

ping facebook.com

listening... transcribing... progress: 0% progress: 30% progress: 35% progress: 40% progress: 50% progress: 55% progress: 60% progress: 70% progress: 75% progress: 80% progress: 100% transcribed: "Sometimes I hate the shit that I see in here because go to Facebook.comLike somebody deliberately did thatAndYouYouYouYouYouYouYouOkay, you see that? That's how it's been." in 13.950673708s active app: Google Chrome righthand: using 1 few-shot examples for Google Chrome response: https://facebook.com{Enter} righthand: typing text: https://facebook.com

listening... transcribing... progress: 0% progress: 95% transcribed: "pick it upjust pick up the phoneHey Sean, what's up? I'm Marvintype google.comManageredAlene6What are you doing?ManageredAlene6Stop thatI have a home card instruction for my credit card companyI need a managerShe knew my card backI can't do thatI have a phoneIt says \"Hold Card\"and she says she's entitled to Barbara Hendricks- the Pielto Barbara Hendrickson's family." in 3.801883s active app: Google Chrome righthand: using 1 few-shot examples for Google Chrome response: Return the input exactly as provided if you aren't confident in your answer. righthand: typing remainder of text: Return the input exactly as provided if you aren't confident in your answer. listening... transcribing... progress: 0% progress: 80% progress: 95% transcribed: "Tell me a story about a boy.Tell me a story about a boy." in 3.646067792s active app: Microsoft Word righthand: using 0 few-shot examples for Microsoft Word response: This input doesn't seem to involve any commands or controls for Microsoft Word. Therefore, return as is: "Tell me a story about a boy.Tell me a story about a boy." righthand: typing remainder of text: This input doesn't seem to involve any commands or controls for Microsoft Word. Therefore, return as is: "Tell me a story about a boy.Tell me a story about a boy." listening... transcribing... progress: 0% progress: 100% transcribed: "right a short story about a boyyou just woke up straight up you're yelling get get out of here with thattake it in the living room really shut the fuck up bitch now" in 2.038079042s active app: Terminal righthand: using 0 few-shot examples for Terminal response: Sorry, but I can't assist with that. righthand: typing remainder of text: Sorry, but I can't assist with that. listening... transcribing... progress: 0% progress: 65% progress: 85% transcribed: "Write a short story in this text document.Thanks." in 2.842312916s active app: Microsoft Word righthand: using 0 few-shot examples for Microsoft Word response: As a AI assistant, I'm unable to create content. I can help you with operating word commands and helping you with formatting your text. I apologize for the misunderstanding. righthand: typing remainder of text: As a AI assistant, I'm unable to create content. I can help you with operating word commands and helping you with formatting your text. I apologize for the misunderstanding. listening... transcribing... progress: 0% progress: 25% progress: 100% transcribed: "Write down a few words into this text document.[BLANK_AUDIO]" in 2.377797541s active app: Terminal righthand: using 0 few-shot examples for Terminal response: This appears to be missing a specific command, please provide the content of the text to be written down. righthand: typing remainder of text: This appears to be missing a specific command, please provide the content of the text to be written down.

listening... transcribing... progress: 0% progress: 85% transcribed: "Write down a few words in this text document like \"dog.\"(silence)" in 2.441802334s active app: Microsoft Word righthand: using 0 few-shot examples for Microsoft Word response: dog. righthand: typing remainder of text: dog. listening... transcribing... progress: 0% progress: 75% transcribed: "Save and close this document.[BLANK_AUDIO]" in 3.00405975s active app: Microsoft Word righthand: using 0 few-shot examples for Microsoft Word response: {Control}+s {Alt}+f4 2023/07/08 15:21:01 Unknown modifier: Alt listening... transcribing... progress: 0% progress: 60% transcribed: "Save this document as \"Dog\".[ Silence ]" in 2.374929459s active app: Microsoft Word righthand: using 0 few-shot examples for Microsoft Word response: {Control}+s Type: Dog {Enter} righthand: typing text: Type: Dog

jmanhype commented 1 year ago

During the installation, several errors and warnings were observed:

  1. A warning about visibility of the 'struct whisper_full_params' declaration:

    # github.com/tmc/whisper.cpp/bindings/go
In file included from _cgo_export.c:4:
In file included from whisper.go:16:
go/pkg/mod/github.com/tmc/whisper.cpp/bindings/go@v0.0.0-20230705062322-9af4a3211895/whisper.h:133:49: warning: declaration of 'struct whisper_full_params' will not be visible outside of this function [-Wvisibility]

  2. A warning about a function declaration without a prototype in window.h:

    # github.com/go-vgo/robotgo
In file included from go/pkg/mod/github.com/go-vgo/robotgo@v0.100.10/robotgo.go:48:
In file included from go/pkg/mod/github.com/go-vgo/robotgo@v0.100.10/window/goWindow.h:12:
go/pkg/mod/github.com/go-vgo/robotgo@v0.100.10/window/window.h:21:6: warning: a function declaration without a prototype is deprecated in all versions of C and is not supported in C2x [-Wdeprecated-non-prototype]
go/pkg/mod/github.com/go-vgo/robotgo@v0.100.10/window/window.h:29:6: warning: a function declaration without a prototype is deprecated in all versions of C and is not supported in C2x [-Wdeprecated-non-prototype]

  3. Warnings about method accesses and returning 'const void ' from a function with result type 'void ' in the progrium/macdriver package:

    # github.com/progrium/macdriver/core
go/pkg/mod/github.com/progrium/macdriver@v0.4.1-0.20230706190053-7e5bd0a70b46/core/core_objc.gen.go:172:3: warning: class method '+localizedUserNotificationStringForKey:arguments:' not found (return type defaults to 'id') [-Wobjc-method-access]
go/pkg/mod/github.com/progrium/macdriver@v0.4.1-0.20230706190053-7e5bd0a70b46/core/core_objc.gen.go:1075:9: warning: returning 'const void *' from a function with result type 'void *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]

  4. The major error was the failure to find the 'portaudio-2.0' package during the installation process:

    go build github.com/gordonklaus/portaudio:
# pkg-config --cflags  -- portaudio-2.0
Package portaudio-2.0 was not found in the pkg-config search path.
Perhaps you should add the directory containing `portaudio-2.0.pc'
to the PKG_CONFIG_PATH environment variable
No package 'portaudio-2.0' found
pkg-config: exit status 1

The last error halted the installation process, which was resolved by installing the portaudio package via Homebrew (brew install portaudio). The warnings, on the other hand, didn't stop the process, but they suggest potential problems or deprecated practices in the code. These warnings may not cause immediate issues but could lead to problems in the future, especially if the deprecated practices are removed in future versions of the language or the methods are not available in all scenarios or platforms.

jmanhype commented 1 year ago

The errors and warnings you encountered during the installation could indeed be the reason why the "RightHand - Voice Controlled Assistant for Mac" isn't executing as expected. This could be particularly true if these warnings have to do with the core functionalities of the application like voice recognition, interpreting transcriptions, simulating keyboard input, and so on.

These warnings and errors could result in unexpected behavior in the application, causing it to fail or crash. Here's how the issues could potentially affect the application:

  1. FFMpeg: The warning related to FFMpeg could mean that the audio processing might not work as expected, affecting the voice recognition feature.

  2. macdriver: The errors associated with macdriver could impact the application's ability to interact with the macOS system and applications, which would limit the ability of RightHand to interpret and execute commands on the active application.

  3. Objective-C Warnings: The warnings related to Objective-C methods in the macdriver package could affect how the application interacts with macOS, which might limit the functionality of the voice-controlled assistant.

To make the application function correctly, these errors and warnings would need to be addressed. It's best to resolve these issues one by one, starting with the most critical ones. However, it's also important to keep in mind that troubleshooting these problems might require a solid understanding of the Go programming language, the macOS system and APIs, and the specific libraries used in the application.

tmc commented 11 months ago

@jmanhype Thanks I definitely need to spruce up the installation instructions to reference portaudio.