I get a bunch of text what do I do next

mrname5 commented 4 months ago

(cvdl) steve@mrname5:~/Documents/Code/bardDraw$ python main.py

J-Fo-S commented 4 months ago

did your webcam turn on? Did you try drawing with your finger? Did you try entering any text (e.g. 'guess' and press enter)?

J-Fo-S commented 4 months ago

I'll post a video over the weekend of how it should look when it runs. Should be easier that way.

mrname5 commented 4 months ago

First I tried with my main laptop without a webcam to try the speech feature. When I set the code up on a laptop with a webcam, when running main.py code without specifying mode, code just exits after the text above. The speech feature waits for me to speak and after I press s it exits. It did not give me a guess or did not ask me to give it feedback (the yes or no thing). Also when testing on the laptop with webcam, my camera light did not turn on. It usually does when webcam is in use.

mrname5 commented 4 months ago

I use Manjaro Linux which is based on Arch linux. I have commented out line 85 and first i got this:

Traceback (most recent call last):
  File "/home/steve/Documents/Code/bardDraw/hoo/lib/python3.11/site-packages/speech_recognition/__init__.py", line 108, in get_pyaudio
    import pyaudio
ModuleNotFoundError: No module named 'pyaudio'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/steve/Documents/Code/bardDraw/main.py", line 135, in <module>
    mic = sr.Microphone()
          ^^^^^^^^^^^^^^^
  File "/home/steve/Documents/Code/bardDraw/hoo/lib/python3.11/site-packages/speech_recognition/__init__.py", line 80, in __init__
    self.pyaudio_module = self.get_pyaudio()
                          ^^^^^^^^^^^^^^^^^^
  File "/home/steve/Documents/Code/bardDraw/hoo/lib/python3.11/site-packages/speech_recognition/__init__.py", line 110, in get_pyaudio
    raise AttributeError("Could not find PyAudio; check installation")
AttributeError: Could not find PyAudio; check installation

I fixed that by doing pip install pyaudio. Maybe you forgot to add that to the requirements.txt script? I don't know.

mrname5 commented 4 months ago

Installing pyaudio seems to allow my webcam to turn on. I can now see a webcam feed and a drawing window. However, Gemini still seems to not be guessing. I tried the instructions you provided here: " Speech mode is the same as keyboard mode, except you will need to press "s" to send the speech to Gemini. It should automatically record when clear speech is detected. Sign mode is more limited. Raise your left index finger to have Gemini guess about the drawing. Raise your left index and middle finger to have Gemini guess again. Raise your left index, middle and ring finger to have Gemini tell a story about the drawing (what Gemini guessed)." However, that did not work. I think it might be related to some of the weird errros I got here

Traceback (most recent call last):
  File "/home/steve/Documents/Code/bardDraw/envname/lib/python3.11/site-packages/pyttsx3/__init__.py", line 20, in init
    eng = _activeEngines[driverName]
          ~~~~~~~~~~~~~~^^^^^^^^^^^^
  File "/usr/lib/python3.11/weakref.py", line 136, in __getitem__
    o = self.data[key]()
        ~~~~~~~~~^^^^^
KeyError: None

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/steve/Documents/Code/bardDraw/main.py", line 166, in <module>
    t2s = T2S()
          ^^^^^
  File "/home/steve/Documents/Code/bardDraw/t2s.py", line 7, in __init__
    self.t2s = pyttsx3.init()
               ^^^^^^^^^^^^^^
  File "/home/steve/Documents/Code/bardDraw/envname/lib/python3.11/site-packages/pyttsx3/__init__.py", line 22, in init
    eng = Engine(driverName, debug)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/steve/Documents/Code/bardDraw/envname/lib/python3.11/site-packages/pyttsx3/engine.py", line 30, in __init__
    self.proxy = driver.DriverProxy(weakref.proxy(self), driverName, debug)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/steve/Documents/Code/bardDraw/envname/lib/python3.11/site-packages/pyttsx3/driver.py", line 50, in __init__
    self._module = importlib.import_module(name)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/steve/Documents/Code/bardDraw/envname/lib/python3.11/site-packages/pyttsx3/drivers/espeak.py", line 9, in <module>
    from . import _espeak, toUtf8, fromUtf8
  File "/home/steve/Documents/Code/bardDraw/envname/lib/python3.11/site-packages/pyttsx3/drivers/_espeak.py", line 18, in <module>
    dll = cdll.LoadLibrary('libespeak.so.1')
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/ctypes/__init__.py", line 454, in LoadLibrary
    return self._dlltype(name)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/ctypes/__init__.py", line 376, in __init__
    self._handle = _dlopen(self._name, mode)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: libespeak.so.1: cannot open shared object file: No such file or directory

J-Fo-S commented 4 months ago

can you solve it through this: https://stackoverflow.com/questions/65274843/oserror-libespeak-so-1-cannot-open-shared-object-file-no-such-file-or-directo

J-Fo-S commented 4 months ago

Also, does the program work for you in the other modes?

mrname5 commented 4 months ago

I figured that out. I had to install a espeak package for my system. I am slightly confused though. So when I run it in default mode, I am supposed to draw something right? Is Gemini supposed to automatically guess after? Or am I supposed to do something. Another question I have is which animals is Gemini allowed to guess. In which files are the animals in?

Another thing is, I seem to get this weird error:

INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
_you are in keyboard mode
ask away!
ALSA lib confmisc.c:165:(snd_config_get_card) Cannot get card index for Loopback
aplay: main:850: audio open error: No such device

guess
_using ticket for image
_sending image
got your image.. please wait one sec..
ALSA lib confmisc.c:165:(snd_config_get_card) Cannot get card index for Loopback
aplay: main:850: audio open error: No such device
Hi kid! Are you drawing a square?  That's a great start!  Squares are fun to draw, and they can be used to make all sorts of things, like houses, cars, and windows.  What are you going to make with your square?
_done
_issuing new ticket
_getting doodle
loading car drawings
load complete
loading car drawings
load complete
ALSA lib confmisc.c:165:(snd_config_get_card) Cannot get card index for Loopback
aplay: main:850: audio open error: No such device

J-Fo-S commented 4 months ago

Hi it looks like your sound card (computer audio) is not recognizable. Possibly it is not configured correctly, but I don't run on Linux these days so I can't offer much more help. Maybe it has something to do with the pyaudio install. Try running through this and see what happens: https://pbaumgarten.com/python/audio.md

mrname5 commented 4 months ago

I agree that it is probably a issue with my Linux configuration but since I found an external webcam, I can use my normal setup which does not seem to have the same issue. Gemini now speaks also. However, it seems like tracking is failing a lot of the time for some reason. I tried setting the --smooth argument to 1 and --mode_len to 30 but none of that seemed to work. Here is a video of that happening. https://youtu.be/b1N7WO012H8

Please let me know if you have any idea on how to fix this or is my webcam just bad.

J-Fo-S commented 4 months ago

Yes, that's to be expected, actually. The problem is not your webcam but the lighting and background. If you have good lighting and a good background contrast (like a wall) then it will work better. If we wanted to improve this more, training a neural net would be the way to go. This weekend I will add some new developments for Gemini - stay tuned and glad you got it working.

mrname5 commented 4 months ago

I tried with different lighting and background but I still got the same result so I decided to switch to another webcam. When I did, I got this error when I tried it the first time. The webcam worked fine after that. I still struggled to get the tracking to work correctly though it did work better with a new webcam.

Exception in thread Thread-2 (capture):
Traceback (most recent call last):
  File "/usr/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.11/threading.py", line 982, in run
    self._target(*self._args, **self._kwargs)
  File "/home/steve/Documents/Code/bardDraw/cv2draw.py", line 33, in capture
    frame = cv2.resize(frame, (canvas.shape[0], canvas.shape[1]))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
cv2.error: OpenCV(4.8.1) /io/opencv/modules/imgproc/src/resize.cpp:4062: error: (-215:Assertion failed) !ssize.empty() in function 'resize'

J-Fo-S commented 4 months ago

interesting, perhaps there was some noise or something in the other webcam's signal. The error is a little bug I'll fix (if i don't forget) caused by there not being a frame sent from the webcam but the code runs anyways, or at least I think. I have made some large and (for now) final changes to the code that I'll push tonight. The drawing will be more controllable, and more modes to infuence Gemini's replies will be available.

J-Fo-S commented 3 months ago

I tried with different lighting and background but I still got the same result so I decided to switch to another webcam. When I did, I got this error when I tried it the first time. The webcam worked fine after that. I still struggled to get the tracking to work correctly though it did work better with a new webcam.
Exception in thread Thread-2 (capture):
Traceback (most recent call last):
  File "/usr/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.11/threading.py", line 982, in run
    self._target(*self._args, **self._kwargs)
  File "/home/steve/Documents/Code/bardDraw/cv2draw.py", line 33, in capture
    frame = cv2.resize(frame, (canvas.shape[0], canvas.shape[1]))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
cv2.error: OpenCV(4.8.1) /io/opencv/modules/imgproc/src/resize.cpp:4062: error: (-215:Assertion failed) !ssize.empty() in function 'resize'

PS try drawing with your thumb sticking out, instead of closed into your fist. See my demo video if it this is not clear.

mrname5 commented 3 months ago

I am testing the new version of the code but when I run the code as python main.py. I get this error:

Exception in thread Thread-2 (capture):
Traceback (most recent call last):
  File "/usr/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.11/threading.py", line 982, in run
    self._target(*self._args, **self._kwargs)
  File "/home/steve/Documents/Code/bardDraw/cv2draw.py", line 48, in capture
    if keyboard.is_pressed('up'):
       ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/steve/Documents/Code/bardDraw/envname/lib/python3.11/site-packages/keyboard/__init__.py", line 410, in is_pressed
    _listener.start_if_necessary()
  File "/home/steve/Documents/Code/bardDraw/envname/lib/python3.11/site-packages/keyboard/_generic.py", line 35, in start_if_necessary
    self.init()
  File "/home/steve/Documents/Code/bardDraw/envname/lib/python3.11/site-packages/keyboard/__init__.py", line 196, in init
    _os_keyboard.init()
  File "/home/steve/Documents/Code/bardDraw/envname/lib/python3.11/site-packages/keyboard/_nixkeyboard.py", line 113, in init
    build_device()
  File "/home/steve/Documents/Code/bardDraw/envname/lib/python3.11/site-packages/keyboard/_nixkeyboard.py", line 109, in build_device
    ensure_root()
  File "/home/steve/Documents/Code/bardDraw/envname/lib/python3.11/site-packages/keyboard/_nixcommon.py", line 174, in ensure_root
    raise ImportError('You must be root to use this library on linux.')
ImportError: You must be root to use this library on linux.

To fix the root permission issues, I run it as sudo python main.py but I get this error:

Traceback (most recent call last):
  File "/home/steve/Documents/Code/bardDraw/main.py", line 121, in <module>
    firefox_cookies = browser_cookie3.firefox(domain_name=domain)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/steve/Documents/Code/bardDraw/envname/lib/python3.11/site-packages/browser_cookie3/__init__.py", line 1209, in firefox
    return Firefox(cookie_file, domain_name).load()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/steve/Documents/Code/bardDraw/envname/lib/python3.11/site-packages/browser_cookie3/__init__.py", line 1002, in __init__
    super().__init__('Firefox', cookie_file, domain_name, **args)
  File "/home/steve/Documents/Code/bardDraw/envname/lib/python3.11/site-packages/browser_cookie3/__init__.py", line 850, in __init__
    self.cookie_file = cookie_file or self.__find_cookie_file(**kwargs)
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/steve/Documents/Code/bardDraw/envname/lib/python3.11/site-packages/browser_cookie3/__init__.py", line 912, in __find_cookie_file
    user_data_path = self.__expand_and_check_path(linux_data_dirs)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/steve/Documents/Code/bardDraw/envname/lib/python3.11/site-packages/browser_cookie3/__init__.py", line 903, in __expand_and_check_path
    raise BrowserCookieError(
browser_cookie3.BrowserCookieError: Could not find Firefox profile directory

J-Fo-S commented 3 months ago

Looks like you either lost your internet connection or somehow otherwise lost connection to Gemini in Firefox. See if making sure you are connected to the internet or opening up Gemini in Firefox and then running the code solves this.

mrname5 commented 3 months ago

I have Gemini open in Firefox and have made sure that I am connected to the internet. That did not solve the issue. I have also tried different networks but I still get the same error.

J-Fo-S commented 3 months ago

thought I replied to this but I forgot to hit send. I'm not sure what you mean about different networks (internet browsers?) but it does have to be Firefox - the browser_cookie3 depends on it. Three things to try in order:

You don't want to be in sudo for the keyboard. I am also wondering if sudo somehow breaks/blocks the cookie code. See the reply below about pyinput which may solve the problem: https://stackoverflow.com/questions/48796147/import-error-you-must-be-root
The code used to work for you right? I did not change anything about the keyboard or cookie code in the latest commit, so you can revert to the prior commit that worked and see if that works. If so, I will be surprised so please let me know.
Try on a Windows-based system if you have access to one.

mrname5 commented 3 months ago

Sorry I was unclear in my reply. By different networks, I mean different wifi connections at different places. I also tested that the internet worked by using it for web browsing and stuff.

So, the Stack Overflow post you gave me suggested a different way of installing which did not work. It also suggested another library but that would require me to change bits of the code which is not really a good solution.

I tried the old version of the code. It seems like you use the same keyboard library but that did not ask for root permissions. I suspected that in the new version you are using a certain feature that might require it. Looking at the error closely, it looks like it is coming from if keyboard.is_pressed('up'): It seems like before you used keyboard.read_key() instead of keyboard.is_pressed. However, when I created a script that only used keyboard.read_key(), I got the same root permissions error. I am very confused, maybe that line of key detection was not used in main.py. It should be I think, I was using --mode speech.

About getting access to a Windows machine, I currently do not have a device that runs Windows. I may setup a windows Virtual Machine soon because I may need to test some other software but I prefer to stick to Linux whenever possible.

J-Fo-S commented 3 months ago

So the old commit still works? That's odd, for in speech mode one does require the keyboard module to send the speech to Gemini. Regarding your second point about using a different library, I disagree - if you succeed in replacing the keyboard module with the pyinput library, then this would be a better solution and you could submit a pull request.

mrname5 commented 3 months ago

I will try to attempt this. It might take some time

J-Fo-S commented 3 months ago

take your time!

mrname5 commented 3 months ago

I seem to be having trouble with some python formatting issue. Probably because indentation is not something I need to worry about with JavaScript. It is really weird that even Chatgpt is having trouble. I gave it some of the code and it failed to change it to use pynput. Anyways, I realized that after I run your code, my cpu usage just increases even after I stop the code. So, before running your code my cpu usage averages at 3% but after I run your code and stop it, it stays at about 22.

J-Fo-S commented 3 months ago

yes, indentation counts - and I am not surprised Chatgpt failed. You will have to make the changes yourself, but I imagine it should be fairly straightforward - just replace the keyboard methods with appropriate pyinput methods. If I have time later I may try to do so myself. As for the cpu increase, this does not happen to me. You are still receiving errors, right? Likely these errors are causing the threaded processes to not exit, and they are hanging in the background as a result. If you kill your environment and restart it, does it go away?

mrname5 commented 3 months ago

No the cpu usage is still higher when the process at least from what I see has been completely killed. The terminal says the process has been killed. I also checked the task manager.

J-Fo-S commented 3 months ago

right, because there are multiple processes running, and these are hidden from the main process. Thus, if the program exits incorrectly (e.g. has an error), then these processes can hang indefinitely in the background.

J-Fo-S / bardDraw

I get a bunch of text what do I do next #1