search

virtual-puppet-project / vpuppr

VTuber application made with Godot 4
Mozilla Public License 2.0
749 stars 62 forks source link

Doesn't see camera? #92

Closed somewhatfrog closed 2 years ago

somewhatfrog commented 2 years ago

Hello, what am I doing wrong? There is only "0 Default camera" in camera selection, I start tracker but nothing happens. Otherwise cam works well, recognized by OBS with no issues. It is the same with github version and aur version.

v4l2-ctl --list-devices
Droidcam (platform:v4l2loopback_dc-000):
    /dev/video0

openseeface-gd
Godot Engine v3.4.3.rc.custom_build.31f3e358d - https://godotengine.org
OpenGL ES 3.0 Renderer: NVIDIA GeForce RTX 3060 Ti/PCIe/SSE2
OpenGL ES Batching: ON

[INFO] 2022-1-25_15:20:40 Begin loading metadata
[INFO] 2022-1-25_15:20:40 Finished loading metadata
[INFO] 2022-1-25_15:20:40 Loading metadata: metadata.xml
[INFO] 2022-1-25_15:20:40 Loading gui file: model.xml
this is a test print in Model.gd. Extend this by modifying resources/gui/Model.gd
[INFO] 2022-1-25_15:20:40 Loading gui file: tracking.xml
[INFO] 2022-1-25_15:20:40 Loading gui file: features.xml
[INFO] 2022-1-25_15:20:40 Loading gui file: presets.xml
[INFO] 2022-1-25_15:20:40 Loading gui file: app-settings.xml
[INFO] 2022-1-25_15:20:40 Begin loading config for user:///.json
[INFO] 2022-1-25_15:20:40 Finished loading config
[INFO] 2022-1-25_15:20:40 Loading Default Model.
[INFO] 2022-1-25_15:20:40 Loading TSCN file.
[INFO] 2022-1-25_15:20:40 TSCN file loaded successfully.
[INFO] 2022-1-25_15:20:40 Press TAB to hide the UI
ERROR: Condition "name == """ is true.
   at: set_name (scene/main/node.cpp:914)
[INFO] 2022-1-25_15:20:47 Starting face tracker.
[INFO] 2022-1-25_15:20:47 Face tracker started, PID is 750575.
[INFO] 2022-1-25_15:20:47 Press spacebar to recenter the model if it's not looking correct!
[INFO] 2022-1-25_15:20:47 Listening for data at 127.0.0.1 :11573
Traceback (most recent call last):
  File "/usr/share/openseeface-gd/OpenSeeFaceFolder/OpenSeeFace/facetracker.py", line 125, in <module>
    import numpy as np
ModuleNotFoundError: No module named 'numpy'
[INFO] 2022-1-25_15:20:52 Saving config
[INFO] 2022-1-25_15:20:52 Finished saving config
[INFO] 2022-1-25_15:21:4 Stopping face tracker.
[INFO] 2022-1-25_15:21:4 Face tracker stopped, PID was 750575.
[INFO] 2022-1-25_15:21:6 Saving config
[INFO] 2022-1-25_15:21:6 Finished saving config
[INFO] 2022-1-25_15:21:6 Exiting. おやすみ。
you-win commented 2 years ago

Somewhat related to #87 , as the program currently relies on a Windows-only feature to list webcams.

More interestingly is the line

Traceback (most recent call last):
  File "/usr/share/openseeface-gd/OpenSeeFaceFolder/OpenSeeFace/facetracker.py", line 125, in <module>
    import numpy as np

What version of python3 do you have installed? I noticed you mentioned aur, so I'm assuming you're using Arch. Some of the dependencies that the program tries to auto-install are only available for python versions below 3.10. (I'm surprised there is an aur version btw, that's pretty neat).

If your python version is below 3.10, can you check if you have pip installed? Something like pip3 --version should do the trick. If not, then pip needs to be installed in order for dependencies to be auto-installed. On Debian-based distros, the command would be sudo apt install python3-pip .

If pip is already installed can you check if this folder exist for you ~/.local/share/godot/app_userdata/openseeface-gd/venv or something similar at ~/.local/share/godot/ ? If it doesn't, can you check if you have the ability to create venvs? The command python3 -m venv venv will create a folder called venv in your current directory.

somewhatfrog commented 2 years ago

pip 20.3.4 from /usr/lib/python3.10/site-packages/pip (python 3.10) Does that mean I need to downgrade python?

venv folder in place.

If I press spacebar it returns "No face tracking data found."

you-win commented 2 years ago

python 3.10 is most likely incompatible with the version of onnxruntime needed to run the facetracker. Are you able to set your default user python to an earlier version (using something like pyenv)?

If not, can you install an earlier version of python, I tested with Python 3.8.7 on Fedora 32, and then manually adjust the script paths in the resources/scripts/* folder included with the download? The scripts are currently looking for a valid python3 or python command on your path. You can remove those lines and provide an absolute path to the earlier version of python.

somewhatfrog commented 2 years ago

I installed 3.8 alongside and I did this change in both files (sorry i'm not a software dev by any means)

if python_command="$(type -p python3)"; then
    python_command="/usr/bin/python3.8"
elif python_command="$(type -p python)"; then
    python_command="/usr/bin/python3.8"
else
    exit 1

Same error returned, but this time it hanged for a bit after I clicked to start tracker.

you-win commented 2 years ago

As a simple example, for create_venv.sh

#!/bin/bash

set -e

python_command="/usr/bin/python3.8"

$python_command -m venv "$1venv"

source "$1venv/bin/activate"

pip install onnxruntime opencv-python pillow numpy
somewhatfrog commented 2 years ago

yep, still same error

you-win commented 2 years ago

Are you able to run these lines manually?

/usr/bin/python3.8 -m venv ~/.local/share/godot/app_userdata/openseeface-gd/venv

source ~/.local/share/godot/app_userdata/openseeface-gd/venv/bin/activate

pip install onnxruntime opencv-python pillow numpy

somewhatfrog commented 2 years ago 
Collecting onnxruntime
  Using cached onnxruntime-1.10.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.9 MB)
Collecting opencv-python
  Using cached opencv_python-4.5.5.62-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (60.4 MB)
Collecting pillow
  Using cached Pillow-9.0.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB)
Collecting numpy
  Using cached numpy-1.22.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.8 MB)
Collecting protobuf
  Using cached protobuf-3.19.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)
Collecting flatbuffers
  Using cached flatbuffers-2.0-py2.py3-none-any.whl (26 kB)
Installing collected packages: protobuf, numpy, flatbuffers, pillow, opencv-python, onnxruntime
Successfully installed flatbuffers-2.0 numpy-1.22.1 onnxruntime-1.10.0 opencv-python-4.5.5.62 pillow-9.0.0 protobuf-3.19.3

WARNING: You are using pip version 21.1.1; however, version 21.3.1 is available.
You should consider upgrading via the '/home/hello/.local/share/godot/app_userdata/OpenSeeFace GD/venv/bin/python3.8 -m pip install --upgrade pip' command.
(venv)

but still the same error, I also tried it with github version

you-win commented 2 years ago

It looks like the packages were installed correctly. You'll need to replace run_osf_linux.sh with this

#!/bin/bash

set -e

source "$1venv/bin/activate"

/usr/bin/python3.8 "$2" -c "$3" -F "$4" -v 0 -s 1 -P 1 --discard-after 0 --scan-every 0 --no-3d-adapt 1 --max-feature-updates 900 --ip "$5" --port "$6"

and i think it should work

somewhatfrog commented 2 years ago

error is still there, i can't see any changes

you-win commented 2 years ago

can you post the log again?

are you sure you replaced the script files correctly? Are you running the correct binary?

if all else fails, you can run the facetracker manually by

  1. manually activating the venv again
  2. running ./resource/scripts/run_osf_linux.sh <path_to_parent_folder>/OpenSeeFaceFolder/OpenSeeFace/facetracker.py 0 12 127.0.0.1 1000 . Note that you need to replace a path in that command
somewhatfrog commented 2 years ago 
$ '/home/hello/Downloads/openseeface-gd_0.8.0_linux/OpenSeeFaceGD.x86_64' 
Godot Engine v3.4.3.rc.custom_build.31f3e358d - https://godotengine.org
OpenGL ES 3.0 Renderer: NVIDIA GeForce RTX 3060 Ti/PCIe/SSE2
OpenGL ES Batching: ON

[INFO] 2022-1-25_18:40:22 Begin loading metadata
[INFO] 2022-1-25_18:40:22 Finished loading metadata
[INFO] 2022-1-25_18:40:22 Loading metadata: metadata.xml
[INFO] 2022-1-25_18:40:22 Loading gui file: model.xml
this is a test print in Model.gd. Extend this by modifying resources/gui/Model.gd
[INFO] 2022-1-25_18:40:22 Loading gui file: tracking.xml
[INFO] 2022-1-25_18:40:22 Loading gui file: features.xml
[INFO] 2022-1-25_18:40:22 Loading gui file: presets.xml
[INFO] 2022-1-25_18:40:22 Loading gui file: app-settings.xml
[INFO] 2022-1-25_18:40:22 Begin loading config for user:///.json
[INFO] 2022-1-25_18:40:22 Finished loading config
[INFO] 2022-1-25_18:40:22 Loading Default Model.
[INFO] 2022-1-25_18:40:22 Loading TSCN file.
[INFO] 2022-1-25_18:40:22 TSCN file loaded successfully.
[INFO] 2022-1-25_18:40:22 Press TAB to hide the UI
ERROR: Condition "name == """ is true.
   at: set_name (scene/main/node.cpp:914)
[INFO] 2022-1-25_18:40:26 Starting face tracker.
[INFO] 2022-1-25_18:40:26 Face tracker started, PID is 958628.
[INFO] 2022-1-25_18:40:26 Press spacebar to recenter the model if it's not looking correct!
[INFO] 2022-1-25_18:40:26 Listening for data at 127.0.0.1:11573
Traceback (most recent call last):
  File "/home/hello/Downloads/openseeface-gd_0.8.0_linux/OpenSeeFaceFolder/OpenSeeFace/facetracker.py", line 125, in <module>
    import numpy as np
ModuleNotFoundError: No module named 'numpy'
[INFO] 2022-1-25_18:40:30 No face tracking data found.
[INFO] 2022-1-25_18:40:31 Saving config
[INFO] 2022-1-25_18:40:31 Finished saving config
[INFO] 2022-1-25_18:40:31 Stopping face tracker.
[INFO] 2022-1-25_18:40:31 Face tracker stopped, PID was 958628.
[INFO] 2022-1-25_18:40:31 Saving config
[INFO] 2022-1-25_18:40:31 Finished saving config
[INFO] 2022-1-25_18:40:31 Exiting. おやすみ。
somewhatfrog commented 2 years ago

I guess I will try next release, thank you for trying to help!

you-win commented 2 years ago

Did you try

if all else fails, you can run the facetracker manually by

manually activating the venv again running ./resource/scripts/run_osf_linux.sh /OpenSeeFaceFolder/OpenSeeFace/facetracker.py 0 12 127.0.0.1 1000 . Note that you need to replace a path in that command

somewhatfrog commented 2 years ago

yes it said "/.../activate no such file or directory" so i tried to edit script with full path to "activate" file, then it returned "/...//facetracker.py "0" no such file of directory (i guess this zero comes from 0 12 127.0.0.1 1000 argument), i messed with it a bit more and decided to make a good dinner instead :)

you-win commented 2 years ago

fair enough, thanks for trying it out

you-win commented 2 years ago

hopefully fixed by #98

somewhatfrog commented 2 years ago

Hey, I just tried again with an updated version from github, and it is still all the same for some reason :( I even reproduced all the things you told me in this thread.

'/home/hello/Downloads/openseeface-gd_0.8.1_linux/export/OpenSeeFaceGD.x86_64' 
Godot Engine v3.4.3.rc.custom_build.6658f5c47 - https://godotengine.org
OpenGL ES 3.0 Renderer: NVIDIA GeForce RTX 3060 Ti/PCIe/SSE2
OpenGL ES Batching: ON

[INFO] 2022-2-17_19:23:33 Begin loading metadata
[INFO] 2022-2-17_19:23:33 Finished loading metadata
[INFO] 2022-2-17_19:23:34 Loading metadata: metadata.xml
[INFO] 2022-2-17_19:23:34 Loading gui file: model.xml
this is a test print in Model.gd. Extend this by modifying resources/gui/Model.gd
[INFO] 2022-2-17_19:23:34 Loading gui file: tracking.xml
[INFO] 2022-2-17_19:23:34 Loading gui file: features.xml
[INFO] 2022-2-17_19:23:34 Loading gui file: presets.xml
[INFO] 2022-2-17_19:23:34 Loading gui file: app-settings.xml
[INFO] 2022-2-17_19:23:34 Begin loading config for user:///.json
[INFO] 2022-2-17_19:23:34 Finished loading config
[INFO] 2022-2-17_19:23:34 Loading Default Model.
[INFO] 2022-2-17_19:23:34 Loading PackedScene file.
[INFO] 2022-2-17_19:23:34 PackedScene file loaded successfully.
[INFO] 2022-2-17_19:23:34 Press ESCAPE to hide the UI
ERROR: Condition "name == """ is true.
   at: set_name (scene/main/node.cpp:914)
[INFO] 2022-2-17_19:23:38 Starting face tracker.
[INFO] 2022-2-17_19:23:38 Face tracker started, PID is 535429.
[INFO] 2022-2-17_19:23:38 Press spacebar to recenter the model if it's not looking correct!
[INFO] 2022-2-17_19:23:38 Listening for data at 127.0.0.1:11573
Traceback (most recent call last):
  File "/home/hello/Downloads/openseeface-gd_0.8.1_linux/export/OpenSeeFaceFolder/OpenSeeFace/facetracker.py", line 125, in <module>
    import numpy as np
ModuleNotFoundError: No module named 'numpy'
[INFO] 2022-2-17_19:23:40 No face tracking data found.
[INFO] 2022-2-17_19:23:43 Stopping face tracker.
[INFO] 2022-2-17_19:23:43 Face tracker stopped, PID was 535429.
[INFO] 2022-2-17_19:23:43 Saving config
[INFO] 2022-2-17_19:23:43 Finished saving config
[INFO] 2022-2-17_19:23:44 Saving config
[INFO] 2022-2-17_19:23:44 Finished saving config
[INFO] 2022-2-17_19:23:44 Exiting. おやすみ。
you-win commented 2 years ago

Can you navigate to $HOME/.local/share/godot/app_userdata/OpenSeeFaceGD and delete the venv folder? Then try running the tracker again?

somewhatfrog commented 2 years ago

it returns this:

[INFO] 2022-2-17_19:29:7 Starting face tracker.
[INFO] 2022-2-17_19:29:7 First time setup: creating venv
[INFO] 2022-2-17_19:29:7 Listening for data at 127.0.0.1:11573
[INFO] 2022-2-17_19:29:8 Face tracker started, PID is 538463.
[INFO] 2022-2-17_19:29:8 Press spacebar to recenter the model if it's not looking correct!
/home/hello/Downloads/openseeface-gd_0.8.1_linux/export/resources/scripts/run_osf_linux.sh: line 5: /home/hello/.local/share/godot/app_userdata/OpenSeeFaceGD/venv/bin/activate: No such file or directory
somewhatfrog commented 2 years ago

Just noticed, it created venv folder in home folder for some reason. And not where it is supposed to. And if I reopen the app and click start tracker again, it says "First time setup" again.

somewhatfrog commented 2 years ago

oh wait it is working now, i did a clean "install" without modifying any files and it works! Thank you! <3

you-win commented 2 years ago

What version of bash do you have installed on your system? I've tested this on both Bash 5.0.x and 5.1.x

you-win commented 2 years ago

Oh neat, good to hear!

somewhatfrog commented 2 years ago

Will there be a hi-dpi support btw?

you-win commented 2 years ago

that's an option i need to make toggleable. It's possible in Godot, just not enabled right now

somewhatfrog commented 2 years ago

Oh I also facing new issues now lol.

  1. duck appear with half of its head outside the window (fixable by moving it down)
  2. "zoom model = on" zooming is always "zoom in" no matter which direction I scroll, and when the duck is zoomed too much it disappears and it is impossible to get it back without deleting ".json" file
  3. it doesn't seem to do anything but moving its head and blinking (preview shows much more types of "eyes")
  4. when i tried to change tracker FPS duck changed eyes to "^ ^" and stuck in that emotion.
you-win commented 2 years ago

duck appear with half of its head outside the window (fixable by moving it down)

This is a quirk of how things are positioned in 3D space + the model's builtin positioning. I could fix it for the default Duck but it's pretty low on my list of things to fix

"zoom model = on" zooming is always "zoom in" no matter which direction I scroll, and when the duck is zoomed too much it disappears and it is impossible to get it back without deleting ".json" file

Good catch, I'll open a new issue to track that. In the meantime, the shortcut ctrl + scroll will allow for zooming in/out. The Reset model transform button will also reset the model position

it doesn't seem to do anything but moving its head and blinking (preview shows much more types of "eyes")

The different expressions for the duck was more a demonstration of loading in custom models. I should probably change the gif in the README to focus more on VRM models

when i tried to change tracker FPS duck changed eyes to "^ ^" and stuck in that emotion.

As you have inadvertently found out, to change the expressions for the duck, you need to press the various number keys :) When changing the tracker FPS, you need to be clicking in the text input field, otherwise your input will be picked up by other parts of the program.

somewhatfrog commented 2 years ago

:D