Any example regarding how to "Get the 2D gaze position on a predefined 3D plan(s) allowing to understand what the user is looking at."?

[ianni67]

Thank you very much for sharing your valuable effort. As stated in the title of the issue, it would be really great if you could provide more information and an example regarding how to "Get the 2D gaze position on a predefined 3D plan(s) allowing to understand what the user is looking at".

[ParisNeo]

Well, that's exactly what this library is about. I've buit it to play with my daughter by putting masks on faces, face swapping, deep faking... And one of the games was chacing objects on a screen by the face.

So yes, there is a way to define a 3D plan, then determine the 2d position of the intersection between the gaze line and the plan. In the example, I use the face vector as it is more precise and easy for the game purpose, but you can just change it with the eye gaze vector. You may change a parameter at the beginning of the file (use_eyes=True). There is also a Kalman filter to filter out any disturbance.

Good luck.

You just need to take a look at face_chacer examples (either in pygame or sqtui).

There are valuable helpers to do 3D eucledian geometry in helpers.geometry.euclidian. There you'll find functions to define a plan and detect intersection between a line and a plan. I've done the calculations myself remembering what I've leaned at school years ago. It sould work.

[ParisNeo]

To help you more here are the keys: Build a plane by providing 3d coordinates of 3 points Here is the screen plane (the z is 0 assuming the camera in in op of the screen) main_plane = get_plane_infos(np.array([0,0, 0]),np.array([100,0, 0]),np.array([0, 100,0])) then you can get the face or eye vector (a vector going out of the face or the eye) li = get_z_line_equation(face_pos, face_ori) Fuinally, use this function to get 3D postion of the intersection of the gaze vector and the planne as well as the 2D position : p, p2d = get_plane_line_intersection(main_plane, li)

I hope this helps.

[ianni67]

Thank you very much for your kind help. Unfortunately I do not have Windows, so it will take a while before I can devote a time slot to port face_chacer.py to Linux. I will post here when I do.

[ParisNeo]

The latest code should work on linux. I have removed the dependency on Direct show. It is not necessary at all. Just pull the latest master update and it should work.

[ianni67]

Hello @ParisNeo , I pulled the latest master, built it, installed it and ran the example face_chacer.py. Unfortunately I still get an error:

$ python face_chacer.py 
pygame 2.1.2 (SDL 2.0.16, Python 3.8.10)
Hello from the pygame community. https://www.pygame.org/contribute.html
Traceback (most recent call last):
  File "face_chacer.py", line 21, in <module>
    from FaceAnalyzer.helpers.ui.pygame import Widget, Button, Label, ProgressBar, ImageBox, WindowManager, Sprite
ModuleNotFoundError: No module named 'FaceAnalyzer.helpers.ui'
$

Is there anything I'm still missing?

[ParisNeo]

If you look at the last version, you'll find that the ui helper does exist. Now this is a new thing as for the old versions there were no sphesticated helpers and they were all placed in the same Helpers.py file.

Make sure you install the latest version of the code from pypi pip install --upgrade FaceAnalyzer

I suspect that you have an old version somewhere on your system and the interpreter is trying to access it.

The new version is completely incampatible with the first ones as many upgrades and code reorganizations have been added.

[ianni67]

I uninstalled everything, cleaned up, re-installed the package again with pip. No changes, I still get the same error. I also tried to clean everything, clone the repository, build and install. Same results. Sorry, I guess I'm missing something. The only examples I can run are those with OpenCV. They now run out of the box, as you removed the references to DSHOW.

[ParisNeo]

Have you installed pygame? I just added pygame to the requirements for version 0.1.13 so that it gets installed automatically when you install FaceAnalyzer. I'm sorry as I couldn't test it on Linux yet.

[ParisNeo]

Sooorrrry, I think I've figured it out. I forgot to add a init.py to the ui subfolder. I think that's what was causing the error. Upgrade to V 0.1.14 and it should work.

[ianni67]

Hello, thank you very much for your help I just removed and re-cloned the repo, re-built and reinstalled it. However, I still get that error

$ python face_chacer.py 
pygame 2.1.2 (SDL 2.0.16, Python 3.8.10)
Hello from the pygame community. https://www.pygame.org/contribute.html
/home/ianni/.virtualenvs/FA/lib/python3.8/site-packages/FaceAnalyzer-0.1.12-py3.8.egg/FaceAnalyzer/Face.py:1507: SyntaxWarning: assertion is always true, perhaps remove parentheses?
/home/ianni/.virtualenvs/FA/lib/python3.8/site-packages/FaceAnalyzer-0.1.12-py3.8.egg/FaceAnalyzer/Face.py:1507: SyntaxWarning: assertion is always true, perhaps remove parentheses?
Traceback (most recent call last):
  File "face_chacer.py", line 21, in <module>
    from FaceAnalyzer.helpers.ui.pygame import Widget, Button, Label, ProgressBar, ImageBox, WindowManager, Sprite
ModuleNotFoundError: No module named 'FaceAnalyzer.helpers.ui'

[ParisNeo]

There is a problem in your error message: The version you have is: 0.1.12 The current version is: 0.1.14

You should try to do this: pip install --upgrade FaceAnalyzer

This will upgrade the library to the latest version and it should work

[ianni67]

Did you update the git repository? I'm still receiving the same version as 6 days ago

[ParisNeo]

Sorry forgot to push :) But the Pypi was uptodate.

[ianni67]

I'm still getting errors, maybe some other init.py is missing?

$ python face_chacer.py 
pygame 2.1.2 (SDL 2.0.16, Python 3.8.10)
Hello from the pygame community. https://www.pygame.org/contribute.html
/home/ianni/.virtualenvs/FA/lib/python3.8/site-packages/FaceAnalyzer-0.1.14-py3.8.egg/FaceAnalyzer/Face.py:1507: SyntaxWarning: assertion is always true, perhaps remove parentheses?
/home/ianni/.virtualenvs/FA/lib/python3.8/site-packages/FaceAnalyzer-0.1.14-py3.8.egg/FaceAnalyzer/Face.py:1507: SyntaxWarning: assertion is always true, perhaps remove parentheses?
Traceback (most recent call last):
  File "face_chacer.py", line 26, in <module>
    from examples.Pygame.face_chacer.Chaceable import Chaceable
ModuleNotFoundError: No module named 'examples'

[ParisNeo]

That one is eazy, change this from examples.Pygame.face_chacer.Chaceable import Chaceable

to this from Chaceable import Chaceable

My editor (vscode did this automatically because I am running the examples from the root folder)

I've corrected this in the last version 0.1.15

Sorry again.

[ianni67]

If I may add one more suggestion, you might want to add pyqt5 and pyqtgraph as dependencies for your software. I had some hard time in order to detect also this small issue. pip install pyqt5 pip install pyqtgraph

[ParisNeo]

The QT library is just for some examples so I don't see the need to add it to the setup of the library. The core do not need those libraries. Maybe I'll add this to the header of each of these files in the examples

And in case you did not realize, the library examples use my other library SQTUI that requires either PyQt5 or PySide.

The idea here is to give the user the opportunity to use both PyQt or PySide. In fact, PyQt5 has two options for licences:

• A paid licence
• A copyleft licence (GPL3).

The problem with GPL3 is that it contaminates your code, so basically anything you build with it becomes GPL3. I prefer MIT for all my licences as it is not contaminitaing and allows the user to use the library in commercial products without forcing the need to open every bit of code.

Of course, MIT licence still obligates the user of the library to mention in its products that he used this library and provide a link to the source code, but the code using the library can be closed and the MIT licence is totally respected.

I'm building a GUI library using pygame that will be under MIT licence. Pygame is good but still lack the object oriented structure that PyQT excells in. So I figured that I need to build a wrapper that makes it possible to use PyGame to do more advanced graphics.