cannot connect to X server & output video broken

[brianw0924]

Hi, I'm trying to run this demo on Colab

I run the command:

!ptgaze --mode mpiigaze --video video.mp4 --o ./pytorch_mpiigaze_demo/assets/results/

and get the error. The video still produced, but it's broken (all black, and always 00:00)

and I want to ask will this demo work if my input video is crop to only eyes?

INFO:ptgaze.main:mode: MPIIGaze device: cpu model: name: resnet_preact face_detector: mode: mediapipe dlib_model_path: /root/.ptgaze/dlib/shape_predictor_68_face_landmarks.dat mediapipe_max_num_faces: 3 gaze_estimator: checkpoint: /root/.ptgaze/models/mpiigaze_resnet_preact.pth camera_params: /tmp/camera_params.yaml use_dummy_camera_params: true normalized_camera_params: /usr/local/lib/python3.7/dist-packages/ptgaze/data/normalized_camera_params/mpiigaze.yaml normalized_camera_distance: 0.6 demo: use_camera: false display_on_screen: true wait_time: 1 image_path: null video_path: video.mp4 output_dir: pytorch_mpiigaze_demo/assets/results output_file_extension: avi head_pose_axis_length: 0.05 gaze_visualization_length: 0.05 show_bbox: true show_head_pose: false show_landmarks: false show_normalized_image: false show_template_model: false PACKAGE_ROOT: /usr/local/lib/python3.7/dist-packages/ptgaze

INFO: Created TensorFlow Lite XNNPACK delegate for CPU. : cannot connect to X server

[hysts]

Hi, @brianw0924

Thanks for reporting this. I updated the repo and added demo.ipynb for Colab. Please check out if it works.

[brianw0924]

thanks, I want to ask if it's possible to detect gaze if my input is cropped to only eyes? or is there other recommendation to do this?

I tested, if my input is only eyes, it can't detect (since it need face detection first?)

[hysts]

@brianw0924

Ah, ok, so that's why you chose to use the MPIIGaze model. If the model doesn't work for your input, you might want to check if the way the eye region is cropped is the same as the way the model was trained. Also, due to the limited variation of the training data, the MPIIGaze model can only handle a limited range of face orientations and gaze directions, so that may be the cause of the problem.

[brianw0924]

Thank you, so is it recommended to finetuning on my own dataset?

[hysts]

@brianw0924

Yes, I think that's a good starting point.

[brianw0924]

Last question, sorry bothering.

did u train mpiigaze model with

original image (include 2 eyes, and cropped like a rectangle)

or

nomalized image? (eye patch)

(like the description in the original website: https://www.mpi-inf.mpg.de/departments/computer-vision-and-machine-learning/research/gaze-based-human-computer-interaction/appearance-based-gaze-estimation-in-the-wild)

[hysts]

@brianw0924

The model provided in this repo was trained with the normalized eye patch images.

[brianw0924]

sorry, I want to ask that if I use mpiigaze, the model would expect the input with only eyes, but no head right?

i.e. the head rotation shouldn't affect the gaze direction (pitch, yaw)

[hysts]

@brianw0924

The head pose is necessary. Well, I think you should read the MPIIGaze paper first.