search

Jiani-CAO / ASGaze

This is the repo for SenSys 2022 paper: "Gaze Tracking on Any Surface with Your Phone".
18 stars 3 forks source link

cv2.error: OpenCV(3.4.2) C:\projects\opencv-python\opencv\modules\imgproc\src\shapedescr.cpp:310: error: (-201:Incorrect size of input array) There should be at least 5 points to fit the ellipse in function 'cv::fitEllipse' #1

Open pvdongen opened 1 year ago

pvdongen commented 1 year ago

Hi, Your code works perfectly with your database. When I run your code on a custom video, I get the following error:

"""### Step 1: Detect and crop the eye using Dilb"""

# There are two types of data, one for gaze tracking ("tracking"), and the other one for mapping ("corner_1", "corner_2", "corner_3", "corner_4")
detection_list = ["tracking","corner_1","corner_2","corner_3","corner_4"]

detection_main(data_path,detection_list,rotate=True)

(testenv) C:\Users\PD\Downloads\ASGaze>C:/Users/PD/miniconda3/envs/testenv/python.exe c:/Users/PD/Downloads/ASGaze/main.py
tracking
Extracting frames from ./Database\raw_data\tracking to ./Database\raw_data\tracking\frame/ ...
Successfully extract 4500 frames.
Traceback (most recent call last):
  File "c:/Users/PD/Downloads/ASGaze/main.py", line 27, in <module>
    detection_main(data_path,detection_list,rotate=True)
  File "c:\Users\PD\Downloads\ASGaze\iris_boundary_detector\data_sources\detection.py", line 127, in detection_main
    dlib_detect(frame_path, crop_path)
  File "c:\Users\PD\Downloads\ASGaze\iris_boundary_detector\data_sources\detection.py", line 95, in dlib_detect
    all_valid = all([item != False for item in eyes])
TypeError: 'bool' object is not iterable

Changing all_valid = all([item != False for item in eyes]) to all_valid = all([eyes]) fixes this problem. But then in step 2 another error pops up:

"""### Step 2: Iris boundary detector"""

gaze_inference = "./iris_boundary_detector/configs/gaze_inference.json"
test_name_list = ["tracking","corner_1","corner_2","corner_3","corner_4"]
for test_name in test_name_list:
    print(test_name) 
    inference_main(gaze_inference,test_name,data_flag=0,video=True)
print('DONE')

tracking
runs_dir C:\Users\PD\Downloads\ASGaze\Database\raw_data\tracking\0207-1231
load checkpoingts from C:\Users\PD\Downloads\ASGaze\Database\output\CMonitor-0207-1231
datapath C:\Users\PD\Downloads\ASGaze\Database\raw_data\tracking\crop\left
Load 3790 samples of tracking/crop Dataset left split from C:/Users/PD/Downloads/ASGaze/Database/raw_data\tracking/crop\left
datapath C:\Users\PD\Downloads\ASGaze\Database\raw_data\tracking\crop\right
Load 3790 samples of tracking/crop Dataset right split from C:/Users/PD/Downloads/ASGaze/Database/raw_data\tracking/crop\right
C:\Users\PD\miniconda3\envs\testenv\lib\site-packages\torch\nn\functional.py:3509: UserWarning: nn.functional.upsample is deprecated. Use nn.functional.interpolate instead.
  warnings.warn("nn.functional.upsample is deprecated. Use nn.functional.interpolate instead.")
C:\Users\PD\miniconda3\envs\testenv\lib\site-packages\torch\nn\functional.py:3635: UserWarning: Default upsampling behavior when mode=bilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
  "See the documentation of nn.Upsample for details.".format(mode)
c:\Users\PD\Downloads\ASGaze\iris_boundary_detector\inference.py:50: RuntimeWarning: invalid value encountered in true_divide
  entropy_partial = (entropy_partial-entropy_partial.min())/(entropy_partial.max()-entropy_partial.min())*255
Traceback (most recent call last):
  File "c:/Users/PD/Downloads/ASGaze/main.py", line 35, in <module>
    inference_main(gaze_inference,test_name,data_flag=0,video=True)
  File "c:\Users\PD\Downloads\ASGaze\iris_boundary_detector\inference.py", line 219, in inference_main
    inference(val_loader_left, model, "left", out_dir, video)
  File "c:\Users\PD\Downloads\ASGaze\iris_boundary_detector\inference.py", line 71, in inference
    flag_l,pt1_l,pt2_l,flag_r,pt1_r,pt2_r = refinement.is_feature_matching_all(entropy_partial_list[0].astype(np.uint8),entropy_partial_list[1].astype(np.uint8))
  File "c:\Users\PD\Downloads\ASGaze\iris_boundary_detector\utils\refinement.py", line 100, in is_feature_matching_all
    center_x = find_center_x(img1)
  File "c:\Users\PD\Downloads\ASGaze\iris_boundary_detector\utils\refinement.py", line 67, in find_center_x
    e1= cv2.fitEllipse(points_np)
cv2.error: OpenCV(3.4.2) C:\projects\opencv-python\opencv\modules\imgproc\src\shapedescr.cpp:310: error: (-201:Incorrect size of input array) There should be at least 5 points to fit the ellipse in function 'cv::fitEllipse'

Environment:

Jiani-CAO commented 1 year ago

Hi,

  1. You can check whether the saved entropy_partital is complete, i.e. whether both left and right parts exist. Since the "feature match" algorithm matches the left and right parts separately, if a frame is missing a left or a right part, then it will not match.

  2. If the above is correct, check that whether the matched "_entropy_p_fd.png" contains more than five pixels. This is because the 'cv::fitEllipse' function requires at least five points to fit an ellipse.

By the way, if you use your own dataset for inference, it is better to train the network with your own dataset as well, for better accuracy.

pvdongen commented 1 year ago

Hi,

  1. You can check whether the saved entropy_partital is complete, i.e. whether both left and right parts exist. Since the "feature match" algorithm matches the left and right parts separately, if a frame is missing a left or a right part, then it will not match.
  2. If the above is correct, check that whether the matched "_entropy_p_fd.png" contains more than five pixels. This is because the 'cv::fitEllipse' function requires at least five points to fit an ellipse.

By the way, if you use your own dataset for inference, it is better to train the network with your own dataset as well, for better accuracy.

Thank you for your quick reply.

  1. I think something is going wrong with my entropy_partial. Though both left and right parts exist, it almost only contains: 
    ...
[nan nan nan ... nan nan nan]
[nan nan nan ... nan nan nan]
[nan nan nan ... nan nan nan]]
[[nan nan nan ... nan nan nan]
[nan nan nan ... nan nan nan]
[nan nan nan ... nan nan nan]
...
  2. Finally, the only folder containing png's with "entropy" even mentioned in the name is the folder C:\Users\PD\Downloads\ASGaze\Database\raw_data\tracking\0207-1231\InferResults\visual_left and this folder only contains "number_entropy" and "number_entropy_p" png's. The other folders under "InferResults" are simply empty.

Thanks for the tip, I will try to train the network with my own dataset but I first want to see the results with the current network.

Jiani-CAO commented 1 year ago

Hi,

  1. Is "entropy.png" normal, please?
  2. Theoretically, "entropy_partital" does not appear to be nan, as this line of code entropy_partial[np.isnan(entropy_partial)] = 0 deals specifically with this case.
pvdongen commented 1 year ago

Hi,

  1. Is "entropy.png" normal, please?
  2. Theoretically, "entropy_partital" does not appear to be nan, as this line of code entropy_partial[np.isnan(entropy_partial)] = 0 deals specifically with this case.

Hi,

  1. The files 'entropy.png' don't look normal (compared to the ones you provide in your database). Furthermore, your visual_left folder has 4 different files (crop, entropy, entropy_p and entropy_p_fr) while mine only has 2 different files (entropy and entropy_p).
  2. You are right. Something is going wrong because there should be no nan.