Inference Error

[ThePassedWind]

When running python run.py --source path/to/video --out path/to/output_folder --future 0 --hoi-thres 0.3 --print Some errors occurred:

Traceback (most recent call last): File "D:\pythonWorkPlace\HOI-Tracking\hoi-prediction-gaze-transformer-main\run.py", line 424, in main(opt) File "D:\codingSoftware\Anaconda\envs\hoi_gaze_cpu\lib\site-packages\torch\autograd\grad_mode.py", line 28, in decorate_context return func(*args, **kwargs) File "D:\pythonWorkPlace\HOI-Tracking\hoi-prediction-gaze-transformer-main\run.py", line 184, in main object_tracking_module.warmup(frame.to(device), frame0) File "D:\pythonWorkPlace\HOI-Tracking\hoi-prediction-gaze-transformer-main\modules\object_tracking\object_tracking.py", line 81, in warmup self.object_tracker.track(frame, original_frame) File "D:\pythonWorkPlace\HOI-Tracking\hoi-prediction-gaze-transformer-main\modules\object_tracking\yolov5_deepsort.py", line 166, in track bbox_detected, label_detected, confidence_detected = self.yolov5_object_detector.detect(frame, original_frame) File "D:\pythonWorkPlace\HOI-Tracking\hoi-prediction-gaze-transformer-main\modules\object_tracking\yolov5_deepsort.py", line 70, in detect pred = non_max_suppression(pred, self.conf_thres, self.iou_thres, classes=self.classes, time_limit=1.0) File "D:\pythonWorkPlace\HOI-Tracking\hoi-prediction-gaze-transformer-main\modules\object_tracking\yolov5_deepsort.py", line 227, in non_max_suppression bs = prediction.shape[0] # batch size AttributeError: 'list' object has no attribute 'shape'

[nizhf]

Thank you for your interest in our work. I cloned a newer version of YOLOv5 after cleaning up the code, but the new version has a different output type... I updated some codes to fix this and several other bugs.

[ThePassedWind]

Thank you for your interest in our work. I cloned a newer version of YOLOv5 after cleaning up the code, but the new version has a different output type... I updated some codes to fix this and several other bugs.

Thanks!

[ThePassedWind]

By the way, Can your model be applied to a video where no heads appear？

[nizhf]

Yes, if no head is detected for a human, the gaze of this human is set to a matrix of 0.

[ThePassedWind]

Is there any other approach to model for first-person perspective？（no head）

[nizhf]

For the first-person perspective (egocentric vision), the task is often called "action recognition". There are also plenty of works in that field, e.g., Anticipative Video Transformer (Girdhar et al., ICCV 2021) and the dataset EPIC-KITCHENS-100 (Damen et al., IJCV 2022). In addition, gaze information is also useful in egocentric action recognition tasks. And the gaze is much easier to obtain because you can wear an eye tracker to accurately detect your gaze direction and then align it with the first-person video.

[ThePassedWind]

OK! Got it! But I still have a question. If the gaze of this human is set to a matrix of 0, would your model output a complete result? I tried to infer a video(no head) just now, and the error is:

[ThePassedWind]

OK! Got it! But I still have a question. If the gaze of this human is set to a matrix of 0, would your model output a complete result? I tried to infer a video(no head) just now, and the error is:

Maybe your model cannot infer the video(no head) in third-person perspective?

[nizhf]

Did you pull all changes of my recent commits? I also fixed that issue in https://github.com/nizhf/hoi-prediction-gaze-transformer/blob/main/common/model_utils.py#L423. This error happens when no human-object pair is detected in a sliding window.

[ThePassedWind]

Thank you very much! I will update it later and try again~