This repository is the official implementation of GaTector, which studies the newly proposed task, gaze object prediction. In this work, we build a novel framework named GaTector to tackle the gaze object prediction problem in a unified way. Particularly, a specific-general-specific (SGS) feature extractor is firstly proposed to utilize a shared backbone to extract general features for both scene and head images. To better consider the specificity of inputs and tasks, SGS introduces two input-specific blocks before the shared backbone and three task-specific blocks after the shared backbone. Specifically, a novel defocus layer is designed to generate object-specific features for object detection task without losing information or requiring extra computations. Moreover, the energy aggregation loss is introduced to guide the gaze heatmap to concentrate on the stared box. In the end, we propose a novel mDAP metric that can reveal the difference between boxes even when they share no overlapping area. Extensive experiments on the GOO dataset verify the superiority of our method in all three tracks, i.e., object detection, gaze estimation, and gaze object prediction.
When I try to train a model with your own goo_real_data, I get this error. How can I fix it have you faced before? I just tried to train.
Start Train
Epoch 1/50: 0%| | 0/225 [00:07<?, ?it/s<class 'dict'>]
Traceback (most recent call last):
File "main.py", line 172, in
epoch_step, epoch_step_val, gen, gen_val, end_epoch, train_mode, Cuda)
File "/root/GaTector-A-Unified-Framework-for-Gaze-Object-Prediction/lib/utils/utils_fit.py", line 47, in fit_one_epoch
outputs = model_train(images, head, faces,train_mode)
File "/opt/conda/envs/Gatector/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, *kwargs)
File "/opt/conda/envs/Gatector/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 169, in forward
return self.gather(outputs, self.output_device)
File "/opt/conda/envs/Gatector/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 181, in gather
return gather(outputs, output_device, dim=self.dim)
File "/opt/conda/envs/Gatector/lib/python3.7/site-packages/torch/nn/parallel/scatter_gather.py", line 78, in gather
res = gather_map(outputs)
File "/opt/conda/envs/Gatector/lib/python3.7/site-packages/torch/nn/parallel/scatter_gather.py", line 73, in gather_map
return type(out)(map(gather_map, zip(outputs)))
File "/opt/conda/envs/Gatector/lib/python3.7/site-packages/torch/nn/parallel/scatter_gather.py", line 73, in gather_map
return type(out)(map(gather_map, zip(*outputs)))
TypeError: zip argument #1 must support iteration
aliguran
commented
1 year ago
Hello,
When I try to train a model with your own goo_real_data, I get this error. How can I fix it have you faced before? I just tried to train.
Start Train Epoch 1/50: 0%| | 0/225 [00:07<?, ?it/s<class 'dict'>] Traceback (most recent call last): File "main.py", line 172, in
epoch_step, epoch_step_val, gen, gen_val, end_epoch, train_mode, Cuda)
File "/root/GaTector-A-Unified-Framework-for-Gaze-Object-Prediction/lib/utils/utils_fit.py", line 47, in fit_one_epoch
outputs = model_train(images, head, faces,train_mode)
File "/opt/conda/envs/Gatector/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, *kwargs)
File "/opt/conda/envs/Gatector/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 169, in forward
return self.gather(outputs, self.output_device)
File "/opt/conda/envs/Gatector/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 181, in gather
return gather(outputs, output_device, dim=self.dim)
File "/opt/conda/envs/Gatector/lib/python3.7/site-packages/torch/nn/parallel/scatter_gather.py", line 78, in gather
res = gather_map(outputs)
File "/opt/conda/envs/Gatector/lib/python3.7/site-packages/torch/nn/parallel/scatter_gather.py", line 73, in gather_map
return type(out)(map(gather_map, zip(outputs)))
File "/opt/conda/envs/Gatector/lib/python3.7/site-packages/torch/nn/parallel/scatter_gather.py", line 73, in gather_map
return type(out)(map(gather_map, zip(*outputs)))
TypeError: zip argument #1 must support iteration
Thanks