search

francescotonini / human-gaze-target-detection-transformer

An implementation of the paper "End-to-End Human-Gaze-Target Detection with Transformers"
16 stars 2 forks source link

Regarding the issue of model accuracy #4

Closed chenwenhe11 closed 9 months ago

chenwenhe11 commented 11 months ago

Why is the accuracy of the model I evaluated after training much lower than the accuracy reported in the paper? Have you experienced this situation?

francescotonini commented 11 months ago

Hi, can you give me more context? What did you train, and which results did you get? Thanks.

chenwenhe11 commented 10 months ago

thank you for your reply! After I used the default model parameters to train on Gazefollow, the accuracy obtained was much lower than the accuracy in the paper, as follows: The model parameters are set as follows: CONFIG ├── datamodule │ └── target: src.datamodules.gazefollow_datamodule.GazeFollowDataModule
│ data_dir: /home/cwh/human-gaze-target-detection-transformer/data/gazefol │ batch_size: 4
│ num_workers: 8
│ pin_memory: true
│ persistent_workers: true
│ num_queries: 20
│ gaze_heatmap_size: 64
│ gaze_heatmap_default_value: 1
│ use_aux_heads_dataset: true
│ use_gaze_inside_only: false
│
├── model │ └── target: src.models.hgttr_module.HGTTRLitModule
│ net:
│ target: src.models.components.HGTTR.HGTTR
│ num_queries: 20
│ gaze_heatmap_size: 64
│ aux_loss: true
│ optimizer:
│ target: torch.optim.AdamW
│ partial: true
│ lr: 0.0001 

   Test metric         ┃        DataLoader 0         ┃

┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━

The evaluation results are as follows： │ val/gaze_heatmap_auc │ 0.8064532279968262 │ │ val/gaze_map │ 0.16594065725803375 │ │ val/gaze_point_avg_distance │ 0.2358965426683426 │ │ val/gaze_point_min_distance │ 0.16742484271526337 │ │ val/loss │ 7.8983845710754395 │ │ val/loss_bbox │ 0.9295105338096619 │ │ val/loss_bbox_0 │ 0.9341297149658203 │ │ val/loss_bbox_1 │ 0.9251079559326172 │ │ val/loss_bbox_2 │ 0.9227374792098999 │ │ val/loss_bbox_3 │ 0.919655978679657 │ │ val/loss_bbox_4 │ 0.9282872080802917 │ │ val/loss_gaze_heatmap │ 0.017752941697835922 │ │ val/loss_gaze_heatmap_0 │ 0.0179615318775177 │ │ val/loss_gaze_heatmap_1 │ 0.017875296995043755 │ │ val/loss_gaze_heatmap_2 │ 0.017761368304491043 │ │ val/loss_gaze_heatmap_3 │ 0.01776522397994995 │ │ val/loss_gaze_heatmap_4 │ 0.017777247354388237 │ │ val/loss_label │ 0.3860939145088196 │ │ val/loss_label_0 │ 0.35364145040512085 │ │ val/loss_label_1 │ 0.3673977553844452 │ │ val/loss_label_2 │ 0.37033170461654663 │ │ val/loss_label_3 │ 0.3788999021053314 │ │ val/loss_label_4 │ 0.3778909146785736 │

Could you please help me find out where the problem lies?

francescotonini commented 10 months ago

Try to increase the weight of the heatmap loss here to 100, it should fix your issue.