search

minghangz / cpl

CPL: Weakly Supervised Temporal Sentence Grounding with Gaussian-based Contrastive Proposal Learning
56 stars 5 forks source link

All the metrics are nan #2

Closed chaudatascience closed 2 years ago

chaudatascience commented 2 years ago

I try to train and evaluate the results on Charades. For training, python train.py --config-path config/charades/main.json, all the metrics are nan, which leads to R@x = 0

Epoch 1, Batch 50, lr = 0.00005, final_loss = nan, nll_loss = nan, ref_nll_loss = nan, ivc_loss = nan, neg_loss_1 = nan, neg_loss_2 = nan, ref_loss = nan, div_loss = nan, 0.676 seconds/batch 2022-06-05 01:48:50,128 - Epoch 1, Batch 50, lr = 0.00005, final_loss = nan, nll_loss = nan, ref_nll_loss = nan, ivc_loss = nan, neg_loss_1 = nan, neg_loss_2 = nan, ref_loss = nan, div_loss = nan, 0.676 seconds/batch Epoch 1, Batch 100, lr = 0.00010, final_loss = nan, nll_loss = nan, ref_nll_loss = nan, ivc_loss = nan, neg_loss_1 = nan, neg_loss_2 = nan, ref_loss = nan, div_loss = nan, 0.442 seconds/batch 2022-06-05 01:49:00,529 - Epoch 1, Batch 100, lr = 0.00010, final_loss = nan, nll_loss = nan, ref_nll_loss = nan, ivc_loss = nan, neg_loss_1 = nan, neg_loss_2 = nan, ref_loss = nan, div_loss = nan, 0.442 seconds/batch ... Epoch 7, Batch 300, lr = 0.00017, final_loss = nan, nll_loss = nan, ref_nll_loss = nan, ivc_loss = nan, neg_loss_1 = nan, neg_loss_2 = nan, ref_loss = nan, div_loss = nan, 0.207 seconds/batch 2022-06-05 01:58:48,449 - Epoch 7, Batch 300, lr = 0.00017, final_loss = nan, nll_loss = nan, ref_nll_loss = nan, ivc_loss = nan, neg_loss_1 = nan, neg_loss_2 = nan, ref_loss = nan, div_loss = nan, 0.207 seconds/batch Epoch 7, Batch 332, lr = 0.00017, final_loss = nan, nll_loss = nan, ref_nll_loss = nan, ivc_loss = nan, neg_loss_1 = nan, neg_loss_2 = nan, ref_loss = nan, div_loss = nan, 0.207 seconds/batch 2022-06-05 01:58:55,113 - Epoch 7, Batch 332, lr = 0.00017, final_loss = nan, nll_loss = nan, ref_nll_loss = nan, ivc_loss = nan, neg_loss_1 = nan, neg_loss_2 = nan, ref_loss = nan, div_loss = nan, 0.207 seconds/batch save model to checkpoints/charades/base/model-7.pt, num_updates 2324. 2022-06-05 01:58:55,170 - save model to checkpoints/charades/base/model-7.pt, num_updates 2324. | R@1,mIoU nan | R@1,IoU@0.1 0.0000 | R@1,IoU@0.3 0.0000 | R@1,IoU@0.5 0.0000 | R@1,IoU@0.7 0.0000 | R@1,IoU@0.9 0.0000 | R@5,mIoU nan | R@5,IoU@0.1 0.0000 | R@5,IoU@0.3 0.0000 | R@5,IoU@0.5 0.0000 | R@5,IoU@0.7 0.0000 | R@5,IoU@0.9 0.0000 | 2022-06-05 01:59:18,912 - | R@1,mIoU nan | R@1,IoU@0.1 0.0000 | R@1,IoU@0.3 0.0000 | R@1,IoU@0.5 0.0000 | R@1,IoU@0.7 0.0000 | R@1,IoU@0.9 0.0000 | R@5,mIoU nan | R@5,IoU@0.1 0.0000 | R@5,IoU@0.3 0.0000 | R@5,IoU@0.5 0.0000 | R@5,IoU@0.7 0.0000 | R@5,IoU@0.9 0.0000 |

...

As for evaluation, when running

python train.py --config-path config/charades/main.json --resume checkpoints/charades/model-best.pt --eval

It gives

| R@1,mIoU nan | R@1,IoU@0.1 0.0000 | R@1,IoU@0.3 0.0000 | R@1,IoU@0.5 0.0000 | R@1,IoU@0.7 0.0000 | R@1,IoU@0.9 0.0000 | R@5,mIoU nan | R@5,IoU@0.1 0.0000 | R@5,IoU@0.3 0.0000 | R@5,IoU@0.5 0.0000 | R@5,IoU@0.7 0.0000 | R@5,IoU@0.9 0.0000 | 2022-06-05 02:01:29,504 - | R@1,mIoU nan | R@1,IoU@0.1 0.0000 | R@1,IoU@0.3 0.0000 | R@1,IoU@0.5 0.0000 | R@1,IoU@0.7 0.0000 | R@1,IoU@0.9 0.0000 | R@5,mIoU nan | R@5,IoU@0.1 0.0000 | R@5,IoU@0.3 0.0000 | R@5,IoU@0.5 0.0000 | R@5,IoU@0.7 0.0000 | R@5,IoU@0.9 0.0000 |

Am I missing something here? Thank you, Chau

chaudatascience commented 2 years ago

When I switched to CPU, the code worked just fine. Probably because of the cuda version. I re-installed Pytorch and it's ok now.