Open GJemini opened 1 year ago
Hello,i'm very grateful for your work.but when i was training the model,there is always a problem that the loss mutated and validation images became irregular or all black，with the iou metrics dropping down to almost 0.i trained the model with a single RTX2080ti and set the batchsize as 4,while the other parameters remained original.later i reduced the max_lr to 1e-3 but still cannot work.here's the wandb report: https://wandb.ai/jemini/cross_view_transformers_test/runs/0404_144821?workspace=user-jemini then i found a pretrained checkpoint later.i trained with it again, and still the same problem. https://wandb.ai/jemini/cross_view_transformers_test/runs/0408_131432?workspace=user-jemini hope for your help!
Hello,i'm very grateful for your work.but when i was training the model,there is always a problem that the loss mutated and validation images became irregular or all black，with the iou metrics dropping down to almost 0.i trained the model with a single RTX2080ti and set the batchsize as 4,while the other parameters remained original.later i reduced the max_lr to 1e-3 but still cannot work.here's the wandb report: https://wandb.ai/jemini/cross_view_transformers_test/runs/0404_144821?workspace=user-jemini then i found a pretrained checkpoint later.i trained with it again, and still the same problem. https://wandb.ai/jemini/cross_view_transformers_test/runs/0408_131432?workspace=user-jemini hope for your help!