bubbliiiing / yolact-pytorch

这是一个yolact-pytorch的库,可用于训练自己的数据集
MIT License
71 stars 12 forks source link

自定义数据集上训练过程中出现loss=='nan' #1

Open serendipity999 opened 2 years ago

serendipity999 commented 2 years ago

默认参数未变更,在自己的数据上微调,调试看到某一时刻output为nan,然后loss计算出错,请问会是什么原因呢?

SAWRJJ commented 1 year ago

我是在训练的过程中出现了loss = ‘inf’的情况 朋友你的情况最后是怎么解决的

bubbliiiing commented 1 year ago

试试全解冻 ,不冻结

abhigoku10 commented 1 year ago

@bubbliiiing even i am facing the same issue , can u please elaborate the solution

bubbliiiing commented 1 year ago

I want to, but I don't know how to reproduce it

abhigoku10 commented 1 year ago

@bubbliiiing i have attached the log file in this comment pls check even if i am having loss values for all the different loss yet total loss is nan/inf yolact_pytorch_bubbling_colab_log.txt

bubbliiiing commented 1 year ago

image Maybe the list of mean is empty

ruonan126 commented 1 year ago

请问loss = 'inf'这个问题有解决方法吗?

Andrew-Ma-2001 commented 1 year ago

Same problem