blackfeather-wang / ISDA-for-Deep-Networks

An efficient implicit semantic augmentation method, complementary to existing non-semantic techniques.
582 stars 93 forks source link

RuntimeError: copy_if failed to synchronize: device-side assert triggered #27

Open lihuikenny opened 3 years ago

lihuikenny commented 3 years ago

weight_CV: tensor([[[nan, nan, nan, ..., nan, nan, nan], [nan, nan, nan, ..., nan, nan, nan], [nan, nan, nan, ..., nan, nan, nan], ..., [nan, nan, nan, ..., nan, nan, nan], [nan, nan, nan, ..., nan, nan, nan], [nan, nan, nan, ..., nan, nan, nan]],

    [[nan, nan, nan,  ..., nan, nan, nan],
     [nan, nan, nan,  ..., nan, nan, nan],
     [nan, nan, nan,  ..., nan, nan, nan],
     ...,
     [nan, nan, nan,  ..., nan, nan, nan],
     [nan, nan, nan,  ..., nan, nan, nan],
     [nan, nan, nan,  ..., nan, nan, nan]],

训练自己的数据的时候报错了,weight_cv一直都是NAN 我尝试过以下这些方案解决,结果不行: 1、调整学习率 2、对FC权重和特征进行归一化

我的分类数是3000多类,我看代码weight_cv,是对label的一个onehot转换。按理应该不会出现这种错误才对。 希望作者能指点一下,谢谢

lihuikenny commented 3 years ago

补充一下:不过二分类好像没问题,