Open aqr1961 opened 1 year ago
在尝试使用30系显卡和cuda11.x 对网络进行复现时，出现了以下错误，并且一时间难以解决，请问是否有人在训练过程中遇到了类似的错误并解决的呢？
no valid loss at: 20 tensor([0.0549, 0.0525, 0.0539, 0.0214, 0.0289, 0.0317, 0.0337, 0.0171, 0.0350, 0.0333, 0.0333, 0.0333, 0.0443, 0.0393, 0.0443, 0.0295, 0.0187, 0.0295, 0.0264, 0.0165, 0.0264, 0.0534, 0.0351, 0.0534, 0.0630, 0.0630, 0.0630, 0.0341, 0.0307, 0.0529, 0.0605, 0.0053, 0.0369, 0.0398, 0.0310, 0.0300, 0.0432, 0.0452, 0.0451, 0.0363, 0.0379, 0.0527, 0.0287, 0.0339, 0.0279, 0.0397, 0.0452, 0.0308], grad_fn=) no valid loss at: 21 tensor([0.0056, 0.0346, 0.0202, 0.0596, 0.0596, 0.0596, 0.0332, 0.0332, 0.0332, 0.0446, 0.0446, 0.0446, 0.0615, 0.0434, 0.0615, 0.0338, 0.0286, 0.0338, 0.0539, 0.0345, 0.0345, 0.0328, 0.0328, 0.0328, 0.0303, 0.0090, 0.0090, 0.0417, 0.0251, 0.0251, 0.0010, 0.0010, 0.0010, 0.0497, 0.0297, 0.0297, 0.0513, 0.0087, 0.0065, 0.0233, 0.0362, 0.0362, 0.0239, 0.0188, 0.0189, 0.0164, 0.0512, 0.0398], grad_fn=) no valid loss at: 22 tensor([0.0361, 0.0361, 0.0361, 0.0397, 0.0030, 0.0370, 0.0209, 0.0344, 0.0213, 0.0478, 0.0478, 0.0478, 0.0134, 0.0134, 0.0134, 0.0368, 0.0368, 0.0368, 0.0492, 0.0512, 0.0219, 0.0155, 0.0155, 0.0597, 0.0430, 0.0430, 0.0430, 0.0329, 0.0402, 0.0329, 0.0271, 0.0271, 0.0271, 0.0491, 0.0387, 0.0491, 0.0378, 0.0378, 0.0612, 0.0402, 0.0402, 0.0402, 0.0167, 0.0459, 0.0442, 0.0413, 0.0177, 0.0378], grad_fn=) no valid loss at: 23 tensor([0.0232, 0.0533, 0.0232, 0.0126, 0.0273, 0.0411, 0.0492, 0.0492, 0.0492, 0.0277, 0.0277, 0.0320, 0.0533, 0.0533, 0.0310, 0.0249, 0.0249, 0.0332, 0.0422, 0.0384, 0.0384, 0.0506, 0.0506, 0.0316, 0.0287, 0.0349, 0.0660, 0.0586, 0.0586, 0.0586, 0.0214, 0.0265, 0.0455, 0.0349, 0.0399, 0.0405, 0.0159, 0.0159, 0.0159, 0.0417, 0.0417, 0.0417, 0.0424, 0.0373, 0.0373, 0.0321, 0.0321, 0.0321], grad_fn=)
在尝试使用30系显卡和cuda11.x 对网络进行复现时，出现了以下错误，并且一时间难以解决，请问是否有人在训练过程中遇到了类似的错误并解决的呢？
no valid loss at: 20 tensor([0.0549, 0.0525, 0.0539, 0.0214, 0.0289, 0.0317, 0.0337, 0.0171, 0.0350, 0.0333, 0.0333, 0.0333, 0.0443, 0.0393, 0.0443, 0.0295, 0.0187, 0.0295, 0.0264, 0.0165, 0.0264, 0.0534, 0.0351, 0.0534, 0.0630, 0.0630, 0.0630, 0.0341, 0.0307, 0.0529, 0.0605, 0.0053, 0.0369, 0.0398, 0.0310, 0.0300, 0.0432, 0.0452, 0.0451, 0.0363, 0.0379, 0.0527, 0.0287, 0.0339, 0.0279, 0.0397, 0.0452, 0.0308], grad_fn=)
no valid loss at: 21 tensor([0.0056, 0.0346, 0.0202, 0.0596, 0.0596, 0.0596, 0.0332, 0.0332, 0.0332,
0.0446, 0.0446, 0.0446, 0.0615, 0.0434, 0.0615, 0.0338, 0.0286, 0.0338,
0.0539, 0.0345, 0.0345, 0.0328, 0.0328, 0.0328, 0.0303, 0.0090, 0.0090,
0.0417, 0.0251, 0.0251, 0.0010, 0.0010, 0.0010, 0.0497, 0.0297, 0.0297,
0.0513, 0.0087, 0.0065, 0.0233, 0.0362, 0.0362, 0.0239, 0.0188, 0.0189,
0.0164, 0.0512, 0.0398], grad_fn=)
no valid loss at: 22 tensor([0.0361, 0.0361, 0.0361, 0.0397, 0.0030, 0.0370, 0.0209, 0.0344, 0.0213,
0.0478, 0.0478, 0.0478, 0.0134, 0.0134, 0.0134, 0.0368, 0.0368, 0.0368,
0.0492, 0.0512, 0.0219, 0.0155, 0.0155, 0.0597, 0.0430, 0.0430, 0.0430,
0.0329, 0.0402, 0.0329, 0.0271, 0.0271, 0.0271, 0.0491, 0.0387, 0.0491,
0.0378, 0.0378, 0.0612, 0.0402, 0.0402, 0.0402, 0.0167, 0.0459, 0.0442,
0.0413, 0.0177, 0.0378], grad_fn=)
no valid loss at: 23 tensor([0.0232, 0.0533, 0.0232, 0.0126, 0.0273, 0.0411, 0.0492, 0.0492, 0.0492,
0.0277, 0.0277, 0.0320, 0.0533, 0.0533, 0.0310, 0.0249, 0.0249, 0.0332,
0.0422, 0.0384, 0.0384, 0.0506, 0.0506, 0.0316, 0.0287, 0.0349, 0.0660,
0.0586, 0.0586, 0.0586, 0.0214, 0.0265, 0.0455, 0.0349, 0.0399, 0.0405,
0.0159, 0.0159, 0.0159, 0.0417, 0.0417, 0.0417, 0.0424, 0.0373, 0.0373,
0.0321, 0.0321, 0.0321], grad_fn=)