loss warm up 后使用arcloss会挂掉

[aa12356jm]

我在训练过程中，也是按照你脚本中设定的loss warm up =35000来设置，先使用softmax做warm up，然后使用arcloss，到34500个迭代后就挂掉，出现下面错误：

Traceback (most recent call last): File "train_script.py", line 269, in <module> train() File "train_script.py", line 201, in train losses = [Loss(yhat, y.astype(dtype, copy=False)) for yhat, y in zip(outputs, labels)] File "train_script.py", line 201, in <listcomp> losses = [Loss(yhat, y.astype(dtype, copy=False)) for yhat, y in zip(outputs, labels)] File "/home/trainer/anaconda3/envs/mxnet_py36/lib/python3.6/site-packages/mxnet/gluon/block.py", line 548, in __call__ out = self.forward(*args) File "/home/trainer/anaconda3/envs/mxnet_py36/lib/python3.6/site-packages/mxnet/gluon/block.py", line 925, in forward return self.hybrid_forward(ndarray, x, *args, **params) File "/home/trainer/anaconda3/envs/mxnet_py36/lib/python3.6/site-packages/gluonfr-1.0.1-py3.6.egg/gluonfr/loss.py", line 210, in hybrid_forward File "/home/trainer/anaconda3/envs/mxnet_py36/lib/python3.6/site-packages/mxnet/gluon/loss.py", line 347, in hybrid_forward loss = -F.pick(pred, label, axis=self._axis, keepdims=True) File "<string>", line 89, in pick File "/home/trainer/anaconda3/envs/mxnet_py36/lib/python3.6/site-packages/mxnet/_ctypes/ndarray.py", line 92, in _imperative_invoke ctypes.byref(out_stypes))) File "/home/trainer/anaconda3/envs/mxnet_py36/lib/python3.6/site-packages/mxnet/base.py", line 253, in check_call raise MXNetError(py_str(_LIB.MXGetLastError())) mxnet.base.MXNetError: Shape inconsistent, Provided = [256,1], inferred shape=[256,256]

[haoxintong]

你的embedding size 和 batch size设置是多少如果都和256对上的话, 看报错应该是提供的yhat/y中数据有问题, 这个里边会涉及到label one hot, 可能会和数据提供的方式相关

[aa12356jm]

embedding size是128,batchsize 是1024，使用4个v100在跑。我下午再调试看看

[PistonY]

@aa12356jm Check out you label shape is totally same as our datasets eg: vgg, emore ...