search

ARCC-RACE / deepracer-for-dummies

a quick way to get up and running with local deepracer training environment
66 stars 28 forks source link

Training crashes after arbitrary iterations #50

Open sushil-bharati opened 4 years ago

sushil-bharati commented 4 years ago

Here is how the error message looks like :

/usr/local/lib/python3.6/dist-packages/numpy/core/_methods.py:26: RuntimeWarning: invalid value encountered in reduce
  return umr_maximum(a, axis, None, out, keepdims)
/usr/local/lib/python3.6/dist-packages/numpy/core/_methods.py:29: RuntimeWarning: invalid value encountered in reduce
  return umr_minimum(a, axis, None, out, keepdims)
{"simapp_exception": {"version": "1.0", "date": "2019-09-11 02:54:19.345983", "function": "training_worker", "message": "NaN detected in loss function, aborting training. Job failed!", "exceptionType": "training_worker.exceptions", "eventType": "system_error", "errorCode": "503"}}