search

ARCC-RACE / deepracer-for-dummies

a quick way to get up and running with local deepracer training environment
66 stars 28 forks source link

Randomly throws error after N iteration? #48

Open sushil-bharati opened 4 years ago

sushil-bharati commented 4 years ago

Here is the error report: 

/usr/local/lib/python3.6/dist-packages/numpy/core/_methods.py:26: RuntimeWarning: invalid value encountered in reduce
  return umr_maximum(a, axis, None, out, keepdims)
/usr/local/lib/python3.6/dist-packages/numpy/core/_methods.py:29: RuntimeWarning: invalid value encountered in reduce
  return umr_minimum(a, axis, None, out, keepdims)
{"simapp_exception": {"version": "1.0", "date": "2019-08-30 04:29:37.384401", "function": "training_worker", "message": "NaN detected in loss function, aborting training. Job failed!", "exceptionType": "training_worker.exceptions", "eventType": "system_error", "errorCode": "503"}}
sushil-bharati commented 4 years ago

This problem occurs when reward is too low!? This is what I've found after some play around. Cannot be concluded.