improved error message text for all NAN in the GW loss

USGS-R / river-dl

Deep learning model for predicting environmental variables on river systems

Creative Commons Zero v1.0 Universal

21 stars 15 forks source link

improved error message text for all NAN in the GW loss #131

Closed janetrbarclay closed 3 years ago

janetrbarclay commented 3 years ago

Updating the error message for when the loss function becomes all NAN in the gw loss. We're still experiencing intermittent issues with this on TG and this causes the job to end with an error rather than running to completion with weird values.