Semantic Role Labeling training failed

[ManOki]

Hi,

i got this error running demo/semantic_role_labeling/train.sh, as described in the tutorial.

Setup is a Virtualbox VM (4 cores, 10GB), with docker image paddledev/paddle:0.10.0rc3 (avx, no gpu) and setting a new PYTHONHOME (like in #1785)

I0429 02:30:03.702711    30 TrainerInternal.cpp:181]  Pass=363 Batch=895 samples=
148647 AvgCost=0.00124597 Eval: __sum_evaluator_0__=1.34547e-05
I0429 02:30:43.996294    30 Tester.cpp:115]  Test samples=148647 cost=0.00108598 Eval: __sum_evaluator_0__=6.72735e-06
I0429 02:30:44.011178    30 GradientMachine.cpp:64] Saving parameters to ./output/pass-00363
..........................................................................................................................
..........................................................................................................................
..........................................................................................................................
..........................................................................................................................
..........................................................................................................................
..........................................................................................................................
F0429 02:32:09.565973    30 LinearChainCRF.cpp:40] Check failed: sum > 0 (-nan vs. 0)
*** Check failure stack trace: ***
    @           0x94966d  google::LogMessage::Fail()
    @           0x94d1b5  google::LogMessage::SendToLog()
    @           0x949193  google::LogMessage::Flush()
    @           0x94e6ce  google::LogMessageFatal::~LogMessageFatal()
    @           0x687b88  paddle::normalizeL1()
    @           0x687e0d  paddle::LinearChainCRF::forward()
    @           0x6d1adf  paddle::CRFLayer::forward()
    @           0x628284  paddle::NeuralNetwork::forward()
    @           0x614f03  paddle::GradientMachine::forwardBackward()
    @           0x7982b3  paddle::TrainerInternal::forwardBackwardBatch()
    @           0x798838  paddle::TrainerInternal::trainOneBatch()
    @           0x793770  paddle::Trainer::trainOneDataBatch()
    @           0x7961de  paddle::Trainer::trainOnePass()
    @           0x797445  paddle::Trainer::train()
    @           0x5fa740  main
    @     0x7fe62d69fb45  __libc_start_main
    @           0x609369  (unknown)
    @              (nil)  (unknown)
/usr/bin/paddle: line 113:    30 Aborted                 (core dumped) ${DEBUGGER} $MYDIR/../opt/paddle/bin/paddle_trainer

Greets, ManOki

[lcy-seso]

• From the log, I guess the training process may crash at this line https://github.com/PaddlePaddle/Paddle/blob/develop/paddle/gserver/layers/LinearChainCRF.cpp#L84. This may due to highly accurate fitting that the edge cases of the optimization may be encountered.

• In, PaddlePaddle LinearChainCRF only calculates transition feature, the function normalizeL1 is used to avoid underflow or overflow when calculating the normalization factor. From the code, I guess underflow may occur.

• According to the log, I find the cost is really low. The training can be stopped, or overfitting may have already happened.