search

guillaumegenthial / tf_ner

Simple and Efficient Tensorflow implementations of NER models with tf.estimator and tf.data
Apache License 2.0
923 stars 275 forks source link

RuntimeError: There was no new checkpoint after the training. Eval status: missing checkpoint #36

Open harirajeev opened 5 years ago

harirajeev commented 5 years ago

While running this model on the CONLL2003 data ( TF 1.11) , i am getting this error

global_step/sec: 3.00248 loss = 5.6152534, step = 101 (33.300 sec) global_step/sec: 2.88663 loss = 4.2953496, step = 201 (34.641 sec) global_step/sec: 3.08292 loss = 3.5392659, step = 301 (32.438 sec) Saving checkpoints for 344 into results\model\model.ckpt. Estimator is not trained yet. Will start an evaluation when a checkpoint is ready.

RuntimeError: There was no new checkpoint after the training. Eval status: missing checkpoint

Could you please paste a picture of your model weights directory ?.

thanks

guillaumegenthial commented 5 years ago

Hi @harirajeev , What's probably happening is that by default the evaluation will be triggered every 120 seconds, if a new checkpoint is available. If your computer is too slow, and it does not save a new checkpoint in this lapse of time, it will try to run evaluation and skip as no new checkpoint is available.

errolPereira commented 3 years ago

How to delay the eval trigger? @guillaumegenthial