search

kyzhouhzau / BERT-NER

Use Google's BERT for named entity recognition （CoNLL-2003 as the dataset）.
MIT License
1.23k stars 335 forks source link

Question: training the model without init_checkpoint #15

Closed sudhanshu817 closed 5 years ago

sudhanshu817 commented 5 years ago

INFO:tensorflow:Error recorded from training_loop: local variable 'initialized_variable_names' referenced before assignment INFO:tensorflow:training_loop marked as finished WARNING:tensorflow:Reraising captured error Traceback (most recent call last): File "BERT_NER.py", line 612, in tf.app.run() File "C:\Users\Sudha\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\platform\app.py", line 125, in run _sys.exit(main(argv)) File "BERT_NER.py", line 545, in main estimator.train(input_fn=train_input_fn, max_steps=num_train_steps) File "C:\Users\Sudha\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\contrib\tpu\python\tpu\tpu_estimator.py", line 2409, in train rendezvous.raise_errors() File "C:\Users\Sudha\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\contrib\tpu\python\tpu\error_handling.py", line 128, in raise_errors six.reraise(typ, value, traceback) File "C:\Users\Sudha\Anaconda3\envs\tensorflow\lib\site-packages\six.py", line 693, in reraise raise value File "C:\Users\Sudha\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\contrib\tpu\python\tpu\tpu_estimator.py", line 2403, in train saving_listeners=saving_listeners File "C:\Users\Sudha\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\estimator\estimator.py", line 354, in train loss = self._train_model(input_fn, hooks, saving_listeners) File "C:\Users\Sudha\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\estimator\estimator.py", line 1207, in _train_model return self._train_model_default(input_fn, hooks, saving_listeners) File "C:\Users\Sudha\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\estimator\estimator.py", line 1237, in _train_model_default features, labels, model_fn_lib.ModeKeys.TRAIN, self.config) File "C:\Users\Sudha\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\contrib\tpu\python\tpu\tpu_estimator.py", line 2195, in _call_model_fn features, labels, mode, config) File "C:\Users\Sudha\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\estimator\estimator.py", line 1195, in _call_model_fn model_fn_results = self._model_fn(features=features, kwargs) File "C:\Users\Sudha\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\contrib\tpu\python\tpu\tpu_estimator.py", line 2479, in _model_fn features, labels, is_export_mode=is_export_mode) File "C:\Users\Sudha\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\contrib\tpu\python\tpu\tpu_estimator.py", line 1259, in call_without_tpu return self._call_model_fn(features, labels, is_export_mode=is_export_mode) File "C:\Users\Sudha\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\contrib\tpu\python\tpu\tpu_estimator.py", line 1533, in _call_model_fn estimator_spec = self._model_fn(features=features, kwargs) File "BERT_NER.py", line 419, in model_fn if var.name in initialized_variable_names: UnboundLocalError: local variable 'initialized_variable_names' referenced before assignment

Training the model without using the init_checkpoint flag returns this error

kyzhouhzau commented 5 years ago

Have you download checkpoint from bert by google?

sudhanshu817 commented 5 years ago

The google BERT implementation provides an option to train without using their checkpoint. So i wanted to try that. Found the issue though: If I do not specify the init.checkpoint flag, the initialized_variable_names is not initialized, but you are trying to access this variable below. I had marked both in bold

if init_checkpoint: (assignment_map, initialized_variable_names) = modeling.get_assignment_map_from_checkpoint(tvars,init_checkpoint) tf.train.init_from_checkpoint(init_checkpoint, assignment_map) if use_tpu: def tpu_scaffold(): tf.train.init_from_checkpoint(init_checkpoint, assignment_map) return tf.train.Scaffold() scaffold_fn = tpu_scaffold else: tf.train.init_from_checkpoint(init_checkpoint, assignment_map) tf.logging.info(" Trainable Variables ") for var in tvars: init_string = "" if var.name in initialized_variable_names: init_string = ", INIT_FROM_CKPT" tf.logging.info(" name = %s, shape = %s%s", var.name, var.shape, init_string)

sudhanshu817 commented 5 years ago

Found the solution. You can just comment out this past or add a if condition to execute it only if init_checkpoint is assigned and not null. for var in tvars: init_string = "" if var.name in initialized_variable_names: init_string = ", INIT_FROM_CKPT" tf.logging.info(" name = %s, shape = %s%s", var.name, var.shape, init_string)