ganeshjawahar
commented
3 years ago Thanks. I solved this issue. I wasn't passing in the right config file.
Closed ganeshjawahar closed 3 years ago
ganeshjawahar
commented
3 years ago Thanks. I solved this issue. I wasn't passing in the right config file.
I've trained GPT-2 model using this repository for few epochs and I'm unable to generate text using the trained model.
When I use the "demo" script, !PYTHONPATH=$(pwd) python scripts/demo.py -ckpt_fn "/content/drive/MyDrive/Colab Notebooks/model.ckpt-1140000_last_ckpt/model.ckpt-1140000" -min_len 50 -samples 10
I get the following error: /content/gpt2-ml INFO:tensorflow:Restoring parameters from /content/drive/MyDrive/Colab Notebooks/model.ckpt-1140000_last_ckpt/model.ckpt-1140000 Traceback (most recent call last): File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call return fn(*args) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn target_list, run_metadata) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.NotFoundError: 2 root error(s) found. (0) Not found: Key newslm/embeddings/LayerNorm_embed_norm/beta not found in checkpoint [[{{node save/RestoreV2}}]] [[save/RestoreV2/_301]] (1) Not found: Key newslm/embeddings/LayerNorm_embed_norm/beta not found in checkpoint [[{{node save/RestoreV2}}]] 0 successful operations. 0 derived errors ignored.
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/training/saver.py", line 1290, in restore {self.saver_def.filename_tensor_name: save_path}) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py", line 956, in run run_metadata_ptr) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py", line 1180, in _run feed_dict_tensor, options, run_metadata) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run run_metadata) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.NotFoundError: 2 root error(s) found. (0) Not found: Key newslm/embeddings/LayerNorm_embed_norm/beta not found in checkpoint [[node save/RestoreV2 (defined at /usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/ops.py:1748) ]] [[save/RestoreV2/_301]] (1) Not found: Key newslm/embeddings/LayerNorm_embed_norm/beta not found in checkpoint [[node save/RestoreV2 (defined at /usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/ops.py:1748) ]] 0 successful operations. 0 derived errors ignored.
Original stack trace for 'save/RestoreV2': File "scripts/demo.py", line 170, in
saver = tf.train.Saver()
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/training/saver.py", line 828, in init
self.build()
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/training/saver.py", line 840, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/training/saver.py", line 878, in _build
build_restore=build_restore)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/training/saver.py", line 508, in _build_internal
restore_sequentially, reshape)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/training/saver.py", line 328, in _AddRestoreOps
restore_sequentially)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/training/saver.py", line 575, in bulk_restore
return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/ops/gen_io_ops.py", line 1696, in restore_v2
name=name)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
op_def=op_def)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
attrs, op_def, compute_device)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
op_def=op_def)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/ops.py", line 1748, in init
self._traceback = tf_stack.extract_stack()
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/training/saver.py", line 1300, in restore names_to_keys = object_graph_key_mapping(save_path) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/training/saver.py", line 1618, in object_graph_key_mapping object_graph_string = reader.get_tensor(trackable.OBJECT_GRAPH_PROTO_KEY) File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/pywrap_tensorflow_internal.py", line 915, in get_tensor return CheckpointReader_GetTensor(self, compat.as_bytes(tensor_str)) tensorflow.python.framework.errors_impl.NotFoundError: Key _CHECKPOINTABLE_OBJECT_GRAPH not found in checkpoint
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "scripts/demo.py", line 171, in
saver.restore(sess, args.ckpt_fn)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/training/saver.py", line 1306, in restore
err, "a Variable name or other graph key that is missing")
tensorflow.python.framework.errors_impl.NotFoundError: Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:
2 root error(s) found. (0) Not found: Key newslm/embeddings/LayerNorm_embed_norm/beta not found in checkpoint [[node save/RestoreV2 (defined at /usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/ops.py:1748) ]] [[save/RestoreV2/_301]] (1) Not found: Key newslm/embeddings/LayerNorm_embed_norm/beta not found in checkpoint [[node save/RestoreV2 (defined at /usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/ops.py:1748) ]] 0 successful operations. 0 derived errors ignored.
Original stack trace for 'save/RestoreV2': File "scripts/demo.py", line 170, in
saver = tf.train.Saver()
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/training/saver.py", line 828, in init
self.build()
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/training/saver.py", line 840, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/training/saver.py", line 878, in _build
build_restore=build_restore)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/training/saver.py", line 508, in _build_internal
restore_sequentially, reshape)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/training/saver.py", line 328, in _AddRestoreOps
restore_sequentially)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/training/saver.py", line 575, in bulk_restore
return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/ops/gen_io_ops.py", line 1696, in restore_v2
name=name)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
op_def=op_def)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
attrs, op_def, compute_device)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
op_def=op_def)
File "/usr/local/lib/python3.7/dist-packages/tensorflow_core/python/framework/ops.py", line 1748, in init
self._traceback = tf_stack.extract_stack()
I don't why some keys "Key newslm/embeddings/LayerNorm_embed_norm/beta" are not found in the checkpoint. Any help will be appreciated. Thanks.