search

rkfg / gpt-2

Code for the paper "Language Models are Unsupervised Multitask Learners"
MIT License
20 stars 7 forks source link

Protobuf::FatalException #9

Closed zendevil closed 5 years ago

zendevil commented 5 years ago 
python3 src/interactive_conditional_samples.py
PATH models/345M/hparams.json
2019-07-08 11:06:36.791192: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-07-08 11:06:36.797641: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2300000000 Hz
2019-07-08 11:06:36.798794: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x561e35855640 executing computations on platform Host. Devices:
2019-07-08 11:06:36.798849: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>
WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
PATH models/345M/checkpoint/run1
WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
The model has 86505216 parameters
Model prompt >>> phones
[libprotobuf FATAL /sentencepiece/src/../third_party/protobuf-lite/google/protobuf/repeated_field.h:1506] CHECK failed: (index) < (current_size_):
terminate called after throwing an instance of 'google::protobuf::FatalException'
  what():  CHECK failed: (index) < (current_size_):
Aborted
rkfg commented 5 years ago

That's something internal to sentencepiece, not my code. Check if the files are correct, especially that the dictionary and hparams.json match. Also your model may be corrupted since you're already missing one file.

zendevil commented 5 years ago

Yeah hparams.json was incorrect. I had specified the wrong number of tokens. I reran encode.sh and everything works fine.