search

minimaxir / gpt-2-simple

Python package to easily retrain OpenAI's GPT-2 text-generating model on new texts
Other
3.4k stars 675 forks source link

finetune json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) #145

Open thiagorova opened 4 years ago

thiagorova commented 4 years ago

Hi! first of all, thanks a lot! Also, this is probably related to #97 and #96.

I'm running this from the pip installed version rather than straight from code. My computer is a MacOS Sierra.

As you can see below, I'm running finetune on a custom file called WPT.npz, generated here: nshepperd.

It's supposed to be a simple test where I added some Wikipedia files from the Portuguese dump to the original 117M gpt-2 training model.

I'm guessing that I'm doing something wrong, I just can't discover what. Any help would really be appreciated!

output (clean): 

$ gpt_2_simple finetune models/WPT.npz --sample_every 500 --model_name WPT10 --steps 10000

np_resource = np.dtype([("resource", np.ubyte, 1)])
Fetching checkpoint: 1.05Mit [00:00, 347Mit/s]                                                      
Fetching encoder.json: 1.05Mit [00:00, 433Mit/s]                                                    
Fetching hparams.json: 1.05Mit [00:00, 451Mit/s]                                                    
Fetching model.ckpt.data-00000-of-00001: 1.05Mit [00:00, 454Mit/s]                                  
Fetching model.ckpt.index: 1.05Mit [00:00, 469Mit/s]                                                
Fetching model.ckpt.meta: 1.05Mit [00:00, 395Mit/s]                                                 
Fetching vocab.bpe: 1.05Mit [00:00, 405Mit/s]                                                       

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/bin/gpt_2_simple", line 8, in <module>
    sys.exit(cmd())
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/gpt_2_simple/gpt_2.py", line 768, in cmd
    multi_gpu=args.multi_gpu)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/gpt_2_simple/gpt_2.py", line 798, in cmd_finetune
    multi_gpu=multi_gpu)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/gpt_2_simple/gpt_2.py", line 178, in finetune
    enc = encoder.get_encoder(checkpoint_path)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/gpt_2_simple/src/encoder.py", line 110, in get_encoder
    encoder = json.load(f)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/__init__.py", line 299, in load
    parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

output (with some TensorFlow FutureWarnings. I doubt this has any use, but...)

$ gpt_2_simple finetune models/WPT.npz --sample_every 500 --model_name WPT10 --steps 10000
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
Fetching checkpoint: 1.05Mit [00:00, 347Mit/s]                                                      
Fetching encoder.json: 1.05Mit [00:00, 433Mit/s]                                                    
Fetching hparams.json: 1.05Mit [00:00, 451Mit/s]                                                    
Fetching model.ckpt.data-00000-of-00001: 1.05Mit [00:00, 454Mit/s]                                  
Fetching model.ckpt.index: 1.05Mit [00:00, 469Mit/s]                                                
Fetching model.ckpt.meta: 1.05Mit [00:00, 395Mit/s]                                                 
Fetching vocab.bpe: 1.05Mit [00:00, 405Mit/s]                                                       
2019-12-04 17:56:05.510008: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/bin/gpt_2_simple", line 8, in <module>
    sys.exit(cmd())
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/gpt_2_simple/gpt_2.py", line 768, in cmd
    multi_gpu=args.multi_gpu)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/gpt_2_simple/gpt_2.py", line 798, in cmd_finetune
    multi_gpu=multi_gpu)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/gpt_2_simple/gpt_2.py", line 178, in finetune
    enc = encoder.get_encoder(checkpoint_path)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/gpt_2_simple/src/encoder.py", line 110, in get_encoder
    encoder = json.load(f)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/__init__.py", line 299, in load
    parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
rockinthespecs commented 4 years ago

Having same issue is there a resolution

Viole-Grace commented 4 years ago

A Finetuning session for the same model does this.

A simple workaround is to add

import tensorflow as tf

tf.reset_default_graph()

before your gpt2.finetune(..)

sample code:

def generate(input_file_name, nsamples=5000, model="124M"):

    import tensorflow as tf
    tf.reset_default_graph()

    if not os.path.isdir(os.path.join("models","124M")):
        print("Downloading the model : 124M pretrained, 500M final tuning")
        gpt2.download_gpt2(model_name="124M")

    filename=input_file_name
    sess=gpt2.start_tf_sess()
    gpt2.finetune(sess, dataset=filename, model_name=model, steps=2000, print_every=10, sample_every=49, save_every=100)
vanh17 commented 4 years ago

Hi I try this and the error still persist. Is there anyway resolution? Thank you

songproducer commented 4 years ago

Also having this issue and also: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA

goshawk22 commented 4 years ago

This is because Tensorflow compile their binaries to support as many CPUs as possible. Not every CPU has AVX2, so Tensorflow isn't compiled to use it. It won't effect any of your code, but will effect performance. To compile from source, you can follow instructions on their website.

songproducer commented 4 years ago

I compiled tensorflow-1.15.0 but still getting the error.

goshawk22 commented 4 years ago

When compiling, you should be asked if you want to build with avx support. Enter the correct flags. E.g. /arch:AVX2. Then build the pip package and install.

jan25 commented 3 years ago

Seems like the google storage objects used in this repo are not publicly available anymore. I worked around this by running a separate script here https://github.com/nshepperd/gpt-2/blob/finetuning/download_model.py#L19 which downloads the model correctly.

ghost commented 3 years ago

No dice.. use this instead 

!git clone https://github.com/ilopezfr/gpt-2/
import os
os.chdir('gpt-2')

#Download model weights with 
# !python download_model.py 117M
# !python download_model.py 345M
#!python download_model.py 774M