error when using create_tfrecords.py

[CrackerHax]

Got 142 files, divided into 1 chunks.
  0% 0/1 [00:00<?, ?it/s]0

Traceback (most recent call last):
  File "./GPT2/datasets/openwebtext/create_tfrecords.py", line 86, in <module>
    good += g
TypeError: unsupported operand type(s) for +: 'int' and 'NoneType'

[ConnorJL]

Hmm, it'd be helpful if you to tell me a bit more about what you were doing, how the files you were trying to convert look etc.

Off the top of my head, I'd say try changing the processes value at the top of the file to 1 and files_per to 10. Also double check your output dir is somewhere you can write to.

[CrackerHax]

Nope, still the same problem. I am trying to turn tv show scripts into tfrecords. The first time I tried it it worked (on 2 files) then it just never worked after that.

Also forgot to mention I am doing this in google colab with runtime set for python3 and TPU.

Here are my settings:

base_dir = "/content/gdrive/My Drive/gpt2/text2" # Path to where your .txt files are located
files_per = 175000 # 175000 ~ 200-300MB
name = "tv-scripts" # Name of output files will be name_i.tfrecords where i is the number of the file
output_dir = "/content/gdrive/My Drive/gpt2/textout"
log_dir = "logs"
files = glob.glob(os.path.join(base_dir, "*.txt"))
processes = 64 # Number of encoding processes to run
encoder_path = "/content/gdrive/My Drive/gpt2/encoder" # Path to encoder files
minimum_size = 25

had to change the 'files =' line because "*/.txt" was not finding the files.

directories look writable:

$ls -lsa
total 28
4 drwx------ 2 root root 4096 Dec  8 06:02 encoder
4 drwx------ 2 root root 4096 Dec  8 06:13 GPT2
4 drwx------ 2 root root 4096 Dec  8 00:27 .ipynb_checkpoints
4 drwx------ 2 root root 4096 Dec  8 00:37 logs
4 drwx------ 2 root root 4096 Dec  8 00:27 text
4 drwx------ 2 root root 4096 Dec  8 00:28 textout

[CrackerHax]

Ok I see the problem... It was doing that because it couldnt overwrite the logs from the previous try. Deleting the logs caused it to work.