search

EleutherAI / gpt-neo

An implementation of model parallel GPT-2 and GPT-3-style models using the mesh-tensorflow library.
https://www.eleuther.ai
MIT License
8.21k stars 945 forks source link

Tokenizing error when training on Colab #233

Closed Marcus-Arcadius closed 2 years ago

Marcus-Arcadius commented 3 years ago

image

bterrific2008 commented 3 years ago

You seem to have a double slash in your path, and you seem to be referencing your gdrive from the colab directory.

Are you mounting your Google Drive correctly with your colab notebook? Here's a stackoverflow answer about the subject.

Marcus-Arcadius commented 3 years ago

You seem to have a double slash in your path, and you seem to be referencing your gdrive from the colab directory.

Are you mounting your Google Drive correctly with your colab notebook? Here's a stackoverflow answer about the subject.

Sorry for the late reply, I honestly thought no one would answer, yes I am mounting my drive to the notebook!

I will give this a try tomorrow morning and hopefully see if it works 😁

Marcus-Arcadius commented 3 years ago

@bterrific2008 I managed to get it to finally work, but now I am experiencing these problems? image

bterrific2008 commented 3 years ago

The error states FileNotFoundError for the configs/colab_XL.json file. Can you check if your working directory is in the gpt-neo repository, and if there exists a configs/colab_XL.json file in your copy of the gpt-neo repo?

Marcus-Arcadius commented 3 years ago

The error states FileNotFoundError for the configs/colab_XL.json file. Can you check if your working directory is in the gpt-neo repository, and if there exists a configs/colab_XL.json file in your copy of the gpt-neo repo?

I cannot find any file in my copy image

StellaAthena commented 3 years ago

The error states FileNotFoundError for the configs/colab_XL.json file. Can you check if your working directory is in the gpt-neo repository, and if there exists a configs/colab_XL.json file in your copy of the gpt-neo repo?

I cannot find any file in my copy image

Try loading configs/gpt3_XL_256_Pile.json instead.

bterrific2008 commented 3 years ago

From my understanding of the notebook, whatever file you write to in the Set Model Configs is the configuration file you should use for the Training from Scratch step.

The help text for the --model parameter of main.py reads as: JSON file that contains model parameters, so you should make sure whatever is contained there matches the config file you want to use.

Marcus-Arcadius commented 3 years ago

From my understanding of the notebook, whatever file you write to in the Set Model Configs is the configuration file you should use for the Training from Scratch step.

The help text for the --model parameter of main.py reads as: JSON file that contains model parameters, so you should make sure whatever is contained there matches the config file you want to use.

As I could not get it to work, I ended up using the pre-trained model for fine tuning - it worked. If in the future, I want to further train the fine-tuned model, what is the process?