search

openai / gpt-2

Code for the paper "Language Models are Unsupervised Multitask Learners"
https://openai.com/blog/better-language-models/
Other
22.32k stars 5.5k forks source link

Installation Guide? #100

Open desmo360 opened 5 years ago

desmo360 commented 5 years ago

Forgive me, I'm very new to coding. Here's what I've done so far.

From here, I really don't know what to do to use the model. Is there anyone willing to lend a hand? Much appreciated.

ak9250 commented 5 years ago

Hi, I made a colab notebook to retrain and run the model as you will need a gpu to get fast performance which is free on colab, check it out here and let me know if you need help. https://github.com/ak9250/gpt-2-colab

desmo360 commented 5 years ago

Hi, I made a colab notebook to retrain and run the model as you will need a gpu to get fast performance which is free on colab, check it out here and let me know if you need help. https://github.com/ak9250/gpt-2-colab

I've started learning Python so I can run this. I know less about Google Colab. How do I run your thing?

ak9250 commented 5 years ago

it runs top to bottom, you can click on the play button of each cell to run it, save a copy to drive before running

desmo360 commented 5 years ago

I copied the code into a Python 3 Notebook in Colab and clicked run, but I got an error on line 9: name 'true' is not defined. I'm pretty sure I'm doing something wrong.

ak9250 commented 5 years ago

did you click copy to drive before running? also this is line 9 correct !PYTHONPATH=src ./train.py --dataset /content/gpt-2/98-0.txt?

desmo360 commented 5 years ago

I have done this and couldn't get it to work. This is my line 9: "include_colab_link": true. Could you walk me through the steps of what to do after cloning the repository?

ak9250 commented 5 years ago

you dont need to clone the repo, colab runs in the cloud click on open in colab in the notebook which takes you to this link https://colab.research.google.com/github/ak9250/gpt-2-colab/blob/master/GPT_2.ipynb, then save a copy in drive

desmo360 commented 5 years ago

Ok. That makes more sense. I have it training now. Can I close the window and will it continue to train?

ak9250 commented 5 years ago

it will run for 12 hours so you will have to stop it before then and save your trained model. Also from https://stackoverflow.com/questions/49062628/does-google-colab-stay-connected-when-i-close-my-browser "The current cell will continue executing once you close your browser, but the outputs will not end up in the notebook in Drive. (If you were to reconnect before your VM were considered idle and deleted, any values you'd computed would still be available in your kernel.)"

mradhub commented 5 years ago

@desmo360 It doesn't seem like train.py replaces the initial language set that gpt-2 was trained with. Does it add additional data to the original set to generate a new model?

desmo360 commented 5 years ago

I have been training with the Colab that ak9250 set up. In one notebook, I gave it the web serial Token. In my other notebook, I trained it on Token, A Tale of Two Cities, and a few books from the Bible. It struggles with the bible language, but it was successfully rearranging the other two works with mixed results. So yes?

walter-grace commented 5 years ago

Hey this is really great! Especially the colab, I just had a question if I want to train it off my own model not <!wget https://www.gutenberg.org/files/98/98-0.txt> say I wanted to use my own twitter data or Facebook would I put my own model where the gutenberg.txt was? Fairly novice, so i'd thought i'd live a comment here.

ak9250 commented 5 years ago

you would need to put your data in something like pastebin and then get the raw text from there

walter-grace commented 5 years ago

any recommendations for doing so? How would one go about doing that?

ak9250 commented 5 years ago

@walter-grace you can do something like curl pastebin url here >> paste.txt then put the paste.txt file in place of 98-0.txt

desmo360 commented 5 years ago

Here's a step-by-step, since this took me longer than it should have to figure out:

  1. Open the colab with the model
  2. Click the black arrow at the top left of the screen
  3. Click Files
  4. Click on the file named '..'
  5. Open 'Content'
  6. Right click on 'gpt-2' and click upload
  7. Upload the .txt file you wish to train
  8. Edit the code below 'lets get our text to train on...' to run your text file instead of 98-0.txt
  9. Run all of the code up to that point
Astraithious commented 5 years ago

getting 0 tokens now on this, is there something wrong?