search

minimaxir / gpt-2-simple

Python package to easily retrain OpenAI's GPT-2 text-generating model on new texts
Other
3.4k stars 675 forks source link

Support TF2.0 #150

Open arlyon opened 4 years ago

arlyon commented 4 years ago

A little tracking issue to discuss what would be needed to migrate to TF2.0:

Currently, the only thing I can see from poking around is the dependency on the graph editor contrib, which was removed (along with the rest of the contribs). Running the 1.x -> 2.0 converter ran smoothly so the package is importable at least (without the contrib).

There are 16 usages, limited to gpt_2_simple/src/memory_saving_gradients.py. There are libraries (CODAIT/graph_def_editor) that could be of use.

danielshoun commented 4 years ago

I managed to get it working on TF2.0 yesterday using the package graph_def_editor found here: https://github.com/CODAIT/graph_def_editor

Replace the tensorflow.contrib.graph_editor import with an import of graph_def_editor, remove all instances of ".value" (no longer necessary in TF2.0), and it will work.

I'm new to using git so I don't know how to suggest this or make a pull request or whatever.

tripzero commented 4 years ago

@danielshoun you should definitely try to make a pull request. There's probably lots of tutorials out there on how to do it. Great learning experience!

nabilmbr commented 4 years ago

@danielshoun can you please share the code with us ?

danielshoun commented 4 years ago

@nabilmbr It's kind of difficult to share the entire files but here's some directions (that worked at least as of when I made that comment in December):

Download graph_def_editor into your env.

In gpt-2-simple/gpt_2_simple/src/memory_saving_gradients.py change this: import tensorflow.contrib.graph_editor as ge to: import graph_def_editor as ge (or whatever import statement is necessary to import graph_def_editor if you didn't put it in the default package location)

Then in gpt_2_simple/src/model.py remove ".value" from line 124, and in gpt_2_simple/src/memory_saving_gradients.py remove ".value" from line 100. Keep everything else on these lines exactly the same, you just no longer need to access a value property because that object type is now the value itself.

tiagovaz commented 3 years ago

Hi there, I've added a little to what @danielshoun proposed and had gpt-2-simple 0.7.2 working with tensorflow 2.6.0 using GPU. I see there is already a PR, so probably there's no point of adding something there... but here is my diff anyway:

gpt-2-simple-0.7.2_tensorflow-2.6.0.txt.

Ayammohsen commented 2 years ago

I work on abuntu 20.04 and I made migration on python program from TF1 to TF2 , this program contains two procedures to run run in CPU successfully (first procedure) but when I want to run with GPU ( second procedure) ,I need to access the file at https://github.com/cybertronai/gradient-checkpointing , program stop after reading the parameters and not training. I ask if can any one help me to solve this problem?