get_constant_schedule() got an unexpected keyword argument 'warmup_steps'

[Kepler-Br]

Tried to finetune your GPT2 Russian on russian dataset, but got this problem:

Traceback (most recent call last):
  File "run_lm_finetuning.py", line 662, in <module>
    main()
  File "run_lm_finetuning.py", line 630, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "run_lm_finetuning.py", line 289, in train
    scheduler = get_constant_schedule(optimizer, warmup_steps=warmup_steps)
TypeError: get_constant_schedule() got an unexpected keyword argument 'warmup_steps'

Here's how I run run_lm_finetuning.py:

python run_lm_finetuning.py \
    --output_dir=$MODEL_PATH \
    --model_type=gpt2 \
    --model_name_or_path=$MODEL_PATH \
    --do_train \
    --train_data_file=$TRAIN_FILE \
    --per_gpu_train_batch_size=$BATCH_SIZE \
    --save_steps=10000 \
    --logging_steps=1 \
    --warmup_samples 16000 \
    --learning_rate $LEARNING_RATE \
    --tokenizer_class YTEncoder \
    --tokenizer_name bpe/yt.model \
    --do_eval \
    --evaluate_during_training \
    --eval_steps 1000 \
    --eval_data_file=$VALIDATION_FILE \
    --num_train_epochs 1.0 \
    --unfreeze_level 0 \
    --overwrite_output_dir 

But after I deleted warmup_steps from function call, training starts successfully but somewhat it cancels it self during first iteration:

Epoch:   0% 0/3 [00:00<?, ?it/s]
Iteration:   0% 0/9928 [00:00<?, ?it/s]^C

I'm using google colab for training Even replacing get_constant_schedule with get_constant_schedule_with_warmup doesn't help: training still cancels it self with ^C. I tried different pip transformers versions, but nothing works. Sampling works flawlessly btw. This is what happens when I try to use TPU on colab:

Traceback (most recent call last):
  File "tpu_lm_finetuning.py", line 697, in <module>
    xmp.spawn(main)
  File "/usr/local/lib/python3.6/dist-packages/torch_xla/distributed/xla_multiprocessing.py", line 182, in spawn
    start_method=start_method)
  File "/usr/local/lib/python3.6/dist-packages/torch/multiprocessing/spawn.py", line 158, in start_processes
    while not context.join():
  File "/usr/local/lib/python3.6/dist-packages/torch/multiprocessing/spawn.py", line 108, in join
    (error_index, name)
Exception: process 2 terminated with signal SIGKILL

Here's my pip packages:

absl-py                       0.9.0          
alabaster                     0.7.12         
albumentations                0.1.12         
altair                        4.1.0          
apex                          0.1            
argon2-cffi                   20.1.0         
asgiref                       3.2.10         
astor                         0.8.1          
astropy                       4.0.1.post1    
astunparse                    1.6.3          
atari-py                      0.2.6          
atomicwrites                  1.4.0          
attrs                         19.3.0         
audioread                     2.1.8          
autograd                      1.3            
Babel                         2.8.0          
backcall                      0.2.0          
backports.tempfile            1.0            
backports.weakref             1.0.post1      
beautifulsoup4                4.6.3          
bleach                        3.1.5          
blis                          0.4.1          
bokeh                         2.1.1          
boto                          2.49.0         
boto3                         1.14.37        
botocore                      1.17.37        
Bottleneck                    1.3.2          
branca                        0.4.1          
bs4                           0.0.1          
bz2file                       0.98           
CacheControl                  0.12.6         
cachetools                    4.1.1          
catalogue                     1.0.0          
certifi                       2020.6.20      
cffi                          1.14.1         
chainer                       7.4.0          
chardet                       3.0.4          
click                         7.1.2          
cloudpickle                   1.3.0          
cmake                         3.12.0         
cmdstanpy                     0.4.0          
colorlover                    0.3.0          
community                     1.0.0b1        
contextlib2                   0.5.5          
convertdate                   2.2.1          
coverage                      3.7.1          
coveralls                     0.5            
crcmod                        1.7            
cufflinks                     0.17.3         
cupy-cuda101                  7.4.0          
cvxopt                        1.2.5          
cvxpy                         1.0.31         
cycler                        0.10.0         
cymem                         2.0.3          
Cython                        0.29.21        
daft                          0.0.4          
dask                          2.12.0         
dataclasses                   0.7            
datascience                   0.10.6         
decorator                     4.4.2          
defusedxml                    0.6.0          
descartes                     1.1.0          
dill                          0.3.2          
distributed                   1.25.3         
Django                        3.1            
dlib                          19.18.0        
dm-sonnet                     1.35           
dm-tree                       0.1.5          
docopt                        0.6.2          
docutils                      0.15.2         
dopamine-rl                   1.0.5          
earthengine-api               0.1.229        
easydict                      1.9            
ecos                          2.0.7.post1    
editdistance                  0.5.3          
en-core-web-sm                2.2.5          
entrypoints                   0.3            
ephem                         3.7.7.1        
et-xmlfile                    1.0.1          
fa2                           0.3.5          
fancyimpute                   0.4.3          
fastai                        1.0.59         
fastapi                       0.61.0         
fastdtw                       0.3.4          
fastprogress                  0.2.5          
fastrlock                     0.5            
fbprophet                     0.6            
feather-format                0.4.1          
featuretools                  0.4.1          
filelock                      3.0.12         
firebase-admin                4.1.0          
fix-yahoo-finance             0.0.22         
Flask                         1.1.2          
folium                        0.8.3          
fsspec                        0.8.0          
future                        0.16.0         
gast                          0.3.3          
GDAL                          2.2.2          
gdown                         3.6.4          
gensim                        3.6.0          
geographiclib                 1.50           
geopy                         1.17.0         
gevent                        1.4.0          
gin-config                    0.3.0          
glob2                         0.7            
google                        2.0.3          
google-api-core               1.16.0         
google-api-python-client      1.7.12         
google-auth                   1.17.2         
google-auth-httplib2          0.0.4          
google-auth-oauthlib          0.4.1          
google-cloud-bigquery         1.21.0         
google-cloud-core             1.0.3          
google-cloud-datastore        1.8.0          
google-cloud-firestore        1.7.0          
google-cloud-language         1.2.0          
google-cloud-storage          1.18.1         
google-cloud-translate        1.5.0          
google-colab                  1.0.0          
google-pasta                  0.2.0          
google-resumable-media        0.4.1          
googleapis-common-protos      1.52.0         
googledrivedownloader         0.4            
graph-nets                    1.0.5          
graphviz                      0.10.1         
greenlet                      0.4.15         
grpcio                        1.31.0         
gspread                       3.0.1          
gspread-dataframe             3.0.7          
gunicorn                      20.0.4         
gym                           0.17.2         
h5py                          2.10.0         
HeapDict                      1.0.1          
holidays                      0.9.12         
holoviews                     1.13.3         
html5lib                      1.0.1          
httpimport                    0.5.18         
httplib2                      0.17.4         
httplib2shim                  0.0.3          
humanize                      0.5.1          
hyperopt                      0.1.2          
ideep4py                      2.0.0.post3    
idna                          2.10           
image                         1.5.32         
imageio                       2.4.1          
imagesize                     1.2.0          
imbalanced-learn              0.4.3          
imblearn                      0.0            
imgaug                        0.2.9          
importlib-metadata            1.7.0          
imutils                       0.5.3          
inflect                       2.1.0          
iniconfig                     1.0.1          
intel-openmp                  2020.0.133     
intervaltree                  2.1.0          
ipykernel                     4.10.1         
ipython                       5.5.0          
ipython-genutils              0.2.0          
ipython-sql                   0.3.9          
ipywidgets                    7.5.1          
itsdangerous                  1.1.0          
jax                           0.1.75         
jaxlib                        0.1.52         
jdcal                         1.4.1          
jedi                          0.17.2         
jieba                         0.42.1         
Jinja2                        2.11.2         
jmespath                      0.10.0         
joblib                        0.16.0         
jpeg4py                       0.1.4          
jsonschema                    2.6.0          
jupyter                       1.0.0          
jupyter-client                5.3.5          
jupyter-console               5.2.0          
jupyter-core                  4.6.3          
kaggle                        1.5.6          
kapre                         0.1.3.1        
Keras                         2.3.1          
Keras-Applications            1.0.8          
Keras-Preprocessing           1.1.2          
keras-vis                     0.4.1          
kfac                          0.2.0          
kiwisolver                    1.2.0          
knnimpute                     0.1.0          
librosa                       0.6.3          
lightgbm                      2.2.3          
llvmlite                      0.31.0         
lmdb                          0.98           
lucid                         0.3.8          
LunarCalendar                 0.0.9          
lxml                          4.2.6          
magenta                       0.3.19         
Markdown                      3.2.2          
MarkupSafe                    1.1.1          
matplotlib                    3.2.2          
matplotlib-venn               0.11.5         
mesh-tensorflow               0.1.12         
mido                          1.2.6          
mir-eval                      0.5            
missingno                     0.4.2          
mistune                       0.8.4          
mizani                        0.6.0          
mkl                           2019.0         
mlxtend                       0.14.0         
more-itertools                8.4.0          
moviepy                       0.2.3.5        
mpi4py                        3.0.3          
mpmath                        1.1.0          
msgpack                       1.0.0          
multiprocess                  0.70.10        
multitasking                  0.0.9          
murmurhash                    1.0.2          
music21                       5.5.0          
natsort                       5.5.0          
nbconvert                     5.6.1          
nbformat                      5.0.7          
networkx                      2.4            
nibabel                       3.0.2          
nltk                          3.2.5          
notebook                      5.3.1          
np-utils                      0.5.12.1       
numba                         0.48.0         
numexpr                       2.7.1          
numpy                         1.18.5         
nvidia-ml-py3                 7.352.0        
oauth2client                  4.1.3          
oauthlib                      3.1.0          
okgrade                       0.4.3          
opencv-contrib-python         4.1.2.30       
opencv-python                 4.1.2.30       
openpyxl                      2.5.9          
opt-einsum                    3.3.0          
osqp                          0.6.1          
packaging                     20.4           
palettable                    3.3.0          
pandas                        1.0.5          
pandas-datareader             0.8.1          
pandas-gbq                    0.11.0         
pandas-profiling              1.4.1          
pandocfilters                 1.4.2          
panel                         0.9.7          
param                         1.9.3          
parso                         0.7.1          
pathlib                       1.0.1          
patsy                         0.5.1          
pbr                           5.4.5          
pexpect                       4.8.0          
pickleshare                   0.7.5          
Pillow                        7.0.0          
pip                           19.3.1         
pip-tools                     4.5.1          
plac                          1.1.3          
plotly                        4.4.1          
plotnine                      0.6.0          
pluggy                        0.7.1          
portpicker                    1.3.1          
prefetch-generator            1.0.1          
preshed                       3.0.2          
pretty-midi                   0.2.8          
prettytable                   0.7.2          
progressbar2                  3.38.0         
prometheus-client             0.8.0          
promise                       2.3            
prompt-toolkit                1.0.18         
protobuf                      3.12.4         
psutil                        5.4.8          
psycopg2                      2.7.6.1        
ptyprocess                    0.6.0          
py                            1.9.0          
pyarrow                       0.14.1         
pyasn1                        0.4.8          
pyasn1-modules                0.2.8          
pycocotools                   2.0.1          
pycparser                     2.20           
pyct                          0.4.6          
pydantic                      1.6.1          
pydata-google-auth            1.1.0          
pydot                         1.3.0          
pydot-ng                      2.0.0          
pydotplus                     2.0.2          
PyDrive                       1.3.1          
pyemd                         0.5.1          
pyglet                        1.5.0          
Pygments                      2.1.3          
pygobject                     3.26.1         
pymc3                         3.7            
PyMeeus                       0.3.7          
pymongo                       3.11.0         
pymystem3                     0.2.0          
PyOpenGL                      3.1.5          
pyparsing                     2.4.7          
pypng                         0.0.20         
pyrsistent                    0.16.0         
pysndfile                     1.3.8          
PySocks                       1.7.1          
pystan                        2.19.1.1       
pytest                        3.6.4          
python-apt                    1.6.5+ubuntu0.3
python-chess                  0.23.11        
python-dateutil               2.8.1          
python-louvain                0.14           
python-rtmidi                 1.4.0          
python-slugify                4.0.1          
python-utils                  2.4.0          
pytz                          2018.9         
pyviz-comms                   0.7.6          
PyWavelets                    1.1.1          
PyYAML                        3.13           
pyzmq                         19.0.2         
qtconsole                     4.7.5          
QtPy                          1.9.0          
regex                         2019.12.20     
requests                      2.23.0         
requests-oauthlib             1.3.0          
resampy                       0.2.2          
retrying                      1.3.3          
rpy2                          3.2.7          
rsa                           4.6            
s3fs                          0.4.2          
s3transfer                    0.3.3          
sacremoses                    0.0.43         
scikit-image                  0.16.2         
scikit-learn                  0.22.2.post1   
scipy                         1.4.1          
screen-resolution-extra       0.0.0          
scs                           2.1.2          
seaborn                       0.10.1         
semantic-version              2.8.4          
Send2Trash                    1.5.0          
sentencepiece                 0.1.91         
setuptools                    49.2.0         
setuptools-git                1.2            
Shapely                       1.7.0          
simplegeneric                 0.8.1          
six                           1.15.0         
sklearn                       0.0            
sklearn-pandas                1.8.0          
smart-open                    2.1.0          
snowballstemmer               2.0.0          
sortedcontainers              2.2.2          
spacy                         2.2.4          
Sphinx                        1.8.5          
sphinxcontrib-serializinghtml 1.1.4          
sphinxcontrib-websupport      1.2.3          
SQLAlchemy                    1.3.18         
sqlparse                      0.3.1          
srsly                         1.0.2          
stable-baselines              2.2.1          
starlette                     0.13.6         
statsmodels                   0.10.2         
sympy                         1.1.1          
tables                        3.4.4          
tabulate                      0.8.7          
tblib                         1.7.0          
tendo                         0.2.15         
tensor2tensor                 1.14.1         
tensorboard                   1.15.0         
tensorboard-plugin-wit        1.7.0          
tensorboardcolab              0.0.22         
tensorflow                    1.15.2         
tensorflow-addons             0.8.3          
tensorflow-datasets           2.1.0          
tensorflow-estimator          1.15.1         
tensorflow-gan                2.0.0          
tensorflow-gcs-config         2.3.0          
tensorflow-hub                0.8.0          
tensorflow-metadata           0.22.2         
tensorflow-privacy            0.2.2          
tensorflow-probability        0.7.0          
termcolor                     1.1.0          
terminado                     0.8.3          
testpath                      0.4.4          
text-unidecode                1.3            
textblob                      0.15.3         
textgenrnn                    1.4.1          
tflearn                       0.3.2          
Theano                        1.0.5          
thinc                         7.4.0          
tifffile                      2020.7.24      
toml                          0.10.1         
toolz                         0.10.0         
torch                         1.6.0+cu101    
torchsummary                  1.5.1          
torchtext                     0.3.1          
torchvision                   0.7.0+cu101    
tornado                       5.1.1          
tqdm                          4.41.1         
traitlets                     4.3.3          
transformers                  2.2.0          
tweepy                        3.6.0          
typeguard                     2.7.1          
typing-extensions             3.7.4.2        
tzlocal                       1.5.1          
umap-learn                    0.4.6          
uritemplate                   3.0.1          
urllib3                       1.24.3         
vega-datasets                 0.8.0          
wasabi                        0.7.1          
wcwidth                       0.2.5          
webencodings                  0.5.1          
Werkzeug                      1.0.1          
wheel                         0.34.2         
widgetsnbextension            3.5.1          
wordcloud                     1.5.0          
wrapt                         1.12.1         
xarray                        0.15.1         
xgboost                       0.90           
xkit                          0.0.0          
xlrd                          1.1.0          
xlwt                          1.3.0          
yellowbrick                   0.9.1          
youtokentome                  1.0.6          
zict                          2.0.0          
zipp                          3.1.0          
zmq                           0.0.0

[Kepler-Br]

Here's my google colab notebook: https://colab.research.google.com/drive/1EYuZXEqouYQa8VPK6jgOgdsuKuaD5Ub8?usp=sharing

[Kepler-Br]

Well, problem solved it self: I was running gpt-2 on cpu. When memory consumption exceeds actual memory, colab will kill cell without saying that. But I need help with notebook anyway.