Closed TatProg closed 3 years ago
Hello! Do you mind sharing a reproducible code example in colab? I ran the following on the bare run_mlm.py
script:
python run_mlm.py --output_dir=here --report_to=tensorboard --dataset_name=wikitext --dataset_config_name=wikitext-2-raw-v1 --model_name_or_path=bert-base-cased --do_train
which works without issue. Maybe @sgugger has more insights.
Nice username and Pycharm project :)
@LysandreJik
I found problem: I used report_to=
in TrainingArguments
without run_name=
Even if I add --run_name=
in command python3 run_mlm.py --report_to wandb --run_name new_run_name
it didn't work. I think this might be fixed quite easy: Throw exception, if report_to=
is 'initialized' in TrainingArguments
without run_name=
My solution. Also tested not only with 'wandb', but with 'tensorboard' and both of them works very well. (All imports are above)
training_args = TrainingArguments(
output_dir='My_lovely_mlm_model',
overwrite_output_dir=True,
do_train=True,
do_eval=True,
per_device_train_batch_size=100,
per_device_eval_batch_size=50,
evaluation_strategy='steps',
logging_steps=10_000,
eval_steps=None,
prediction_loss_only=True,
num_train_epochs=50,
save_steps=10_000,
save_total_limit=10,
report_to='wandb'
run_name="new_run" #ADDED THIS.
)
P.S. If I want to continue training my model from checkpoint, i should change model = AutoModelForMaskedLM.from_pretrained('bert-base-multilingual-cased')
to model = AutoModelForMaskedLM.from_pretrained('project/pretrained_model_name/checkpoint-190000')
right?
If you use the Trainer, just use resume_from_checkpoint=path_to_checkpoint
when calling trainer.train
.
Environment info
transformers
version: 4.6.1Who can help
@sgugger @LysandreJik
Information
Model I am using BERT - 'bert-base-multilingual-cased'
The problem arises when using:
The tasks I am working on is:
To reproduce
Steps to reproduce the behavior:
pip install tf-nightly pip install tensorboard
Run script with button in PyCharm or in console with python
run_mlm.py --report_to wandb --run_name new_run_name
Enjoy error message
Also, i tried to remove 'tensorboard' or 'wandb', but caught same error again and again
Expected behavior
Script must run without this error. Wandb folder must be created.