NielsRogge
commented
3 years ago I've assigned Patrick, but looking at the docs of Wav2Vec2, is says:
Wav2Vec2 models that have set config.feat_extract_norm == "group", such as wav2vec2-base, have not been trained using attention_mask. For such models, input_values should simply be padded with 0 and no attention_mask should be passed.
For Wav2Vec2 models that have set config.feat_extract_norm == "layer", such as wav2vec2-lv60, attention_mask should be passed for batched inference.
It seems like the pre-training script currently only supports models that are pre-trained using an attention mask, such as patrickvonplaten/wav2vec2-base-libri-100h.
LifaSun
Nian-Chen
patrickvonplaten
padmalcom
Environment info
transformersversion: 4.9.1Who can help
Models: @patrickvonplaten
Information
Model I am using Wav2vec Pretrain:
The problem arises when using: https://github.com/huggingface/transformers/blob/master/examples/research_projects/wav2vec2/run_pretrain.py
The tasks I am working on is:
To reproduce
Steps to reproduce the behavior:
python run_pretrain.py --output_dir="./wav2vec2-base" \ --num_train_epochs="3" \ --per_device_train_batch_size="32" \ --per_device_eval_batch_size="32" \ --gradient_accumulation_steps="2" \ --save_total_limit="3" \ --save_steps="500" \ --logging_steps="10" \ --learning_rate="5e-4" \ --weight_decay="0.01" \ --warmup_steps="3000" \ --model_name_or_path="facebook/wav2vec2-base" \ --dataset_name="timit_asr" \ --train_split_name="train" \ --preprocessing_num_workers="4" \ --max_duration_in_seconds="10.0" \ --group_by_length \ --verbose_logging \
Expected behavior
Running training Num examples = 185 Num Epochs = 3 Instantaneous batch size per device = 32 Total train batch size (w. parallel, distributed & accumulation) = 64 Gradient Accumulation steps = 2 Total optimization steps = 9 0% 0/9 [00:00<?, ?it/s]Traceback (most recent call last): File "wav2vec_pretrain.py", line 388, in
main()
File "wav2vec_pretrain.py", line 384, in main
trainer.train()
File "/usr/local/lib/python3.7/dist-packages/transformers/trainer.py", line 1254, in train
for step, inputs in enumerate(epoch_iterator):
File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 521, in next
data = self._next_data()
File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 561, in _next_data
data = self._dataset_fetcher.fetch(index) # may raise StopIteration
File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
return self.collate_fn(data)
File "wav2vec_pretrain.py", line 176, in call
if batch["attention_mask"] is not None:
File "/usr/local/lib/python3.7/dist-packages/transformers/feature_extraction_utils.py", line 81, in getitem
return self.data[item]
KeyError: 'attention_mask'
Thank you very much!