Closed Sreyan88 closed 2 years ago
Hi Sreyan, thanks for your feedback. This issue is caused by the recent commit on EMA. It seems that some PyTorch modules cannot be deepcopied. I'm working on the problem and trying to fix it. Stay tuned!
Best, Aryan
Hi @arxyzan ,
Thank you for the reply. I also had the question if the code worked (till backprop while pre-training from your end). I made some hacks for the above loss, however, there seem to be some more errors:
This line here seems to produce an output of (batch_size x time_steps x time_steps x 768) for audio pre-training which is not expected. This in turn is returning an error while loss calculation.
Can you please check into these? Any help would be appreciated! Thanks much!
Can you paste the output of your error? I'm pretty sure it worked the last time I ran it.
File "train.py", line 25, in <module>
trainer.train()
File "/speech/sreyan/slt/data2vec-pytorch/audio/trainer.py", line 140, in train
train_loss = self.train_epoch(epoch)
File "/speech/sreyan/slt/data2vec-pytorch/audio/trainer.py", line 105, in train_epoch
loss = self.train_step(batch)
File "/speech/sreyan/slt/data2vec-pytorch/audio/trainer.py", line 66, in train_step
loss.backward()
File "/speech/sreyan/anaconda3/envs/data2vec/lib/python3.7/site-packages/torch/_tensor.py", line 307, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
File "/speech/sreyan/anaconda3/envs/data2vec/lib/python3.7/site-packages/torch/autograd/__init__.py", line 150, in backward
grad_tensors_ = _make_grads(tensors, grad_tensors_)
File "/speech/sreyan/anaconda3/envs/data2vec/lib/python3.7/site-packages/torch/autograd/__init__.py", line 51, in _make_grads
raise RuntimeError("grad can be implicitly created only for scalar outputs")
RuntimeError: grad can be implicitly created only for scalar outputs
You can try running :
python train.py --config audio/configs/wav2vec2-pretraining.yaml
to reproduce.
Unfortunately, I don't have access to the datasets anymore, hence cannot run pretraining script right now. Instead can you get the shape of loss tensor before loss.backward()
?
Hi, Sorry for the delay, I will post it today, was busy with conference deadlines. @arxyzan can you please re-open this?
Hi there!
Great repo! I am trying to pre-train wav2vec2 using this and I get the error:
RuntimeError: Only Tensors created explicitly by the user (graph leaves) support the deepcopy protocol at the moment
any fix for this?