Colab notebook doesn't work

[epris]

Hello, I'm trying to run the Colab online demo but I obtain different errors at the runtime 1)``` ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. torchdata 0.6.1 requires torch==2.0.1, but you have torch 1.11.0 which is incompatible. torchtext 0.15.2 requires torch==2.0.1, but you have torch 1.11.0 which is incompatible.

2) `ERROR: Cannot install pyglet==1.5.27, pyopengl==3.1.5, pyrender==0.1, pyrender==0.1.1, pyrender==0.1.10, pyrender==0.1.11, pyrender==0.1.12, pyrender==0.1.13, pyrender==0.1.14, pyrender==0.1.15, pyrender==0.1.16, pyrender==0.1.17, pyrender==0.1.18, pyrender==0.1.2, pyrender==0.1.20, pyrender==0.1.21, pyrender==0.1.22, pyrender==0.1.23, pyrender==0.1.24, pyrender==0.1.25, pyrender==0.1.26, pyrender==0.1.27, pyrender==0.1.28, pyrender==0.1.29, pyrender==0.1.3, pyrender==0.1.30, pyrender==0.1.31, pyrender==0.1.32, pyrender==0.1.33, pyrender==0.1.34, pyrender==0.1.35, pyrender==0.1.36, pyrender==0.1.39, pyrender==0.1.4, pyrender==0.1.40, pyrender==0.1.41, pyrender==0.1.42, pyrender==0.1.43, pyrender==0.1.44, pyrender==0.1.45, pyrender==0.1.5, pyrender==0.1.6, pyrender==0.1.7, pyrender==0.1.8 and pyrender==0.1.9 because these package versions have conflicting dependencies.`

3) ```
Building wheels for collected packages: tokenizers, sacremoses
  error: subprocess-exited-with-error

  × Building wheel for tokenizers (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  Building wheel for tokenizers (pyproject.toml) ... error
  ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects
Hit:1 https://cloud.r-project.org/bin/linux/ubuntu jammy-cran40/ InRelease
Hit:2 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64  InRelease
Hit:3 http://security.ubuntu.com/ubuntu jammy-security InRelease
Hit:4 http://archive.ubuntu.com/ubuntu jammy InRelease
Hit:5 https://ppa.launchpadcontent.net/c2d4u.team/c2d4u4.0+/ubuntu jammy InRelease
Hit:6 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
Hit:7 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
Hit:8 https://ppa.launchpadcontent.net/deadsnakes/ppa/ubuntu jammy InRelease
Hit:9 https://ppa.launchpadcontent.net/graphics-drivers/ppa/ubuntu jammy InRelease
Hit:10 https://ppa.launchpadcontent.net/ubuntugis/ppa/ubuntu jammy InRelease
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
16 packages can be upgraded. Run 'apt list --upgradable' to see them.

4)``` Traceback (most recent call last): File "/content/CodeTalker/main/demo.py", line 9, in from transformers import Wav2Vec2Processor ModuleNotFoundError: No module named 'transformers'

[enrico310786]

I have the same error. Any suggestions?

[Doubiiu]

Hi. It seems that google colab changes its default environment, which leads to setup error. I will update the colab code ASAP. :( I am sorry that I am currently busy working on an academic project :(

[itxtayab]

Still the same issue

[MohitPanpaliya]

still the same issue

[Doubiiu]

Prepare a new Colab notebook here. However, there are some artifacts in the VOCASET-template results due to the difference between the installed versions of packages and those in this repo/paper. It is hard to configure the environment in Colab to be the same as the one in this repo and I have tried my best.

[IH-Chung]

Prepare a new Colab notebook here. However, there are some artifacts in the VOCASET-template results due to the difference between the installed versions of packages and those in this repo/paper. It is hard to configure the environment in Colab to be the same as the one in this repo and I have tried my best.

In new Colab notebook show:

!sh scripts/demo.sh vocaset

/content/CodeTalker
Some weights of Wav2Vec2Model were not initialized from the model checkpoint at facebook/wav2vec2-base-960h and are newly initialized: ['wav2vec2.masked_spec_embed']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
=> loading checkpoint 'vocaset/vocaset_stage2.pth.tar'
=> loaded checkpoint 'vocaset/vocaset_stage2.pth.tar'
Generating facial animation for demo/wav/man.wav...
2024-03-20 12:34:40.717035: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-03-20 12:34:40.717087: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-03-20 12:34:40.718591: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-03-20 12:34:41.921161: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Traceback (most recent call last):
File "/content/CodeTalker/main/demo.py", line 219, in <module>
main()
File "/content/CodeTalker/main/demo.py", line 129, in main
test(model, cfg.demo_wav_path, save_folder, condition, subject)
File "/content/CodeTalker/main/demo.py", line 167, in test
prediction = model.predict(audio_feature, template, one_hot)
File "/content/CodeTalker/models/stage2.py", line 115, in predict
hidden_states = self.audio_encoder(audio, self.dataset).last_hidden_state
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1110, in _call_impl
return forward_call(*input, **kwargs)
File "/content/CodeTalker/models/lib/wav2vec.py", line 132, in forward
encoder_outputs = self.encoder(
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1110, in _call_impl
return forward_call(*input, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 787, in forward
position_embeddings = self.pos_conv_embed(hidden_states)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1110, in _call_impl
return forward_call(*input, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 399, in forward
hidden_states = hidden_states.transpose(1, 2)
IndexError: Dimension out of range (expected to be in range of [-2, 1], but got 2)

!sh scripts/demo.sh BIWI

Some weights of Wav2Vec2Model were not initialized from the model checkpoint at facebook/wav2vec2-base-960h and are newly initialized: ['wav2vec2.masked_spec_embed']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
=> loading checkpoint 'BIWI/biwi_stage2.pth.tar'
=> loaded checkpoint 'BIWI/biwi_stage2.pth.tar'
Generating facial animation for demo/wav/man.wav...
2024-03-20 12:37:31.046308: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-03-20 12:37:31.046378: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-03-20 12:37:31.048398: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-03-20 12:37:32.752418: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
Traceback (most recent call last):
File "/content/CodeTalker/main/demo.py", line 219, in <module>
main()
File "/content/CodeTalker/main/demo.py", line 129, in main
test(model, cfg.demo_wav_path, save_folder, condition, subject)
File "/content/CodeTalker/main/demo.py", line 167, in test
prediction = model.predict(audio_feature, template, one_hot)
File "/content/CodeTalker/models/stage2.py", line 115, in predict
hidden_states = self.audio_encoder(audio, self.dataset).last_hidden_state
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1110, in _call_impl
return forward_call(*input, **kwargs)
File "/content/CodeTalker/models/lib/wav2vec.py", line 132, in forward
encoder_outputs = self.encoder(
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1110, in _call_impl
return forward_call(*input, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 787, in forward
position_embeddings = self.pos_conv_embed(hidden_states)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1110, in _call_impl
return forward_call(*input, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/transformers/models/wav2vec2/modeling_wav2vec2.py", line 399, in forward
hidden_states = hidden_states.transpose(1, 2)
IndexError: Dimension out of range (expected to be in range of [-2, 1], but got 2)