WishperX with CUDA crashing in Docker

[FlyGoat]

2024-11-01 14:35:17.502 Lightning automatically upgraded your loaded checkpoint from v1.5.4 to v2.3.3. To apply the upgrade to your files permanently, run `python -m pytorch_lightning.utilities.upgrade_checkpoint ../root/.cache/torch/whisperx-vad-segmentation.bin`
2024-11-01 14:35:17.506 Created a temporary directory at /tmp/tmpjel29n24
2024-11-01 14:35:17.506 Writing /tmp/tmpjel29n24/_remote_module_non_scriptable.py
/opt/conda/lib/python3.10/site-packages/pyannote/audio/utils/reproducibility.py:74: ReproducibilityWarning: TensorFloat-32 (TF32) has been disabled as it might lead to reproducibility issues and lower accuracy.
It can be re-enabled by calling
   >>> import torch
   >>> torch.backends.cuda.matmul.allow_tf32 = True
   >>> torch.backends.cudnn.allow_tf32 = True
See https://github.com/pyannote/pyannote-audio/issues/1370 for more details.

  warnings.warn(
Unable to load any of {libcudnn_ops.so.9.1.0, libcudnn_ops.so.9.1, libcudnn_ops.so.9, libcudnn_ops.so}
Invalid handle. Cannot load symbol cudnnCreateTensorDescriptor

We are using cudnn8 in torch docker base image, this needs to be updated.

I'm now checking if updating breaks anything.

[abdeladim-s]

Yes, why not, please give it a try and let me know. Feel free to submit a PR if there are no issues.

[alfuraiji]

I'm facing the same issue and getting the same error with WhisperX CUDA works fine with the regular whisper

[abdeladim-s]

@alfuraiji, Can you please update the docker image to a newer cudnn and check if that solves the issue ?

[alfuraiji]

I really dont know how to do that Sorry I’m not that advance interms of doing stuff inside docker

BTW I’m running docker on windows and using WSL2 with it

[abdeladim-s]

It's Ok, I will try to take a look on it.