Support for large model.

[Mybrc91]

Hi! Are you willing to add support for large models?

[pando85]

Any ideas about it? I created a docker image for running in old CPUs with CUDA support and I would be great to have support in the official package for large models.

[mackowskim]

I would second that. I'd really like to have some larger models avilable for GPU processing.

[sebastian-bugajny]

Would be great to be able to use larger models.

[neowisard]

Using Large model for English speaking users is not problem here.

1. Just download models https://huggingface.co/Systran/faster-whisper-large-v3/

2. Change config in wyoming_faster_whisper/download.py - just add in """Available faster-whisper models."""

   LARGE = "large-v3" LARGE_INT8 = "large-v3i" and start (my line for Cuda and INT8 for TeslaP40)

/usr/bin/python3 -m wyoming_faster_whisper --uri 'tcp://0.0.0.0:10300' --data-dir /ai/models/whisper --model large-v3i --beam-size 1 --language auto --download-dir /ai/models/whisper --device cuda --debug --compute-type int8_float32

Using latest version https://github.com/SYSTRAN/faster-whisper/ is problem and using anothers language too ( i speak ro , es and have translated english output only ).

[neowisard]

Just compile it from few forks https://github.com/neowisard/wyoming-faster-whisper/

[sebastian-bugajny]

@neowisard Is there a chance to pack it all into the docker image ?

[neowisard]

To do this you need to build nvidia\cuda container or other drivers in docker, it all depends on each configuration

[sebastian-bugajny]

Yeah I use nvidia/cuda 11.8 drivers but it can be passed via volumes like:

volumes:
  - /opt/docker/whisper:/data
  - /usr/local/lib/python3.10/dist-packages/nvidia/cudnn/lib/libcudnn_ops_infer.so.8:/usr/local/cuda-11.8/lib64/libcudnn_ops_infer.so.8:ro
  - /usr/local/lib/python3.10/dist-packages/nvidia/cudnn/lib/libcudnn_cnn_infer.so.8:/usr/local/cuda-11.8/lib64/libcudnn_cnn_infer.so.8:ro
  - /usr/local/lib/python3.10/dist-packages/nvidia/cublas/lib/libcublasLt.so.11:/usr/local/cuda-11.8/lib64/libcublasLt.so.11:ro
  - /usr/local/lib/python3.10/dist-packages/nvidia/cublas/lib/libcublas.so.11:/usr/local/cuda-11.8/lib64/libcublas.so.11:ro

[sebastian-bugajny]

@neowisard Trying to run it locally but after installation when I try to run it I'm getting:

Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 187, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/usr/lib/python3.10/runpy.py", line 146, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/usr/lib/python3.10/runpy.py", line 110, in _get_module_details
    __import__(pkg_name)
  File "/usr/local/lib/python3.10/dist-packages/wyoming_faster_whisper/__init__.py", line 7, in <module>
    __version__ = _VERSION_PATH.read_text(encoding="utf-8").strip()
  File "/usr/lib/python3.10/pathlib.py", line 1134, in read_text
    with self.open(mode='r', encoding=encoding, errors=errors) as f:
  File "/usr/lib/python3.10/pathlib.py", line 1119, in open
    return self._accessor.open(self, mode, buffering, encoding, errors,
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.10/dist-packages/wyoming_faster_whisper/VERSION'

[neowisard]

@neowisard Trying to run it locally but after installation when I try to run it I'm getting:

Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 187, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/usr/lib/python3.10/runpy.py", line 146, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/usr/lib/python3.10/runpy.py", line 110, in _get_module_details
    __import__(pkg_name)
  File "/usr/local/lib/python3.10/dist-packages/wyoming_faster_whisper/__init__.py", line 7, in <module>
    __version__ = _VERSION_PATH.read_text(encoding="utf-8").strip()
  File "/usr/lib/python3.10/pathlib.py", line 1134, in read_text
    with self.open(mode='r', encoding=encoding, errors=errors) as f:
  File "/usr/lib/python3.10/pathlib.py", line 1119, in open
    return self._accessor.open(self, mode, buffering, encoding, errors,
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.10/dist-packages/wyoming_faster_whisper/VERSION'

I only fixed this bug in my fork of https://github.com/neowisard/wyoming-faster-whisper/. The original version cannot use large versions.

[sebastian-bugajny]

@neowisard I used your fork and got such error.

[neowisard]

@neowisard I used your fork and got such error.

fixed https://github.com/neowisard/wyoming-faster-whisper/blob/master/README.md just follow readme

[sebastian-bugajny]

Unfortunately still doesn't work.

FYI pip install -r requirements.txt doesn't install wyoming_faster_whisper, there is only installed wyoming and faster-whisper based on requirements file.

I've done it like that

pip3 install . and it installed but now I'm getting this error:

Traceback (most recent call last):
  File "/usr/lib/python3.9/runpy.py", line 188, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/usr/lib/python3.9/runpy.py", line 147, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/usr/lib/python3.9/runpy.py", line 111, in _get_module_details
    __import__(pkg_name)
  File "/usr/local/lib/python3.9/dist-packages/wyoming_faster_whisper/__init__.py", line 7, in <module>
    __version__ = _VERSION_PATH.read_text(encoding="utf-8").strip()
  File "/usr/lib/python3.9/pathlib.py", line 1255, in read_text
    with self.open(mode='r', encoding=encoding, errors=errors) as f:
  File "/usr/lib/python3.9/pathlib.py", line 1241, in open
    return io.open(self, mode, buffering, encoding, errors, newline,
  File "/usr/lib/python3.9/pathlib.py", line 1109, in _opener
    return self._accessor.open(self, flags, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib/python3.9/dist-packages/wyoming_faster_whisper/VERSION'

[neowisard]

[Errno 2] No such file or directory: '/usr/local/lib/python3.9/dist-packages/wyoming_faster_whisper/VERSION'

We're not talking about my version right now. In my version (you can also create a ticket there) this file is there. You have already installed some version of the package which is launched and there is no version file there. You can try uninstalling it pip uninstall wyoming_faster_whisper and then run this package without installing it, from the source files (from my fork) with the -m switch. (Python3 -m git folder)

[sebastian-bugajny]

I tried exactly like you described, I run it without installation directly form repo directory

root@e02d182b6ea3:/opt/wyoming-faster-whisper# /usr/bin/python3 -m wyoming_faster_whisper --uri 'tcp://0.0.0.0:10300' --data-dir /opt/ --model large-v3 --beam-size 1 --language pl --download-dir /opt/ --compute-type float16 --device cuda --initial-prompt "promt"
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/opt/wyoming-faster-whisper/wyoming_faster_whisper/__main__.py", line 145, in <module>
    asyncio.run(main())
  File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/opt/wyoming-faster-whisper/wyoming_faster_whisper/__main__.py", line 103, in main
    AsrModel(
TypeError: AsrModel.__init__() got an unexpected keyword argument 'version'

[gruberdev]

@sebastian-bugajny

Yeah I use nvidia/cuda 11.8 drivers but it can be passed via volumes like:

volumes:

  - /opt/docker/whisper:/data

  - /usr/local/lib/python3.10/dist-packages/nvidia/cudnn/lib/libcudnn_ops_infer.so.8:/usr/local/cuda-11.8/lib64/libcudnn_ops_infer.so.8:ro

  - /usr/local/lib/python3.10/dist-packages/nvidia/cudnn/lib/libcudnn_cnn_infer.so.8:/usr/local/cuda-11.8/lib64/libcudnn_cnn_infer.so.8:ro

  - /usr/local/lib/python3.10/dist-packages/nvidia/cublas/lib/libcublasLt.so.11:/usr/local/cuda-11.8/lib64/libcublasLt.so.11:ro

  - /usr/local/lib/python3.10/dist-packages/nvidia/cublas/lib/libcublas.so.11:/usr/local/cuda-11.8/lib64/libcublas.so.11:ro

This is a bad idea. You shouldn't have to mount the driver files using a bind from the host system, CuBLaS should work by only exposing your GPU device to the dockershim layer.

How does your whole docker_compose.yml for the container looks like?

@neowisard

@neowisard I used your fork and got such error.

fixed https://github.com/neowisard/wyoming-faster-whisper/blob/master/README.md

just follow readme

I will use your fork to build and test my GPU inference on Kubernetes, and will report back with results! Thanks for the GPU-compatible initialization command.

[neowisard]

I tried exactly like you described, I run it without installation directly form repo directory

root@e02d182b6ea3:/opt/wyoming-faster-whisper# /usr/bin/python3 -m wyoming_faster_whisper --uri 'tcp://0.0.0.0:10300' --data-dir /opt/ --model large-v3 --beam-size 1 --language pl --download-dir /opt/ --compute-type float16 --device cuda --initial-prompt "promt"
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/opt/wyoming-faster-whisper/wyoming_faster_whisper/__main__.py", line 145, in <module>
    asyncio.run(main())
  File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/opt/wyoming-faster-whisper/wyoming_faster_whisper/__main__.py", line 103, in main
    AsrModel(
TypeError: AsrModel.__init__() got an unexpected keyword argument 'version'

In my fork these errors fixed. Tested/checked.

[sebastian-bugajny]

@gruberdev any luck with building it and running ? I tried to modified version parameter to 1.1.0 in setup.py but it didn't help.

[synesthesiam]

Support for the large model(s) has been added in 2.0.0 (see: https://github.com/rhasspy/wyoming-faster-whisper/releases/tag/v2.0.0)