Unable to update or run (quant-cuda error)

Describe the bug

Was working fine for the last few days, and I update before each launch fine. today though i get an error updating. I am using the one click install version and updating via the bat file I did try to re-install the requirements.txt manually to see if that resolved it but it did not. The app has now lost the ability to load models.

Found a similar issue in the search that turned out the person was using the cpu version of pytorch, unless that has changed on my syatem (cant see why it would have) then i do not feel this is the same issue

Is there an existing issue for this?

[X] I have searched the existing issues

Reproduction

run update_windows.bat

Screenshot

No response

Logs

Building wheels for collected packages: quant-cuda
  Building wheel for quant-cuda (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [52 lines of output]
      No CUDA runtime is found, using CUDA_HOME='C:\AI\oobabooga_windows\installer_files\env'
      running bdist_wheel
      running build
      running build_ext
      C:\Users\mrgre\AppData\Roaming\Python\Python310\site-packages\torch\utils\cpp_extension.py:359: UserWarning: Error checking compiler version for cl: [WinError 2] The system cannot find the file specified
        warnings.warn(f'Error checking compiler version for {compiler}: {error}')
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "C:\AI\oobabooga_windows\text-generation-webui\repositories\GPTQ-for-LLaMa\setup.py", line 4, in <module>
          setup(
        File "C:\AI\oobabooga_windows\installer_files\env\lib\site-packages\setuptools\__init__.py", line 87, in setup
          return distutils.core.setup(**attrs)
        File "C:\AI\oobabooga_windows\installer_files\env\lib\site-packages\setuptools\_distutils\core.py", line 185, in setup
          return run_commands(dist)
        File "C:\AI\oobabooga_windows\installer_files\env\lib\site-packages\setuptools\_distutils\core.py", line 201, in run_commands
          dist.run_commands()
        File "C:\AI\oobabooga_windows\installer_files\env\lib\site-packages\setuptools\_distutils\dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "C:\AI\oobabooga_windows\installer_files\env\lib\site-packages\setuptools\dist.py", line 1208, in run_command
          super().run_command(command)
        File "C:\AI\oobabooga_windows\installer_files\env\lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command
          cmd_obj.run()
        File "C:\AI\oobabooga_windows\installer_files\env\lib\site-packages\wheel\bdist_wheel.py", line 325, in run
          self.run_command("build")
        File "C:\AI\oobabooga_windows\installer_files\env\lib\site-packages\setuptools\_distutils\cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "C:\AI\oobabooga_windows\installer_files\env\lib\site-packages\setuptools\dist.py", line 1208, in run_command
          super().run_command(command)
        File "C:\AI\oobabooga_windows\installer_files\env\lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command
          cmd_obj.run()
        File "C:\AI\oobabooga_windows\installer_files\env\lib\site-packages\setuptools\_distutils\command\build.py", line 132, in run
          self.run_command(cmd_name)
        File "C:\AI\oobabooga_windows\installer_files\env\lib\site-packages\setuptools\_distutils\cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "C:\AI\oobabooga_windows\installer_files\env\lib\site-packages\setuptools\dist.py", line 1208, in run_command
          super().run_command(command)
        File "C:\AI\oobabooga_windows\installer_files\env\lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command
          cmd_obj.run()
        File "C:\AI\oobabooga_windows\installer_files\env\lib\site-packages\setuptools\command\build_ext.py", line 84, in run
          _build_ext.run(self)
        File "C:\AI\oobabooga_windows\installer_files\env\lib\site-packages\setuptools\_distutils\command\build_ext.py", line 346, in run
          self.build_extensions()
        File "C:\Users\mrgre\AppData\Roaming\Python\Python310\site-packages\torch\utils\cpp_extension.py", line 499, in build_extensions
          _check_cuda_version(compiler_name, compiler_version)
        File "C:\Users\mrgre\AppData\Roaming\Python\Python310\site-packages\torch\utils\cpp_extension.py", line 383, in _check_cuda_version
          torch_cuda_version = packaging.version.parse(torch.version.cuda)
        File "C:\AI\oobabooga_windows\installer_files\env\lib\site-packages\pkg_resources\_vendor\packaging\version.py", line 49, in parse
          return Version(version)
        File "C:\AI\oobabooga_windows\installer_files\env\lib\site-packages\pkg_resources\_vendor\packaging\version.py", line 264, in __init__
          match = self._regex.search(version)
      TypeError: expected string or bytes-like object
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for quant-cuda
  Running setup.py clean for quant-cuda
Failed to build quant-cuda
Installing collected packages: quant-cuda
  Attempting uninstall: quant-cuda
    Found existing installation: quant-cuda 0.0.0
    Uninstalling quant-cuda-0.0.0:
      Successfully uninstalled quant-cuda-0.0.0
  Running setup.py install for quant-cuda ... error
  error: subprocess-exited-with-error

  × Running setup.py install for quant-cuda did not run successfully.
  │ exit code: 1
  ╰─> [56 lines of output]
      No CUDA runtime is found, using CUDA_HOME='C:\AI\oobabooga_windows\installer_files\env'
      running install
      C:\AI\oobabooga_windows\installer_files\env\lib\site-packages\setuptools\command\install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_ext
      C:\Users\mrgre\AppData\Roaming\Python\Python310\site-packages\torch\utils\cpp_extension.py:359: UserWarning: Error checking compiler version for cl: [WinError 2] The system cannot find the file specified
        warnings.warn(f'Error checking compiler version for {compiler}: {error}')
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "C:\AI\oobabooga_windows\text-generation-webui\repositories\GPTQ-for-LLaMa\setup.py", line 4, in <module>
          setup(
        File "C:\AI\oobabooga_windows\installer_files\env\lib\site-packages\setuptools\__init__.py", line 87, in setup
          return distutils.core.setup(**attrs)
        File "C:\AI\oobabooga_windows\installer_files\env\lib\site-packages\setuptools\_distutils\core.py", line 185, in setup
          return run_commands(dist)
        File "C:\AI\oobabooga_windows\installer_files\env\lib\site-packages\setuptools\_distutils\core.py", line 201, in run_commands
          dist.run_commands()
        File "C:\AI\oobabooga_windows\installer_files\env\lib\site-packages\setuptools\_distutils\dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "C:\AI\oobabooga_windows\installer_files\env\lib\site-packages\setuptools\dist.py", line 1208, in run_command
          super().run_command(command)
        File "C:\AI\oobabooga_windows\installer_files\env\lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command
          cmd_obj.run()
        File "C:\AI\oobabooga_windows\installer_files\env\lib\site-packages\setuptools\command\install.py", line 68, in run
          return orig.install.run(self)
        File "C:\AI\oobabooga_windows\installer_files\env\lib\site-packages\setuptools\_distutils\command\install.py", line 698, in run
          self.run_command('build')
        File "C:\AI\oobabooga_windows\installer_files\env\lib\site-packages\setuptools\_distutils\cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "C:\AI\oobabooga_windows\installer_files\env\lib\site-packages\setuptools\dist.py", line 1208, in run_command
          super().run_command(command)
        File "C:\AI\oobabooga_windows\installer_files\env\lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command
          cmd_obj.run()
        File "C:\AI\oobabooga_windows\installer_files\env\lib\site-packages\setuptools\_distutils\command\build.py", line 132, in run
          self.run_command(cmd_name)
        File "C:\AI\oobabooga_windows\installer_files\env\lib\site-packages\setuptools\_distutils\cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "C:\AI\oobabooga_windows\installer_files\env\lib\site-packages\setuptools\dist.py", line 1208, in run_command
          super().run_command(command)
        File "C:\AI\oobabooga_windows\installer_files\env\lib\site-packages\setuptools\_distutils\dist.py", line 988, in run_command
          cmd_obj.run()
        File "C:\AI\oobabooga_windows\installer_files\env\lib\site-packages\setuptools\command\build_ext.py", line 84, in run
          _build_ext.run(self)
        File "C:\AI\oobabooga_windows\installer_files\env\lib\site-packages\setuptools\_distutils\command\build_ext.py", line 346, in run
          self.build_extensions()
        File "C:\Users\mrgre\AppData\Roaming\Python\Python310\site-packages\torch\utils\cpp_extension.py", line 499, in build_extensions
          _check_cuda_version(compiler_name, compiler_version)
        File "C:\Users\mrgre\AppData\Roaming\Python\Python310\site-packages\torch\utils\cpp_extension.py", line 383, in _check_cuda_version
          torch_cuda_version = packaging.version.parse(torch.version.cuda)
        File "C:\AI\oobabooga_windows\installer_files\env\lib\site-packages\pkg_resources\_vendor\packaging\version.py", line 49, in parse
          return Version(version)
        File "C:\AI\oobabooga_windows\installer_files\env\lib\site-packages\pkg_resources\_vendor\packaging\version.py", line 264, in __init__
          match = self._regex.search(version)
      TypeError: expected string or bytes-like object
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  Rolling back uninstall of quant-cuda
  Moving to c:\ai\oobabooga_windows\installer_files\env\lib\site-packages\quant_cuda-0.0.0.dist-info\
   from C:\AI\oobabooga_windows\installer_files\env\Lib\site-packages\~uant_cuda-0.0.0.dist-info
  Moving to c:\ai\oobabooga_windows\installer_files\env\lib\site-packages\quant_cuda.cp310-win_amd64.pyd
   from C:\AI\oobabooga_windows\installer_files\pip-uninstall-o0ea0v5m\quant_cuda.cp310-win_amd64.pyd
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> quant-cuda

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

Done!
Press any key to continue . . .

System Info

windows 11 64bit
rtx 3090
64gb ram

Tried a fresh install, just before selecting a model I got the same error.

Noticed some people with similar issues needed c++ build tools. installed the full visual studio community edition made sure teh latest c++ build tools are installed in the individual modules section.

ran the updater.... same error

running the start bat and choosing a model shows this error:

`bin C:\AI\oobabooga_windows\installer_files\env\lib\site-packages\bitsandbytes\libbitsandbytes_cpu.dll C:\AI\oobabooga_windows\installer_files\env\lib\site-packages\bitsandbytes\cextension.py:33: UserWarning: The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable. warn("The installed version of bitsandbytes was compiled without GPU support. " The following models are available:

openaccess-ai-collective_wizard-mega-13b
TheBloke_wizard-mega-13B-GPTQ
TheBloke_Wizard-Vicuna-13B-Uncensored-GPTQ
TheBloke_Wizard-Vicuna-13B-Uncensored-HF
TheBloke_WizardLM-7B-uncensored-GPTQ
TheBloke_WizardLM-13B-Uncensored-GGML

Which one do you want to load? 1-6

INFO:Loading TheBloke_WizardLM-7B-uncensored-GPTQ... CUDA extension not installed. INFO:Found the following quantized model: models\TheBloke_WizardLM-7B-uncensored-GPTQ\WizardLM-7B-uncensored-GPTQ-4bit-128g.compat.no-act-order.safetensors Traceback (most recent call last): File "C:\AI\oobabooga_windows\text-generation-webui\server.py", line 984, in shared.model, shared.tokenizer = load_model(shared.model_name) File "C:\AI\oobabooga_windows\text-generation-webui\modules\models.py", line 95, in load_model output = load_func(model_name) File "C:\AI\oobabooga_windows\text-generation-webui\modules\models.py", line 272, in GPTQ_loader model = modules.GPTQ_loader.load_quantized(model_name) File "C:\AI\oobabooga_windows\text-generation-webui\modules\GPTQ_loader.py", line 199, in load_quantized model = model.to(torch.device('cuda:0')) File "C:\AI\oobabooga_windows\installer_files\env\lib\site-packages\transformers\modeling_utils.py", line 1896, in to return super().to(*args, **kwargs) File "C:\Users\mrgre\AppData\Roaming\Python\Python310\site-packages\torch\nn\modules\module.py", line 1145, in to return self._apply(convert) File "C:\Users\mrgre\AppData\Roaming\Python\Python310\site-packages\torch\nn\modules\module.py", line 797, in _apply module._apply(fn) File "C:\Users\mrgre\AppData\Roaming\Python\Python310\site-packages\torch\nn\modules\module.py", line 797, in _apply module._apply(fn) File "C:\Users\mrgre\AppData\Roaming\Python\Python310\site-packages\torch\nn\modules\module.py", line 820, in _apply param_applied = fn(param) File "C:\Users\mrgre\AppData\Roaming\Python\Python310\site-packages\torch\nn\modules\module.py", line 1143, in convert return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking) File "C:\Users\mrgre\AppData\Roaming\Python\Python310\site-packages\torch\cuda__init__.py", line 239, in _lazy_init raise AssertionError("Torch not compiled with CUDA enabled") AssertionError: Torch not compiled with CUDA enabled

Done! Press any key to continue . . .`

A couple of days ago this was working 100% I have no idea why it suddenly wont update or run, the fact a fresh install wont run either and that I seem to be the only one with this exact issue, makes me think its something on my system but no idea what

I'd give this a shot: #1289 (comment)

sadly this fails at the first hurdle

Microsoft Windows [Version 10.0.22621.1702]
(c) Microsoft Corporation. All rights reserved.

C:\AI\oobabooga_windows>python -m torch.utils.collect_env
Collecting environment information...
Traceback (most recent call last):
  File "C:\Program Files\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Program Files\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\mrgre\AppData\Roaming\Python\Python310\site-packages\torch\utils\collect_env.py", line 602, in <module>
    main()
  File "C:\Users\mrgre\AppData\Roaming\Python\Python310\site-packages\torch\utils\collect_env.py", line 585, in main
    output = get_pretty_env_info()
  File "C:\Users\mrgre\AppData\Roaming\Python\Python310\site-packages\torch\utils\collect_env.py", line 580, in get_pretty_env_info
    return pretty_str(get_env_info())
  File "C:\Users\mrgre\AppData\Roaming\Python\Python310\site-packages\torch\utils\collect_env.py", line 422, in get_env_info
    pip_version, pip_list_output = get_pip_packages(run_lambda)
  File "C:\Users\mrgre\AppData\Roaming\Python\Python310\site-packages\torch\utils\collect_env.py", line 394, in get_pip_packages
    out = run_with_pip(sys.executable + ' -mpip')
  File "C:\Users\mrgre\AppData\Roaming\Python\Python310\site-packages\torch\utils\collect_env.py", line 382, in run_with_pip
    for line in out.splitlines()
AttributeError: 'NoneType' object has no attribute 'splitlines'

C:\AI\oobabooga_windows>

comfyui, automatic1111 all work fine so I assume torch and cuda is installed fine...but not sure

Correct me if I'm wrong, but from the log output you provided, it doesn't look like you used cmd_windows.bat.

But first, I would suggest you:

Remove or rename the env folder in: .\oobaboogaAI\installer_files\
Then click the start_windows.bat to rebuild the environment. (You can then remove the old env folder if you renamed it).
Once that is done, redo those commands using that bat file. https://github.com/oobabooga/text-generation-webui/issues/1289#issuecomment-1518719987

Correct me if I'm wrong, but from the log output you provided, it doesn't look like you used cmd_windows.bat.

But first, I would suggest you:

Remove or rename the env folder in: .\oobaboogaAI\installer_files\

Then click the start_windows.bat to rebuild the environment. (You can then remove the old env folder if you renamed it).

Once that is done, redo those commands using that bat file. Cuda extension not installed #1289 (comment)

I did not know about that bat file, usually i will just do cmd in the address bar and go from there. followed the steps you linked to in your earlier post (i have tried the fresh install and the deleting of the installer files before, tried tons of things in the last couple of days lol) The steps you linked too earlier allowed me to do the update with out error! yay and it loads too! Thankyou! I was starting to lose hope that it could be fixed with out a full system re-install

Ah it's easy to miss the conda script, just be sure to have the latest one-click installers, my previous environment got messed up and I was using the old installers, so a fresh install with the new scripts worked.

But glad to hear it's working, be sure to close this issue :)

The problem with updating is that after downloading GPTQ-for-LLaMa repository, the file setup_cuda.py gets renamed to setup.py.

Next time you run update_windows.bat, the git pull command fails for that repository due to unstaged changes caused by this rename:

TL;DR — someone should submit a pull request for that repo to change the file name so it doesn't have to be done locally, or just change whatever logic is used to use the original file name without renaming it.

This issue has been closed due to inactivity for 30 days. If you believe it is still relevant, please leave a comment below.

well the issue remains, there is no easy to follow guide on getting qlora training or gptq 4bit training to work in textgen webui. There are the occasional posts that asay things like grab x and build its kernal... which makes little sense to people like myself (and I am an old geek)

I feel lora training is just abandoned at this stage.

oobabooga / text-generation-webui