Exllama can't be imported

TiagoMarinho commented 9 months ago

Describe the bug

A recent update has made it so that exllama does not work anymore when installing or migrating the webui from the old one-click installers.

2023-10-08 13:51:31 WARNING:exllama module failed to import. Will attempt to import from repositories/.
2023-10-08 13:51:44 ERROR:Could not find repositories/exllama. Please ensure that exllama (https://github.com/turboderp/exllama) is cloned inside repositories/ and is up to date.
2023-10-08 13:51:44 ERROR:Failed to load the model.

Note that despite what the error says, repositories/exllama does exist.

Is there an existing issue for this?

[X] I have searched the existing issues

Reproduction

Install from scratch using the Windows one-click installer
Try running any model using exllama

Screenshot

No response

Logs

Traceback (most recent call last):
  File "E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\modules\exllama.py", line 13, in <module>
    from exllama.generator import ExLlamaGenerator
  File "E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\exllama\__init__.py", line 1, in <module>
    from . import cuda_ext, generator, model, tokenizer
  File "E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\exllama\cuda_ext.py", line 9, in <module>
    import exllama_ext
ImportError: DLL load failed while importing exllama_ext: The specified procedure could not be found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\utils\cpp_extension.py", line 2100, in _run_ninja_build
    subprocess.run(
  File "E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\modules\ui_model_menu.py", line 201, in load_model_wrapper
    shared.model, shared.tokenizer = load_model(shared.model_name, loader)
  File "E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\modules\models.py", line 79, in load_model
    output = load_func_map[loader](model_name)
  File "E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\modules\models.py", line 324, in ExLlama_loader
    from modules.exllama import ExllamaModel
  File "E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\modules\exllama.py", line 22, in <module>
    from generator import ExLlamaGenerator
  File "E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\repositories\exllama\generator.py", line 1, in <module>
    import cuda_ext
  File "E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\repositories\exllama\cuda_ext.py", line 43, in <module>
    exllama_ext = load(
  File "E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\utils\cpp_extension.py", line 1308, in load
    return _jit_compile(
  File "E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\utils\cpp_extension.py", line 1710, in _jit_compile
    _write_ninja_file_and_build_library(
  File "E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\utils\cpp_extension.py", line 1823, in _write_ninja_file_and_build_library
    _run_ninja_build(
  File "E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\utils\cpp_extension.py", line 2116, in _run_ninja_build
    raise RuntimeError(message) from e
RuntimeError: Error building extension 'exllama_ext': [1/12] E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\bin\nvcc --generate-dependencies-with-compile --dependency-output q4_matmul.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4068 -Xcompiler /wd4067 -Xcompiler /wd4624 -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=exllama_ext -DTORCH_API_INCLUDE_EXTENSION_H -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\repositories\exllama\exllama_ext -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\torch\csrc\api\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\TH -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\THC -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17 -lineinfo -c E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\repositories\exllama\exllama_ext\cuda_func\q4_matmul.cu -o q4_matmul.cuda.o
FAILED: q4_matmul.cuda.o
E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\bin\nvcc --generate-dependencies-with-compile --dependency-output q4_matmul.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4068 -Xcompiler /wd4067 -Xcompiler /wd4624 -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=exllama_ext -DTORCH_API_INCLUDE_EXTENSION_H -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\repositories\exllama\exllama_ext -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\torch\csrc\api\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\TH -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\THC -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17 -lineinfo -c E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\repositories\exllama\exllama_ext\cuda_func\q4_matmul.cu -o q4_matmul.cuda.o
CreateProcess failed: The system cannot find the file specified.
[2/12] E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\bin\nvcc --generate-dependencies-with-compile --dependency-output column_remap.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4068 -Xcompiler /wd4067 -Xcompiler /wd4624 -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=exllama_ext -DTORCH_API_INCLUDE_EXTENSION_H -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\repositories\exllama\exllama_ext -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\torch\csrc\api\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\TH -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\THC -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17 -lineinfo -c E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\repositories\exllama\exllama_ext\cuda_func\column_remap.cu -o column_remap.cuda.o
FAILED: column_remap.cuda.o
E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\bin\nvcc --generate-dependencies-with-compile --dependency-output column_remap.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4068 -Xcompiler /wd4067 -Xcompiler /wd4624 -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=exllama_ext -DTORCH_API_INCLUDE_EXTENSION_H -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\repositories\exllama\exllama_ext -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\torch\csrc\api\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\TH -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\THC -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17 -lineinfo -c E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\repositories\exllama\exllama_ext\cuda_func\column_remap.cu -o column_remap.cuda.o
CreateProcess failed: The system cannot find the file specified.
[3/12] E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\bin\nvcc --generate-dependencies-with-compile --dependency-output cuda_buffers.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4068 -Xcompiler /wd4067 -Xcompiler /wd4624 -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=exllama_ext -DTORCH_API_INCLUDE_EXTENSION_H -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\repositories\exllama\exllama_ext -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\torch\csrc\api\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\TH -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\THC -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17 -lineinfo -c E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\repositories\exllama\exllama_ext\cuda_buffers.cu -o cuda_buffers.cuda.o
FAILED: cuda_buffers.cuda.o
E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\bin\nvcc --generate-dependencies-with-compile --dependency-output cuda_buffers.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4068 -Xcompiler /wd4067 -Xcompiler /wd4624 -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=exllama_ext -DTORCH_API_INCLUDE_EXTENSION_H -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\repositories\exllama\exllama_ext -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\torch\csrc\api\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\TH -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\THC -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17 -lineinfo -c E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\repositories\exllama\exllama_ext\cuda_buffers.cu -o cuda_buffers.cuda.o
CreateProcess failed: The system cannot find the file specified.
[4/12] E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\bin\nvcc --generate-dependencies-with-compile --dependency-output q4_matrix.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4068 -Xcompiler /wd4067 -Xcompiler /wd4624 -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=exllama_ext -DTORCH_API_INCLUDE_EXTENSION_H -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\repositories\exllama\exllama_ext -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\torch\csrc\api\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\TH -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\THC -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17 -lineinfo -c E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\repositories\exllama\exllama_ext\cuda_func\q4_matrix.cu -o q4_matrix.cuda.o
FAILED: q4_matrix.cuda.o
E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\bin\nvcc --generate-dependencies-with-compile --dependency-output q4_matrix.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4068 -Xcompiler /wd4067 -Xcompiler /wd4624 -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=exllama_ext -DTORCH_API_INCLUDE_EXTENSION_H -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\repositories\exllama\exllama_ext -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\torch\csrc\api\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\TH -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\THC -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17 -lineinfo -c E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\repositories\exllama\exllama_ext\cuda_func\q4_matrix.cu -o q4_matrix.cuda.o
CreateProcess failed: The system cannot find the file specified.
[5/12] E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\bin\nvcc --generate-dependencies-with-compile --dependency-output half_matmul.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4068 -Xcompiler /wd4067 -Xcompiler /wd4624 -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=exllama_ext -DTORCH_API_INCLUDE_EXTENSION_H -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\repositories\exllama\exllama_ext -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\torch\csrc\api\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\TH -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\THC -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17 -lineinfo -c E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\repositories\exllama\exllama_ext\cuda_func\half_matmul.cu -o half_matmul.cuda.o
FAILED: half_matmul.cuda.o
E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\bin\nvcc --generate-dependencies-with-compile --dependency-output half_matmul.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4068 -Xcompiler /wd4067 -Xcompiler /wd4624 -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=exllama_ext -DTORCH_API_INCLUDE_EXTENSION_H -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\repositories\exllama\exllama_ext -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\torch\csrc\api\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\TH -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\THC -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17 -lineinfo -c E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\repositories\exllama\exllama_ext\cuda_func\half_matmul.cu -o half_matmul.cuda.o
CreateProcess failed: The system cannot find the file specified.
[6/12] E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\bin\nvcc --generate-dependencies-with-compile --dependency-output q4_attn.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4068 -Xcompiler /wd4067 -Xcompiler /wd4624 -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=exllama_ext -DTORCH_API_INCLUDE_EXTENSION_H -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\repositories\exllama\exllama_ext -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\torch\csrc\api\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\TH -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\THC -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17 -lineinfo -c E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\repositories\exllama\exllama_ext\cuda_func\q4_attn.cu -o q4_attn.cuda.o
FAILED: q4_attn.cuda.o
E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\bin\nvcc --generate-dependencies-with-compile --dependency-output q4_attn.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4068 -Xcompiler /wd4067 -Xcompiler /wd4624 -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=exllama_ext -DTORCH_API_INCLUDE_EXTENSION_H -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\repositories\exllama\exllama_ext -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\torch\csrc\api\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\TH -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\THC -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17 -lineinfo -c E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\repositories\exllama\exllama_ext\cuda_func\q4_attn.cu -o q4_attn.cuda.o
CreateProcess failed: The system cannot find the file specified.
[7/12] E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\bin\nvcc --generate-dependencies-with-compile --dependency-output rms_norm.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4068 -Xcompiler /wd4067 -Xcompiler /wd4624 -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=exllama_ext -DTORCH_API_INCLUDE_EXTENSION_H -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\repositories\exllama\exllama_ext -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\torch\csrc\api\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\TH -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\THC -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17 -lineinfo -c E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\repositories\exllama\exllama_ext\cuda_func\rms_norm.cu -o rms_norm.cuda.o
FAILED: rms_norm.cuda.o
E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\bin\nvcc --generate-dependencies-with-compile --dependency-output rms_norm.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4068 -Xcompiler /wd4067 -Xcompiler /wd4624 -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=exllama_ext -DTORCH_API_INCLUDE_EXTENSION_H -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\repositories\exllama\exllama_ext -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\torch\csrc\api\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\TH -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\THC -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17 -lineinfo -c E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\repositories\exllama\exllama_ext\cuda_func\rms_norm.cu -o rms_norm.cuda.o
CreateProcess failed: The system cannot find the file specified.
[8/12] E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\bin\nvcc --generate-dependencies-with-compile --dependency-output q4_mlp.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4068 -Xcompiler /wd4067 -Xcompiler /wd4624 -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=exllama_ext -DTORCH_API_INCLUDE_EXTENSION_H -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\repositories\exllama\exllama_ext -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\torch\csrc\api\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\TH -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\THC -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17 -lineinfo -c E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\repositories\exllama\exllama_ext\cuda_func\q4_mlp.cu -o q4_mlp.cuda.o
FAILED: q4_mlp.cuda.o
E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\bin\nvcc --generate-dependencies-with-compile --dependency-output q4_mlp.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4068 -Xcompiler /wd4067 -Xcompiler /wd4624 -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=exllama_ext -DTORCH_API_INCLUDE_EXTENSION_H -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\repositories\exllama\exllama_ext -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\torch\csrc\api\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\TH -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\THC -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17 -lineinfo -c E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\repositories\exllama\exllama_ext\cuda_func\q4_mlp.cu -o q4_mlp.cuda.o
CreateProcess failed: The system cannot find the file specified.
[9/12] E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\bin\nvcc --generate-dependencies-with-compile --dependency-output rope.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4068 -Xcompiler /wd4067 -Xcompiler /wd4624 -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=exllama_ext -DTORCH_API_INCLUDE_EXTENSION_H -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\repositories\exllama\exllama_ext -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\torch\csrc\api\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\TH -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\THC -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17 -lineinfo -c E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\repositories\exllama\exllama_ext\cuda_func\rope.cu -o rope.cuda.o
FAILED: rope.cuda.o
E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\bin\nvcc --generate-dependencies-with-compile --dependency-output rope.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4068 -Xcompiler /wd4067 -Xcompiler /wd4624 -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=exllama_ext -DTORCH_API_INCLUDE_EXTENSION_H -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\repositories\exllama\exllama_ext -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\torch\csrc\api\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\TH -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\lib\site-packages\torch\include\THC -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\include -IE:\AI\oobabooga\text-gen-webui-2\text-generation-webui\installer_files\env\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -std=c++17 -lineinfo -c E:\AI\oobabooga\text-gen-webui-2\text-generation-webui\repositories\exllama\exllama_ext\cuda_func\rope.cu -o rope.cuda.o
CreateProcess failed: The system cannot find the file specified.
ninja: fatal: ReadFile: The handle is invalid.

System Info

OS: Windows 11 22H2
GPU: RTX 3080 10GB
CPU: i9-9900KF
RAM: 48GB DDR4

pishiver commented 9 months ago

I'm having this exact same problem. Installed it several times over the last few days with no issues. (I was experimenting with different linux distros, got fed up with linux and switched back to win11) and all of a sudden today it stopped being able to load models on exllama, exllama2,(and the hf versions of both), autogptq, and autoawq....all no-gos with similar errors.

Edit: it seems that loading GGUFs is the only thing working

Apicio commented 9 months ago

Same for me with Exllama and AutoAWQ it's impossibile to load models

HotChocut commented 9 months ago

I tried out some different builds and 06fff3b2e918f37e0c7d477d111a08d042e72968 seems to be the latest one that works if you wanna revert as a temporary fix I guess bumping to pytorch 11.8 in d33facc9feea19158420ddcbc842509825a4e88b is what broke it ¯\_(ツ)_/¯

Hayanaga commented 9 months ago

I tried out some different builds and 06fff3b seems to be the latest one that works if you wanna revert as a temporary fix I guess bumping to pytorch 11.8 in d33facc is what broke it ¯_(ツ)_/¯

Thanks, I was hitting my face against a wall for almost 3 hours trying to figure out that mess, I was trying to upgrade from a very old webui version only to crash land myself with so many issues, even installing from scratch,..

Ended up cloning 06fff3b and everything works as it should, I'm gonna keep this version for now,,,

M4st3rfun commented 9 months ago

Hello, i have the same problem, i cant load Llama v1 AI LLM stuff but llamav2 works but it starts to lag a lot and when they want to answers back to me, the AI takes like 200-300 seconds, before it took like 20-50 seconds consistent, my system specs are intel core i5 8600K 2080 Gaming OC, 32 GB Ram win10 i use Mythalion-13B-GPTQ and Pygmalion-2-13B-GPTQ.

how do i do revert back to the old version until it gets fixed?, i have no idea how to do it! can somebody help out? thank you so much! I will upload 2 pictures, i didn't had this problem 2 weeks ago, i dont know what happened! srg www

AlyxMS commented 9 months ago

https://github.com/oobabooga/text-generation-webui/issues/4225#issuecomment-1759981218

how do i do revert back to the old version until it gets fixed?, i have no idea how to do it! can somebody help out? thank you

Goto the code page and find releases on the right side. Then click releases to view all past releases. 1.7 feels too recent so I didn't try that one. 1.6.1 worked flawlessly for me so far. (Previously tried 06fff3b, loads the model fine but errors on inference) Click download source code and use the typical bat file to install. Copy over any models, lora or other configuration files you have from the old version and never touch the update bat file until problem is fixed.

HotChocut commented 9 months ago

I tired the latest version of webui (c0ffb77) and it seems the issue has been fixed (at least for me)

M4st3rfun commented 9 months ago

I tried to install the new version too but i get a -> but i have gradio installed but still says the same thing

HotChocut commented 9 months ago

I had to do a clean install to switch to the newest version so maybe try that? (Assuming what you did was try and run update.bat) I should also probably mention that I'm on Windows 10 so results may vary with other operating systems

TiagoMarinho commented 9 months ago

I tired the latest version of webui (c0ffb77) and it seems the issue has been fixed (at least for me)

can confirm the problem does not happen after updating the webui, closing the issue If anyone else is still suffering from this, I think it's best to create a new issue, as the one I reported is now fixed

oobabooga / text-generation-webui