Not working with gpu_layers

MNekoRain commented 6 months ago

Hello! I have a problem. If I set gpu_layers above 0, that it's not working. If I try set below number, but it's still not working. Code:

from ctransformers import AutoModelForCausalLM
AutoModelForCausalLM.from_pretrained("G:/Models/llama-2-7b-chat.Q4_K_M.gguf", gpu_layers=35)

And error:

Traceback (most recent call last):
  File "G:\Programs\Newest\AI Tests\test.py", line 3, in <module>
    model = AutoModelForCausalLM.from_pretrained("G:/Models/llama-2-7b-chat.Q4_K_M.gguf", gpu_layers=35)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\ctransformers\hub.py", line 175, in from_pretrained
    llm = LLM(
          ^^^^
  File "C:\Python311\Lib\site-packages\ctransformers\llm.py", line 273, in __init__
    self._llm = self._lib.ctransformers_llm_create(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: [WinError -1073741795] Windows Error 0xc000001d

Commands, which I use:

pip install ctransformers pip install ctransformers[cuda]

Cuda 12.3 installed on Windows 10

Thank you in advance.

xiaoxiaobt commented 6 months ago

Hello, are you using an old CPU (for me 2-gen Xeon) that does not support AVX2? If so, check this reply https://github.com/marella/ctransformers/issues/167#issuecomment-1868235386

MNekoRain commented 6 months ago

Hello, are you using an old CPU (for me 2-gen Xeon) that does not support AVX2? If so, check this reply #167 (comment)

Thanks, It's works

marella / ctransformers

Not working with gpu_layers #198