search

marella / ctransformers

Python bindings for the Transformer models implemented in C/C++ using GGML library.
MIT License
1.79k stars 135 forks source link

Can i run on windows? #76

Open talhaanwarch opened 1 year ago

talhaanwarch commented 1 year ago

I tried to run on windows and i get this error

ERROR:    Traceback (most recent call last):
  File "C:\streamlit-llama\venv\lib\site-packages\starlette\routing.py", line 677, in lifespan
    async with self.lifespan_context(app) as maybe_state:
  File "C:\streamlit-llama\venv\lib\site-packages\starlette\routing.py", line 566, in __aenter__
    await self._router.startup()
  File "C:\streamlit-llama\venv\lib\site-packages\starlette\routing.py", line 654, in startup
    await handler()
  File "C:\streamlit-llama\api.py", line 22, in startup_event
    app.state.model = load_model()
  File "C:\streamlit-llama\api.py", line 15, in load_model
    return AutoModelForCausalLM.from_pretrained(
  File "C:\streamlit-llama\venv\lib\site-packages\ctransformers\hub.py", line 168, in from_pretrained
    return LLM(
  File "C:\streamlit-llama\venv\lib\site-packages\ctransformers\llm.py", line 206, in __init__
    self._lib = load_library(lib, cuda=config.gpu_layers > 0)
  File "C:\streamlit-llama\venv\lib\site-packages\ctransformers\llm.py", line 102, in load_library
    lib = CDLL(path)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.3056.0_x64__qbz5n2kfra8p0\lib\ctypes\__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
FileNotFoundError: Could not find module 'C:\streamlit-llama\venv\Lib\site-packages\ctransformers\lib\avx2\ctransformers.dll' (or one of its dependencies). Try using the full path with constructor syntax.

ERROR:    Application startup failed. Exiting.

Windows 11

bbartling commented 1 year ago

Windows 10 and python 3.10 is working for me just fine.

bbartling commented 1 year ago

I am just experimenting with this on the fly without even a virtual environment and havent ran into any issues, here's a pip list on my end:

> py -3.10 -m  pip list
Package                      Version
---------------------------- -----------
absl-py                      1.4.0
aiohttp                      3.8.3
aiohttp-basicauth            1.0.0
aiosignal                    1.3.1
ale-py                       0.7.4
altair                       4.2.0
altgraph                     0.17.3
annoy                        1.17.3
anyio                        3.6.2
APScheduler                  3.9.1.post1
argcomplete                  3.1.1
argon2-cffi                  21.3.0
argon2-cffi-bindings         21.2.0
arrow                        1.2.3
arviz                        0.12.1
asciichartpy                 1.5.25
asttokens                    2.2.1
astunparse                   1.6.3
async-timeout                4.0.2
asyncio-mqtt                 0.16.1
attrs                        22.2.0
autopep8                     1.7.0
AutoROM                      0.6.1
AutoROM.accept-rom-license   0.6.1
BAC0                         21.12.3
backcall                     0.2.0
backoff                      2.2.1
bacpypes                     0.18.6
Beaker                       1.12.1
beautifulsoup4               4.12.2
black                        23.3.0
bleach                       6.0.0
blinker                      1.5
branca                       0.6.0
Brotli                       1.0.9
cachetools                   5.2.0
censys                       2.2.4
certifi                      2022.12.7
cffi                         1.15.1
cftime                       1.6.2
charset-normalizer           2.1.1
chroma-hnswlib               0.7.1
chromadb                     0.4.3
click                        8.1.3
click-plugins                1.1.1
cligj                        0.7.2
cloudpickle                  2.2.1
colorama                     0.4.5
coloredlogs                  15.0.1
comm                         0.1.3
commonmark                   0.9.1
contourpy                    1.0.6
cryptography                 39.0.0
cssselect2                   0.7.0
ctransformers                0.2.20
cvxpy                        1.3.1
cycler                       0.11.0
dataclasses-json             0.5.13
datasets                     2.14.1
debugpy                      1.6.7
decorator                    5.1.1
defusedxml                   0.7.1
deprecat                     2.1.1
dill                         0.3.7
diskcache                    5.6.1
docopt                       0.6.2
docx                         0.2.4
ecos                         2.0.12
eemeter                      3.1.1
energyplus-api-helpers       0.4
entrypoints                  0.4
et-xmlfile                   1.1.0
exceptiongroup               1.1.0
executing                    1.2.0
faiss-cpu                    1.7.4
fastapi                      0.95.2
fastjsonschema               2.16.3
fastprogress                 1.0.3
filelock                     3.12.0
Fiona                        1.9.4.post1
Flask                        2.2.2
Flask-Classful               0.14.2
Flask-Cors                   3.0.10
Flask-Pydantic               0.11.0
Flask-SQLAlchemy             3.0.2
flatbuffers                  23.1.21
folium                       0.14.0
fonttools                    4.38.0
fqdn                         1.5.1
frozenlist                   1.3.3
fsspec                       2023.6.0
future                       0.18.2
g36-ahu-faults               0.0.1
gast                         0.4.0
gensim                       4.3.1
geopandas                    0.13.2
gitdb                        4.0.10
GitPython                    3.1.29
google-auth                  2.16.0
google-auth-oauthlib         0.4.6
google-pasta                 0.2.0
greenlet                     2.0.1
grpcio                       1.51.1
gym                          0.21.0
h11                          0.14.0
h5netcdf                     1.1.0
h5py                         3.7.0
html5lib                     1.1
httptools                    0.6.0
huggingface-hub              0.16.4
humanfriendly                10.0
idna                         3.4
importlib-metadata           4.13.0
importlib-resources          5.12.0
iniconfig                    2.0.0
install                      1.3.5
InstructorEmbedding          1.0.1
ipykernel                    6.22.0
ipython                      8.13.2
ipython-genutils             0.2.0
ipywidgets                   8.0.6
isoduration                  20.11.0
itsdangerous                 2.1.2
jedi                         0.18.2
Jinja2                       3.1.2
joblib                       1.2.0
jsonpointer                  2.3
jsonschema                   4.17.3
jupyter                      1.0.0
jupyter_client               8.2.0
jupyter-console              6.6.3
jupyter_core                 5.3.0
jupyter-events               0.6.3
jupyter_server               2.5.0
jupyter_server_terminals     0.4.4
jupyterlab-pygments          0.2.2
jupyterlab-widgets           3.0.7
keras                        2.11.0
keyboard                     0.13.5
kiwisolver                   1.4.4
langchain                    0.0.247
langsmith                    0.0.15
Levenshtein                  0.21.0
libclang                     15.0.6.1
llama-cpp-python             0.1.77
lxml                         4.9.2
Markdown                     3.4.1
MarkupSafe                   2.1.2
marshmallow                  3.20.1
matplotlib                   3.6.2
matplotlib-inline            0.1.6
matplotlib2tikz              0.7.6
mistune                      2.0.5
modbus-tk                    1.1.2
monotonic                    1.6
mpmath                       1.3.0
multidict                    6.0.4
multiprocess                 0.70.15
mypy-extensions              1.0.0
nbclassic                    1.0.0
nbclient                     0.7.4
nbconvert                    7.3.1
nbformat                     5.8.0
nest-asyncio                 1.5.6
netCDF4                      1.6.3
nltk                         3.8.1
notebook                     6.5.4
notebook_shim                0.2.3
num2words                    0.5.12
numexpr                      2.8.4
numpy                        1.25.1
oauthlib                     3.2.2
onnxruntime                  1.15.1
openai                       0.27.8
openapi-schema-pydantic      1.2.4
opencv-python                4.6.0.66
openleadr                    0.5.28
openpyxl                     3.1.2
openvino                     2022.3.0
opt-einsum                   3.3.0
osqp                         0.6.2.post8
overrides                    7.3.1
packaging                    23.1
paho-mqtt                    1.6.1
pandas                       1.5.0
pandocfilters                1.5.0
parso                        0.8.3
pathspec                     0.11.1
patsy                        0.5.3
pdf2image                    1.16.3
pdfkit                       1.0.0
pdfminer.six                 20221105
pdfminer3k                   1.3.4
pdfplumber                   0.10.1
pefile                       2022.5.30
pickleshare                  0.7.5
Pillow                       9.3.0
pip                          23.2.1
platformdirs                 3.5.0
pluggy                       1.0.0
ply                          3.11
posthog                      3.0.1
prometheus-client            0.16.0
prompt-toolkit               3.0.38
protobuf                     4.23.4
psutil                       5.9.5
PuLP                         2.7.0
pulsar-client                3.2.0
pump                         0.1.2
pure-eval                    0.2.2
py-cpuinfo                   9.0.0
pyarrow                      10.0.1
pyasn1                       0.4.8
pyasn1-modules               0.2.8
pycausalimpact               0.1.1
pycocotools                  2.0.6
pycodestyle                  2.9.1
pycparser                    2.21
pydantic                     1.10.2
pydeck                       0.8.0
pydyf                        0.6.0
pygame                       2.4.0
pyglet                       1.5.27
Pygments                     2.13.0
pyinstaller                  5.6.2
pyinstaller-hooks-contrib    2022.13
pymc3                        3.11.5
pymodbus                     3.0.2
Pympler                      1.0.1
pyOpenSSL                    23.0.0
pyparsing                    3.0.9
PyPDF2                       3.0.1
pypdfium2                    4.18.0
pyphen                       0.14.0
PyPika                       0.48.9
pyproj                       3.6.0
PyQt5                        5.15.7
PyQt5-Qt5                    5.15.2
PyQt5-sip                    12.11.0
pyreadline3                  3.4.1
pyrsistent                   0.19.2
pyserial                     3.5
pysparklines                 1.4
pytesseract                  0.3.10
pytest                       7.2.1
python-dateutil              2.8.2
python-docx                  0.8.11
python-dotenv                1.0.0
python-json-logger           2.0.7
python-Levenshtein           0.21.0
python-statemachine          2.0.0
pytz                         2022.7.1
pytz-deprecation-shim        0.1.0.post0
pywin32                      306
pywin32-ctypes               0.2.0
pywinpty                     2.0.10
PyYAML                       6.0
pyzmq                        25.0.2
qdldl                        0.1.5.post3
qtconsole                    5.4.2
QtPy                         2.3.1
rapidfuzz                    3.0.0
regex                        2023.5.5
requests                     2.31.0
requests-oauthlib            1.3.1
rfc3339-validator            0.1.4
rfc3986-validator            0.1.1
rich                         12.6.0
rsa                          4.9
safetensors                  0.3.1
scapy                        2.4.5
scikit-learn                 1.2.0
scipy                        1.7.3
scs                          3.2.2
seaborn                      0.12.2
semver                       2.13.0
Send2Trash                   1.8.2
sentence-transformers        0.4.1.2
sentencepiece                0.1.99
setuptools                   66.1.1
shapely                      2.0.1
signxml                      2.10.1
sip                          6.7.4
six                          1.16.0
sklearn                      0.0.post1
slate3k                      0.5.3
smart-open                   6.3.0
smmap                        5.0.0
sniffio                      1.3.0
soupsieve                    2.4.1
SQLAlchemy                   1.4.45
stable-baselines3            1.8.0
stack-data                   0.6.2
starlette                    0.27.0
statsmodels                  0.14.0
streamlit                    1.25.0
swig                         4.1.1
sympy                        1.11.1
tenacity                     8.2.2
tensorboard                  2.11.2
tensorboard-data-server      0.6.1
tensorboard-plugin-wit       1.8.1
tensorflow                   2.11.0
tensorflow-estimator         2.11.0
tensorflow-intel             2.11.0
tensorflow-io-gcs-filesystem 0.30.0
termcolor                    2.2.0
terminado                    0.17.1
Theano-PyMC                  1.1.2
threadpoolctl                3.1.0
tikzplotlib                  0.10.1
tinycss2                     1.2.1
tokenizers                   0.13.3
toml                         0.10.2
tomli                        2.0.1
toolz                        0.12.0
torch                        1.13.0
torchvision                  0.14.0
tornado                      6.2
tqdm                         4.65.0
traitlets                    5.9.0
transformers                 4.31.0
typing_extensions            4.7.1
typing-inspect               0.9.0
tzdata                       2022.7
tzlocal                      4.2
UliEngineering               0.3.10
uri-template                 1.2.0
urllib3                      1.26.12
uvicorn                      0.22.0
validators                   0.20.0
watchdog                     2.2.0
watchfiles                   0.19.0
wcwidth                      0.2.6
weasyprint                   58.1
webcolors                    1.13
webencodings                 0.5.1
websocket-client             1.5.1
websockets                   11.0.3
Werkzeug                     2.2.2
wheel                        0.38.4
widgetsnbextension           4.0.7
wrapt                        1.14.1
WTForms                      3.0.1
xarray                       2023.5.0
xarray-einstats              0.5.1
xgboost                      1.7.6
XlsxWriter                   3.1.0
xlwings                      0.30.4
xmltodict                    0.13.0
xxhash                       3.3.0
yarl                         1.8.2
zipp                         3.11.0
zopfli                       0.2.2
talhaanwarch commented 1 year ago

are you using conda ?

bbartling commented 1 year ago

No not using Conda. Just ordinary or "vanilla" python.

> py -3.10
Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug  1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>

I most likely just installed the .exe executable from and added PATH during the installation:

I think this one: `

Windows installer (64-bit) | Windows | Recommended | 8f46453e68ef38e5544a76d84df3994c | 28916488 | SIG -- | -- | -- | -- | -- | --

Windows installer (64-bit) Windows Recommended 8f46453e68ef38e5544a76d84df3994c 28916488 SIG`

andreapagliacci commented 1 year ago

I have the same issue in Mac OS X, with python 3.10, ctransformers 0.2.2.4 (the last) using Conda

Traceback (most recent call last): File "/Volumes/OrientalPhase/Programmazione/cheshire-cat-ai/llm-api/main.py", line 17, in llm = AutoModelForCausalLM.from_pretrained('/Volumes/OrientalPhase/Programmazione/cheshire-cat-ai/llm-api/model/llama-2-7b-chat.ggmlv3.q2_K.bin', model_type='llama') File "/Volumes/OrientalPhase/Programmazione/cheshire-cat-ai/llm-api/.conda/lib/python3.10/site-packages/ctransformers/hub.py", line 173, in from_pretrained return LLM( File "/Volumes/OrientalPhase/Programmazione/cheshire-cat-ai/llm-api/.conda/lib/python3.10/site-packages/ctransformers/llm.py", line 238, in init self._lib = load_library(lib, cuda=config.gpu_layers > 0) File "/Volumes/OrientalPhase/Programmazione/cheshire-cat-ai/llm-api/.conda/lib/python3.10/site-packages/ctransformers/llm.py", line 125, in load_library lib = CDLL(path) File "/Volumes/OrientalPhase/Programmazione/cheshire-cat-ai/llm-api/.conda/lib/python3.10/ctypes/init.py", line 374, in init self._handle = _dlopen(self._name, mode) OSError: dlopen(/Volumes/OrientalPhase/Programmazione/cheshire-cat-ai/llm-api/.conda/lib/python3.10/site-packages/ctransformers/lib/avx2/libctransformers.dylib, 6): no suitable image found. Did find: /Volumes/OrientalPhase/Programmazione/cheshire-cat-ai/llm-api/.conda/lib/python3.10/site-packages/ctransformers/lib/avx2/libctransformers.dylib: cannot load 'libctransformers.dylib' (load command 0x80000034 is unknown) /Volumes/OrientalPhase/Programmazione/cheshire-cat-ai/llm-api/.conda/lib/python3.10/site-packages/ctransformers/lib/avx2/libctransformers.dylib: cannot load 'libctransformers.dylib' (load command 0x80000034 is unknown)