arcee-ai / fastmlx

FastMLX is a high performance production ready API to host MLX models.
Other
208 stars 21 forks source link

No chat template specified for llava models error #6

Open stewartugelow opened 3 months ago

stewartugelow commented 3 months ago

Getting this message:

File "/anaconda3/lib/python3.11/site-packages/transformers/processing_utils.py", line 926, in apply_chat_template raise ValueError( ValueError: No chat template is set for this processor. Please either set the chat_template attribute, or provide a chat template as an argument.

Happened with:

models--mlx-community--llava-1.5-7b-4bit models--mlx-community--llava-llama-3-8b-v1_1-8bit

Blaizzy commented 3 months ago

Don't worry, it shouldn't affect outputs.

This warning appears because we are using the default chat_template since these models don't have the chat_template key in the tokenizer_config.json

I will patch them today to stop the warning :)

Blaizzy commented 3 months ago

But to be sure,

could you share the full trace back and reproducible example?

stewartugelow commented 3 months ago

The code:

import requests
import json

url = "http://localhost:8000/v1/chat/completions"
headers = {"Content-Type": "application/json"}
data = {
    "model": "mlx-community/llava-1.5-7b-4bit",
    "image": "http://images.cocodataset.org/val2017/000000039769.jpg",
    "messages": [{"role": "user", "content": "What are these"}],
    "max_tokens": 100
}

response = requests.post(url, headers=headers, data=json.dumps(data))
print(response.json())

Traceback from the request:

Traceback (most recent call last):
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/requests/models.py", line 971, in json
    return complexjson.loads(self.text, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stewart/anaconda3/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stewart/anaconda3/lib/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stewart/anaconda3/lib/python3.11/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/stewart/Dropbox/dev/temp-fastmlx/llava.py", line 14, in <module>
    print(response.json())
          ^^^^^^^^^^^^^^^
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/requests/models.py", line 975, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Trace back from the server:

INFO:     127.0.0.1:50193 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 426, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/middleware/cors.py", line 93, in __call__
    await self.simple_response(scope, receive, send, request_headers=headers)
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/middleware/cors.py", line 148, in simple_response
    await self.app(scope, receive, send)
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/routing.py", line 756, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/routing.py", line 776, in app
    await route.handle(scope, receive, send)
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
    await self.app(scope, receive, send)
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/routing.py", line 72, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/fastmlx/fastmlx.py", line 109, in chat_completion
    prompt = processor.apply_chat_template(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/transformers/processing_utils.py", line 926, in apply_chat_template
    raise ValueError(
ValueError: No chat template is set for this processor. Please either set the `chat_template` attribute, or provide a chat template as an argument. See https://huggingface.co/docs/transformers/main/en/chat_templating for more information.
Blaizzy commented 3 months ago

I ran the same script you ran with all version but couldn't manage to replicate your issue:

You should only see a warning like this: No chat template is set for this tokenizer, falling back to a default class-level template. This is very error-prone, because models are often trained with templates different from the class default! Default chat templates are a legacy feature and will be removed in Transformers v4.43, at which point any code depending on them will stop working. We recommend setting a valid chat template before then to ensure that this model continues working without issues.

Responses:

v0.0.1

{
    'id': 'chatcmpl-d55bb013',
    'object': 'chat.completion',
    'created': 1720734350,
    'model': 'mlx-community/llava-1.5-7b-4bit',
    'choices': [
        {
            'index': 0,
            'message': {
                'role': 'assistant',
                'content': '</s>\nTwo cats are sleeping on a couch.'
            }, 'finish_reason': 'stop'
        }
    ]
}

v0.1.0

{
    'id': 'chatcmpl-5fd3cde5',
    'object': 'chat.completion',
    'created': 1720734521,
    'model': 'mlx-community/llava-1.5-7b-4bit',
    'choices': [
        {
            'index': 0,
            'message': {
                'role': 'assistant',
                'content': '</s>\nTwo cats are laying on a couch, one on the left side and the other on the right side.'
            }, 'finish_reason': 'stop'
        }
    ]
}
Blaizzy commented 3 months ago

Can you please share the output of pip list?

stewartugelow commented 3 months ago
Package                       Version
----------------------------- ------------
aiobotocore                   2.5.0
aiofiles                      22.1.0
aiohttp                       3.8.5
aioitertools                  0.7.1
aiosignal                     1.2.0
aiosqlite                     0.18.0
alabaster                     0.7.12
altair                        5.1.2
anaconda-anon-usage           0.4.2
anaconda-catalogs             0.2.0
anaconda-client               1.12.1
anaconda-cloud-auth           0.1.3
anaconda-navigator            2.5.0
anaconda-project              0.11.1
annotated-types               0.7.0
anyio                         3.5.0
appdirs                       1.4.4
applaunchservices             0.3.0
appnope                       0.1.2
appscript                     1.1.2
argon2-cffi                   21.3.0
argon2-cffi-bindings          21.2.0
arrow                         1.2.3
assemblyai                    0.20.0
astroid                       2.14.2
astropy                       5.1
asttokens                     2.0.5
async-timeout                 4.0.2
asyncio                       3.4.3
atomicwrites                  1.4.0
attrs                         22.1.0
audioread                     3.0.1
Automat                       20.2.0
autopep8                      1.6.0
av                            10.0.0
Babel                         2.11.0
backcall                      0.2.0
backports.functools-lru-cache 1.6.4
backports.tempfile            1.0
backports.weakref             1.0.post1
bcrypt                        3.2.0
beautifulsoup4                4.12.2
binaryornot                   0.4.4
black                         0.0
bleach                        4.1.0
blinker                       1.6.2
blis                          0.7.11
blosc2                        2.7.0
bokeh                         3.2.1
boltons                       23.0.0
botocore                      1.29.76
Bottleneck                    1.3.5
brotlipy                      0.7.0
cachetools                    5.3.1
catalogue                     2.0.10
certifi                       2023.7.22
cffi                          1.15.1
chardet                       4.0.0
charset-normalizer            2.0.4
chunkipy                      0.0.3
click                         8.1.7
cloudpathlib                  0.18.1
cloudpickle                   2.2.1
clyent                        1.2.2
colorama                      0.4.6
colorcet                      3.0.1
coloredlogs                   15.0.1
comm                          0.1.2
conda                         23.7.4
conda-build                   3.26.1
conda-content-trust           0.2.0
conda_index                   0.3.0
conda-libmamba-solver         23.7.0
conda-pack                    0.6.0
conda-package-handling        2.2.0
conda_package_streaming       0.9.0
conda-repo-cli                1.0.75
conda-token                   0.4.0
conda-verify                  3.4.2
confection                    0.1.5
constantly                    15.1.0
contourpy                     1.0.5
cookiecutter                  1.7.3
coverage                      7.3.2
cryptography                  41.0.3
cssselect                     1.1.0
ctranslate2                   3.22.0
cycler                        0.11.0
cymem                         2.0.8
Cython                        3.0.10
cytoolz                       0.12.0
dask                          2023.6.0
dataclasses-json              0.6.1
datasets                      2.12.0
datashader                    0.15.2
datashape                     0.5.4
debugpy                       1.6.7
decorator                     5.1.1
deepgram-sdk                  2.11.0
defusedxml                    0.7.1
diff-match-patch              20200713
dill                          0.3.6
distributed                   2023.6.0
dnspython                     2.6.1
docopt                        0.6.2
docstring-to-markdown         0.11
docutils                      0.18.1
email_validator               2.2.0
emoji                         2.8.0
en-core-web-sm                3.7.1
entrypoints                   0.4
et-xmlfile                    1.1.0
executing                     0.8.3
fastapi                       0.111.0
fastapi-cli                   0.0.4
faster-whisper                0.9.0
fastjsonschema                2.16.2
fastmlx                       0.1.0
ffmpeg-python                 0.2.0
ffmpy                         0.3.2
filelock                      3.9.0
flake8                        6.0.0
Flask                         3.0.3
flatbuffers                   23.5.26
fonttools                     4.25.0
frozenlist                    1.3.3
fsspec                        2024.6.1
fst-pso                       1.8.1
future                        0.18.3
FuzzyTM                       2.0.9
gensim                        4.3.0
gitdb                         4.0.10
GitPython                     3.1.37
glob2                         0.7
gmpy2                         2.1.2
gradio                        4.37.2
gradio_client                 1.0.2
greenlet                      2.0.1
h11                           0.14.0
h5py                          3.9.0
HeapDict                      1.0.1
holoviews                     1.17.1
httpcore                      1.0.2
httptools                     0.6.1
httpx                         0.25.2
huggingface-hub               0.23.4
humanfriendly                 10.0
hvplot                        0.8.4
hyperlink                     21.0.0
idna                          3.4
imagecodecs                   2023.1.23
imageio                       2.31.1
imagesize                     1.4.1
imbalanced-learn              0.10.1
importlib-metadata            6.0.0
importlib_resources           6.4.0
incremental                   21.3.0
inflection                    0.5.1
iniconfig                     1.1.1
intake                        0.6.8
intervaltree                  3.1.0
ipykernel                     6.25.0
ipython                       8.15.0
ipython-genutils              0.2.0
ipywidgets                    8.0.4
isort                         5.9.3
itemadapter                   0.3.0
itemloaders                   1.0.4
itsdangerous                  2.2.0
jaraco.classes                3.2.1
jedi                          0.18.1
jellyfish                     1.0.1
Jinja2                        3.1.2
jinja2-time                   0.2.0
jmespath                      0.10.0
joblib                        1.2.0
json5                         0.9.6
jsonpatch                     1.33
jsonpointer                   2.1
jsonschema                    4.17.3
jupyter                       1.0.0
jupyter_client                7.4.9
jupyter-console               6.6.3
jupyter_core                  5.3.0
jupyter-events                0.6.3
jupyter-server                1.23.4
jupyter_server_fileid         0.9.0
jupyter_server_ydoc           0.8.0
jupyter-ydoc                  0.2.4
jupyterlab                    3.6.3
jupyterlab-pygments           0.1.2
jupyterlab_server             2.22.0
jupyterlab-widgets            3.0.5
kaleido                       0.2.1
keyring                       23.13.1
kiwisolver                    1.4.4
langchain                     0.0.310
langcodes                     3.4.0
langdetect                    1.0.9
langsmith                     0.0.43
language_data                 1.2.0
lazy_loader                   0.2
lazy-object-proxy             1.6.0
libarchive-c                  2.9
libmambapy                    1.5.1
librosa                       0.10.2.post1
linkify-it-py                 2.0.0
llvmlite                      0.40.0
lmdb                          1.4.1
locket                        1.0.0
lxml                          4.9.3
lz4                           4.3.2
marisa-trie                   1.2.0
Markdown                      3.4.1
markdown-it-py                2.2.0
MarkupSafe                    2.1.1
marshmallow                   3.20.1
matplotlib                    3.9.1
matplotlib-inline             0.1.6
mccabe                        0.7.0
mdit-py-plugins               0.3.0
mdurl                         0.1.0
miniful                       0.0.6
mistune                       0.8.4
mlx                           0.15.2
mlx-lm                        0.15.2
mlx-vlm                       0.0.11
more-itertools                8.12.0
mpmath                        1.3.0
msgpack                       1.0.3
multidict                     6.0.2
multipledispatch              0.6.0
multiprocess                  0.70.14
munkres                       1.1.4
murmurhash                    1.0.10
mypy-extensions               1.0.0
navigator-updater             0.4.0
nbclassic                     0.5.5
nbclient                      0.5.13
nbconvert                     6.5.4
nbformat                      5.9.2
ndindex                       1.8
neo4j                         5.21.0
neo4j-rust-ext                5.21.0.0
nest-asyncio                  1.5.6
networkx                      3.1
nltk                          3.8.1
noisereduce                   3.0.2
notebook                      6.5.4
notebook_shim                 0.2.2
num2words                     0.5.13
numba                         0.57.1
numexpr                       2.8.4
numpy                         1.24.4
numpydoc                      1.5.0
onnxruntime                   1.16.3
openai                        0.28.1
openai-whisper                20231117
openpyxl                      3.0.10
optimum                       1.15.0.dev0
orjson                        3.10.6
packaging                     23.1
pandas                        1.5.3
pandocfilters                 1.5.0
panel                         1.2.3
param                         1.13.0
parsel                        1.6.0
parso                         0.8.3
partd                         1.4.0
pathlib                       1.0.1
pathspec                      0.10.3
patsy                         0.5.3
pep8                          1.7.1
pexpect                       4.8.0
phi-3-vision-mlx              0.0.2
pickleshare                   0.7.5
Pillow                        9.4.0
pip                           24.1.2
pkce                          1.0.3
pkginfo                       1.9.6
platformdirs                  3.10.0
plotly                        5.9.0
pluggy                        1.0.0
ply                           3.11
pooch                         1.8.2
poyo                          0.5.0
preshed                       3.0.9
prometheus-client             0.14.1
prompt-toolkit                3.0.36
Protego                       0.1.16
protobuf                      4.24.4
psutil                        5.9.0
ptyprocess                    0.7.0
pure-eval                     0.2.2
py-cpuinfo                    8.0.0
pyarrow                       11.0.0
pyasn1                        0.4.8
pyasn1-modules                0.2.8
pycodestyle                   2.10.0
pycosat                       0.6.4
pycparser                     2.21
pyct                          0.5.0
pycurl                        7.45.2
pydantic                      2.8.2
pydantic_core                 2.20.1
pydeck                        0.8.1b0
PyDispatcher                  2.0.5
pydocstyle                    6.3.0
pydub                         0.25.1
pyerfa                        2.0.0
pyflakes                      3.0.1
pyFUME                        0.3.4
Pygments                      2.15.1
PyJWT                         2.4.0
pylint                        2.16.2
pylint-venv                   2.3.0
pyls-spyder                   0.4.0
pyobjc-core                   9.0
pyobjc-framework-Cocoa        9.0
pyobjc-framework-CoreServices 9.0
pyobjc-framework-FSEvents     9.0
pyodbc                        4.0.34
pyOpenSSL                     23.2.0
pyparsing                     3.0.9
PyQt5-sip                     12.11.0
pyrsistent                    0.18.0
PySocks                       1.7.1
pytest                        7.4.0
pytest-cov                    4.1.0
python-dateutil               2.8.2
python-dotenv                 0.21.0
python-json-logger            2.0.7
python-lsp-black              1.2.1
python-lsp-jsonrpc            1.0.0
python-lsp-server             1.7.2
python-multipart              0.0.9
python-slugify                5.0.2
python-snappy                 0.6.1
pytoolconfig                  1.2.5
pytube                        15.0.0
pytz                          2023.3.post1
pyviz-comms                   2.3.0
PyWavelets                    1.4.1
PyYAML                        6.0
pyzmq                         23.2.0
QDarkStyle                    3.0.2
qstylizer                     0.2.2
QtAwesome                     1.2.2
qtconsole                     5.4.2
QtPy                          2.2.0
queuelib                      1.5.0
regex                         2022.7.9
requests                      2.31.0
requests-file                 1.5.1
requests-toolbelt             1.0.0
responses                     0.13.3
rfc3339-validator             0.1.4
rfc3986-validator             0.1.1
rich                          13.6.0
rope                          1.7.0
Rtree                         1.0.1
ruamel.yaml                   0.17.21
ruamel-yaml-conda             0.17.21
ruff                          0.5.1
s3fs                          2023.4.0
safetensors                   0.4.3
scikit-image                  0.20.0
scikit-learn                  1.3.0
scipy                         1.13.1
Scrapy                        2.8.0
seaborn                       0.12.2
semantic-version              2.10.0
Send2Trash                    1.8.0
sentencepiece                 0.1.99
service-identity              18.1.0
setuptools                    68.0.0
shellingham                   1.5.4
simpful                       2.12.0
sip                           6.6.2
six                           1.16.0
smart-open                    5.2.1
smmap                         5.0.1
sniffio                       1.2.0
snowballstemmer               2.2.0
sortedcontainers              2.4.0
sounddevice                   0.4.6
soundfile                     0.12.1
soupsieve                     2.4
soxr                          0.3.7
spacy                         3.7.5
spacy-legacy                  3.0.12
spacy-loggers                 1.0.5
Sphinx                        5.0.2
sphinxcontrib-applehelp       1.0.2
sphinxcontrib-devhelp         1.0.2
sphinxcontrib-htmlhelp        2.0.0
sphinxcontrib-jsmath          1.0.1
sphinxcontrib-qthelp          1.0.3
sphinxcontrib-serializinghtml 1.1.5
spyder                        5.4.3
spyder-kernels                2.4.4
SQLAlchemy                    1.4.39
srsly                         2.4.8
stack-data                    0.2.0
stanza                        1.4.2
starlette                     0.37.2
statsmodels                   0.14.0
streamlit                     1.36.0
sympy                         1.11.1
tables                        3.9.2
tabulate                      0.8.10
tblib                         1.7.0
tenacity                      8.2.2
terminado                     0.17.1
text-unidecode                1.3
textdistance                  4.2.1
thinc                         8.2.5
threadpoolctl                 2.2.0
three-merge                   0.1.1
tifffile                      2023.4.12
tiktoken                      0.5.1
tinycss2                      1.2.1
tldextract                    3.2.0
tokenizers                    0.19.1
toml                          0.10.2
tomlkit                       0.12.0
toolz                         0.12.0
torch                         2.1.0
tornado                       6.3.2
tqdm                          4.65.0
traitlets                     5.7.1
transformers                  4.42.3
Twisted                       22.10.0
typer                         0.12.3
typing_extensions             4.12.2
typing-inspect                0.9.0
tzdata                        2023.3
tzlocal                       5.1
uc-micro-py                   1.0.1
ujson                         5.4.0
Unidecode                     1.2.0
urllib3                       2.2.2
uvicorn                       0.23.2
uvloop                        0.19.0
validators                    0.22.0
w3lib                         1.21.0
wasabi                        1.1.3
watchdog                      2.1.6
watchfiles                    0.22.0
wcwidth                       0.2.5
weasel                        0.4.1
webencodings                  0.5.1
websocket-client              0.58.0
websockets                    11.0.3
Werkzeug                      3.0.3
whatthepatch                  1.0.2
wheel                         0.38.4
whisper-ctranslate2           0.3.2
widgetsnbextension            4.0.5
wrapt                         1.14.1
wurlitzer                     3.0.2
xarray                        2023.6.0
xlwings                       0.29.1
xxhash                        2.0.2
xyzservices                   2022.9.0
y-py                          0.5.9
yapf                          0.31.0
yarl                          1.8.1
ypy-websocket                 0.8.2
zict                          2.2.0
zipp                          3.11.0
zope.interface                5.4.0
zstandard                     0.19.0
Blaizzy commented 3 months ago

I haven't been able to replicate this issue. I ran the script on 2 different machines and it runs normally.

Could you create a new virtual environment and try again ?

stewartugelow commented 3 months ago

Tried again in a new vent. Same result. This is python 3.11 on an M1 MacBook Pro with 16 GB ram.

Request:

  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/requests/models.py", line 971, in json
    return complexjson.loads(self.text, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stewart/anaconda3/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stewart/anaconda3/lib/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stewart/anaconda3/lib/python3.11/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/stewart/Dropbox/dev/vmlx/llava-panda.py", line 14, in <module>
    print(response.json())
          ^^^^^^^^^^^^^^^
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/requests/models.py", line 975, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Server:

INFO:     127.0.0.1:53180 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/routing.py", line 756, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/routing.py", line 776, in app
    await route.handle(scope, receive, send)
  File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
    await self.app(scope, receive, send)
  File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/routing.py", line 72, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/fastmlx/fastmlx.py", line 135, in chat_completion
    prompt = processor.apply_chat_template(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/transformers/processing_utils.py", line 926, in apply_chat_template
    raise ValueError(
ValueError: No chat template is set for this processor. Please either set the `chat_template` attribute, or provide a chat template as an argument. See https://huggingface.co/docs/transformers/main/en/chat_templating for more information.

PIP List:

Package                   Version
------------------------- -----------
aiofiles                  23.2.1
altair                    5.3.0
annotated-types           0.7.0
anyio                     4.4.0
attrs                     23.2.0
certifi                   2024.7.4
charset-normalizer        3.3.2
click                     8.1.7
contourpy                 1.2.1
cycler                    0.12.1
dnspython                 2.6.1
email_validator           2.2.0
fastapi                   0.111.0
fastapi-cli               0.0.4
fastmlx                   0.1.0
ffmpy                     0.3.2
filelock                  3.15.4
fonttools                 4.53.1
fsspec                    2024.6.1
gradio                    4.37.2
gradio_client             1.0.2
h11                       0.14.0
httpcore                  1.0.5
httptools                 0.6.1
httpx                     0.27.0
huggingface-hub           0.23.4
idna                      3.7
importlib_resources       6.4.0
Jinja2                    3.1.4
jsonschema                4.23.0
jsonschema-specifications 2023.12.1
kiwisolver                1.4.5
markdown-it-py            3.0.0
MarkupSafe                2.1.5
matplotlib                3.9.1
mdurl                     0.1.2
mlx                       0.16.0
mlx-lm                    0.15.2
mlx-vlm                   0.0.11
numpy                     1.26.4
orjson                    3.10.6
packaging                 24.1
pandas                    2.2.2
pillow                    10.4.0
pip                       23.2.1
protobuf                  5.27.2
pydantic                  2.8.2
pydantic_core             2.20.1
pydub                     0.25.1
Pygments                  2.18.0
pyparsing                 3.1.2
python-dateutil           2.9.0.post0
python-dotenv             1.0.1
python-multipart          0.0.9
pytz                      2024.1
PyYAML                    6.0.1
referencing               0.35.1
regex                     2024.5.15
requests                  2.32.3
rich                      13.7.1
rpds-py                   0.19.0
ruff                      0.5.1
safetensors               0.4.3
scipy                     1.13.1
semantic-version          2.10.0
setuptools                65.5.0
shellingham               1.5.4
six                       1.16.0
sniffio                   1.3.1
starlette                 0.37.2
tokenizers                0.19.1
tomlkit                   0.12.0
toolz                     0.12.1
tqdm                      4.66.4
transformers              4.42.4
typer                     0.12.3
typing_extensions         4.12.2
tzdata                    2024.1
ujson                     5.10.0
urllib3                   2.2.2
uvicorn                   0.30.1
uvloop                    0.19.0
watchfiles                0.22.0
websockets                11.0.3
Blaizzy commented 3 months ago

Could you delete and re-download this model: mlx-community/llava-1.5-7b-4bit?

stewartugelow commented 3 months ago

Done, but no luck.

Request

Traceback (most recent call last):
  File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/requests/models.py", line 974, in json
    return complexjson.loads(self.text, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stewart/anaconda3/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stewart/anaconda3/lib/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stewart/anaconda3/lib/python3.11/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/stewart/Dropbox/dev/vmlx/llava.py", line 14, in <module>
    print(response.json())
          ^^^^^^^^^^^^^^^
  File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/requests/models.py", line 978, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

SERVER:

500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/routing.py", line 756, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/routing.py", line 776, in app
    await route.handle(scope, receive, send)
  File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
    await self.app(scope, receive, send)
  File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/routing.py", line 72, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/fastmlx/fastmlx.py", line 135, in chat_completion
    prompt = processor.apply_chat_template(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/transformers/processing_utils.py", line 926, in apply_chat_template
    raise ValueError(
ValueError: No chat template is set for this processor. Please either set the `chat_template` attribute, or provide a chat template as an argument. See https://huggingface.co/docs/transformers/main/en/chat_templating for more information.

pip list

Package                   Version
------------------------- -----------
aiofiles                  23.2.1
altair                    5.3.0
annotated-types           0.7.0
anyio                     4.4.0
attrs                     23.2.0
certifi                   2024.7.4
charset-normalizer        3.3.2
click                     8.1.7
contourpy                 1.2.1
cycler                    0.12.1
dnspython                 2.6.1
email_validator           2.2.0
fastapi                   0.111.0
fastapi-cli               0.0.4
fastmlx                   0.1.0
ffmpy                     0.3.2
filelock                  3.15.4
fonttools                 4.53.1
fsspec                    2024.6.1
gradio                    4.37.2
gradio_client             1.0.2
h11                       0.14.0
httpcore                  1.0.5
httptools                 0.6.1
httpx                     0.27.0
huggingface-hub           0.23.4
idna                      3.7
importlib_resources       6.4.0
Jinja2                    3.1.4
jsonschema                4.23.0
jsonschema-specifications 2023.12.1
kiwisolver                1.4.5
markdown-it-py            3.0.0
MarkupSafe                2.1.5
matplotlib                3.9.1
mdurl                     0.1.2
mlx                       0.16.0
mlx-lm                    0.15.2
mlx-vlm                   0.0.11
numpy                     1.26.4
orjson                    3.10.6
packaging                 24.1
pandas                    2.2.2
pillow                    10.4.0
pip                       23.2.1
protobuf                  5.27.2
pydantic                  2.8.2
pydantic_core             2.20.1
pydub                     0.25.1
Pygments                  2.18.0
pyparsing                 3.1.2
python-dateutil           2.9.0.post0
python-dotenv             1.0.1
python-multipart          0.0.9
pytz                      2024.1
PyYAML                    6.0.1
referencing               0.35.1
regex                     2024.5.15
requests                  2.32.3
rich                      13.7.1
rpds-py                   0.19.0
ruff                      0.5.1
safetensors               0.4.3
scipy                     1.13.1
semantic-version          2.10.0
setuptools                65.5.0
shellingham               1.5.4
six                       1.16.0
sniffio                   1.3.1
starlette                 0.37.2
tokenizers                0.19.1
tomlkit                   0.12.0
toolz                     0.12.1
tqdm                      4.66.4
transformers              4.42.4
typer                     0.12.3
typing_extensions         4.12.2
tzdata                    2024.1
ujson                     5.10.0
urllib3                   2.2.2
uvicorn                   0.30.1
uvloop                    0.19.0
watchfiles                0.22.0
websockets                11.0.3
stewartugelow commented 3 months ago

Is there debugging I could add to the fastmlx.py file that would be helpful?

Blaizzy commented 3 months ago

Yes, there is.

You can print the processor and its attributes. And print the configuration.

Also, as a sanity you can comment out the chat template part and pass the formatted text directly.

Example:

USER: <image>\nWhat are these?\nASSISTANT:

I can share the code later. I'm currently travelling, and I'm on my phone.

stewartugelow commented 3 months ago

Upon further investigation, I can confirm that this is an mlx-vlm bug, not a fastmlx bug.

I get the same chat template errors with all of the following:

models--mlx-community--llava-1.5-7b-4bit models--mlx-community--llava-llama-3-8b-v1_1-8bit models--mlx-community--llava-phi-3-mini-4bit models--mlx-community--llava-v1.6-mistral-7b-8bit

Moving this to https://github.com/Blaizzy/mlx-vlm/issues/51