Open stewartugelow opened 2 months ago
Blaizzy
commented
2 months ago Don't worry, it shouldn't affect outputs.
This warning appears because we are using the default chat_template since these models don't have the chat_template key in the tokenizer_config.json
I will patch them today to stop the warning :)
Blaizzy
commented
2 months ago But to be sure,
could you share the full trace back and reproducible example?
stewartugelow
commented
2 months ago The code:
import requests
import json
url = "http://localhost:8000/v1/chat/completions"
headers = {"Content-Type": "application/json"}
data = {
"model": "mlx-community/llava-1.5-7b-4bit",
"image": "http://images.cocodataset.org/val2017/000000039769.jpg",
"messages": [{"role": "user", "content": "What are these"}],
"max_tokens": 100
}
response = requests.post(url, headers=headers, data=json.dumps(data))
print(response.json())Traceback from the request:
Traceback (most recent call last):
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/requests/models.py", line 971, in json
return complexjson.loads(self.text, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/stewart/anaconda3/lib/python3.11/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/stewart/anaconda3/lib/python3.11/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/stewart/anaconda3/lib/python3.11/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/stewart/Dropbox/dev/temp-fastmlx/llava.py", line 14, in <module>
print(response.json())
^^^^^^^^^^^^^^^
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/requests/models.py", line 975, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Trace back from the server:
INFO: 127.0.0.1:50193 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 426, in run_asgi
result = await app( # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
await super().__call__(scope, receive, send)
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
await self.middleware_stack(scope, receive, send)
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
raise exc
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
await self.app(scope, receive, _send)
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/middleware/cors.py", line 93, in __call__
await self.simple_response(scope, receive, send, request_headers=headers)
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/middleware/cors.py", line 148, in simple_response
await self.app(scope, receive, send)
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
raise exc
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/routing.py", line 756, in __call__
await self.middleware_stack(scope, receive, send)
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/routing.py", line 776, in app
await route.handle(scope, receive, send)
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
await self.app(scope, receive, send)
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
raise exc
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/routing.py", line 72, in app
response = await func(request)
^^^^^^^^^^^^^^^^^^^
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app
raw_response = await run_endpoint_function(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
return await dependant.call(**values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/fastmlx/fastmlx.py", line 109, in chat_completion
prompt = processor.apply_chat_template(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/transformers/processing_utils.py", line 926, in apply_chat_template
raise ValueError(
ValueError: No chat template is set for this processor. Please either set the `chat_template` attribute, or provide a chat template as an argument. See https://huggingface.co/docs/transformers/main/en/chat_templating for more information.I ran the same script you ran with all version but couldn't manage to replicate your issue:
You should only see a warning like this:
No chat template is set for this tokenizer, falling back to a default class-level template. This is very error-prone, because models are often trained with templates different from the class default! Default chat templates are a legacy feature and will be removed in Transformers v4.43, at which point any code depending on them will stop working. We recommend setting a valid chat template before then to ensure that this model continues working without issues.
v0.0.1
{
'id': 'chatcmpl-d55bb013',
'object': 'chat.completion',
'created': 1720734350,
'model': 'mlx-community/llava-1.5-7b-4bit',
'choices': [
{
'index': 0,
'message': {
'role': 'assistant',
'content': '</s>\nTwo cats are sleeping on a couch.'
}, 'finish_reason': 'stop'
}
]
}v0.1.0
{
'id': 'chatcmpl-5fd3cde5',
'object': 'chat.completion',
'created': 1720734521,
'model': 'mlx-community/llava-1.5-7b-4bit',
'choices': [
{
'index': 0,
'message': {
'role': 'assistant',
'content': '</s>\nTwo cats are laying on a couch, one on the left side and the other on the right side.'
}, 'finish_reason': 'stop'
}
]
}Can you please share the output of pip list?
stewartugelow
commented
2 months ago Package Version
----------------------------- ------------
aiobotocore 2.5.0
aiofiles 22.1.0
aiohttp 3.8.5
aioitertools 0.7.1
aiosignal 1.2.0
aiosqlite 0.18.0
alabaster 0.7.12
altair 5.1.2
anaconda-anon-usage 0.4.2
anaconda-catalogs 0.2.0
anaconda-client 1.12.1
anaconda-cloud-auth 0.1.3
anaconda-navigator 2.5.0
anaconda-project 0.11.1
annotated-types 0.7.0
anyio 3.5.0
appdirs 1.4.4
applaunchservices 0.3.0
appnope 0.1.2
appscript 1.1.2
argon2-cffi 21.3.0
argon2-cffi-bindings 21.2.0
arrow 1.2.3
assemblyai 0.20.0
astroid 2.14.2
astropy 5.1
asttokens 2.0.5
async-timeout 4.0.2
asyncio 3.4.3
atomicwrites 1.4.0
attrs 22.1.0
audioread 3.0.1
Automat 20.2.0
autopep8 1.6.0
av 10.0.0
Babel 2.11.0
backcall 0.2.0
backports.functools-lru-cache 1.6.4
backports.tempfile 1.0
backports.weakref 1.0.post1
bcrypt 3.2.0
beautifulsoup4 4.12.2
binaryornot 0.4.4
black 0.0
bleach 4.1.0
blinker 1.6.2
blis 0.7.11
blosc2 2.7.0
bokeh 3.2.1
boltons 23.0.0
botocore 1.29.76
Bottleneck 1.3.5
brotlipy 0.7.0
cachetools 5.3.1
catalogue 2.0.10
certifi 2023.7.22
cffi 1.15.1
chardet 4.0.0
charset-normalizer 2.0.4
chunkipy 0.0.3
click 8.1.7
cloudpathlib 0.18.1
cloudpickle 2.2.1
clyent 1.2.2
colorama 0.4.6
colorcet 3.0.1
coloredlogs 15.0.1
comm 0.1.2
conda 23.7.4
conda-build 3.26.1
conda-content-trust 0.2.0
conda_index 0.3.0
conda-libmamba-solver 23.7.0
conda-pack 0.6.0
conda-package-handling 2.2.0
conda_package_streaming 0.9.0
conda-repo-cli 1.0.75
conda-token 0.4.0
conda-verify 3.4.2
confection 0.1.5
constantly 15.1.0
contourpy 1.0.5
cookiecutter 1.7.3
coverage 7.3.2
cryptography 41.0.3
cssselect 1.1.0
ctranslate2 3.22.0
cycler 0.11.0
cymem 2.0.8
Cython 3.0.10
cytoolz 0.12.0
dask 2023.6.0
dataclasses-json 0.6.1
datasets 2.12.0
datashader 0.15.2
datashape 0.5.4
debugpy 1.6.7
decorator 5.1.1
deepgram-sdk 2.11.0
defusedxml 0.7.1
diff-match-patch 20200713
dill 0.3.6
distributed 2023.6.0
dnspython 2.6.1
docopt 0.6.2
docstring-to-markdown 0.11
docutils 0.18.1
email_validator 2.2.0
emoji 2.8.0
en-core-web-sm 3.7.1
entrypoints 0.4
et-xmlfile 1.1.0
executing 0.8.3
fastapi 0.111.0
fastapi-cli 0.0.4
faster-whisper 0.9.0
fastjsonschema 2.16.2
fastmlx 0.1.0
ffmpeg-python 0.2.0
ffmpy 0.3.2
filelock 3.9.0
flake8 6.0.0
Flask 3.0.3
flatbuffers 23.5.26
fonttools 4.25.0
frozenlist 1.3.3
fsspec 2024.6.1
fst-pso 1.8.1
future 0.18.3
FuzzyTM 2.0.9
gensim 4.3.0
gitdb 4.0.10
GitPython 3.1.37
glob2 0.7
gmpy2 2.1.2
gradio 4.37.2
gradio_client 1.0.2
greenlet 2.0.1
h11 0.14.0
h5py 3.9.0
HeapDict 1.0.1
holoviews 1.17.1
httpcore 1.0.2
httptools 0.6.1
httpx 0.25.2
huggingface-hub 0.23.4
humanfriendly 10.0
hvplot 0.8.4
hyperlink 21.0.0
idna 3.4
imagecodecs 2023.1.23
imageio 2.31.1
imagesize 1.4.1
imbalanced-learn 0.10.1
importlib-metadata 6.0.0
importlib_resources 6.4.0
incremental 21.3.0
inflection 0.5.1
iniconfig 1.1.1
intake 0.6.8
intervaltree 3.1.0
ipykernel 6.25.0
ipython 8.15.0
ipython-genutils 0.2.0
ipywidgets 8.0.4
isort 5.9.3
itemadapter 0.3.0
itemloaders 1.0.4
itsdangerous 2.2.0
jaraco.classes 3.2.1
jedi 0.18.1
jellyfish 1.0.1
Jinja2 3.1.2
jinja2-time 0.2.0
jmespath 0.10.0
joblib 1.2.0
json5 0.9.6
jsonpatch 1.33
jsonpointer 2.1
jsonschema 4.17.3
jupyter 1.0.0
jupyter_client 7.4.9
jupyter-console 6.6.3
jupyter_core 5.3.0
jupyter-events 0.6.3
jupyter-server 1.23.4
jupyter_server_fileid 0.9.0
jupyter_server_ydoc 0.8.0
jupyter-ydoc 0.2.4
jupyterlab 3.6.3
jupyterlab-pygments 0.1.2
jupyterlab_server 2.22.0
jupyterlab-widgets 3.0.5
kaleido 0.2.1
keyring 23.13.1
kiwisolver 1.4.4
langchain 0.0.310
langcodes 3.4.0
langdetect 1.0.9
langsmith 0.0.43
language_data 1.2.0
lazy_loader 0.2
lazy-object-proxy 1.6.0
libarchive-c 2.9
libmambapy 1.5.1
librosa 0.10.2.post1
linkify-it-py 2.0.0
llvmlite 0.40.0
lmdb 1.4.1
locket 1.0.0
lxml 4.9.3
lz4 4.3.2
marisa-trie 1.2.0
Markdown 3.4.1
markdown-it-py 2.2.0
MarkupSafe 2.1.1
marshmallow 3.20.1
matplotlib 3.9.1
matplotlib-inline 0.1.6
mccabe 0.7.0
mdit-py-plugins 0.3.0
mdurl 0.1.0
miniful 0.0.6
mistune 0.8.4
mlx 0.15.2
mlx-lm 0.15.2
mlx-vlm 0.0.11
more-itertools 8.12.0
mpmath 1.3.0
msgpack 1.0.3
multidict 6.0.2
multipledispatch 0.6.0
multiprocess 0.70.14
munkres 1.1.4
murmurhash 1.0.10
mypy-extensions 1.0.0
navigator-updater 0.4.0
nbclassic 0.5.5
nbclient 0.5.13
nbconvert 6.5.4
nbformat 5.9.2
ndindex 1.8
neo4j 5.21.0
neo4j-rust-ext 5.21.0.0
nest-asyncio 1.5.6
networkx 3.1
nltk 3.8.1
noisereduce 3.0.2
notebook 6.5.4
notebook_shim 0.2.2
num2words 0.5.13
numba 0.57.1
numexpr 2.8.4
numpy 1.24.4
numpydoc 1.5.0
onnxruntime 1.16.3
openai 0.28.1
openai-whisper 20231117
openpyxl 3.0.10
optimum 1.15.0.dev0
orjson 3.10.6
packaging 23.1
pandas 1.5.3
pandocfilters 1.5.0
panel 1.2.3
param 1.13.0
parsel 1.6.0
parso 0.8.3
partd 1.4.0
pathlib 1.0.1
pathspec 0.10.3
patsy 0.5.3
pep8 1.7.1
pexpect 4.8.0
phi-3-vision-mlx 0.0.2
pickleshare 0.7.5
Pillow 9.4.0
pip 24.1.2
pkce 1.0.3
pkginfo 1.9.6
platformdirs 3.10.0
plotly 5.9.0
pluggy 1.0.0
ply 3.11
pooch 1.8.2
poyo 0.5.0
preshed 3.0.9
prometheus-client 0.14.1
prompt-toolkit 3.0.36
Protego 0.1.16
protobuf 4.24.4
psutil 5.9.0
ptyprocess 0.7.0
pure-eval 0.2.2
py-cpuinfo 8.0.0
pyarrow 11.0.0
pyasn1 0.4.8
pyasn1-modules 0.2.8
pycodestyle 2.10.0
pycosat 0.6.4
pycparser 2.21
pyct 0.5.0
pycurl 7.45.2
pydantic 2.8.2
pydantic_core 2.20.1
pydeck 0.8.1b0
PyDispatcher 2.0.5
pydocstyle 6.3.0
pydub 0.25.1
pyerfa 2.0.0
pyflakes 3.0.1
pyFUME 0.3.4
Pygments 2.15.1
PyJWT 2.4.0
pylint 2.16.2
pylint-venv 2.3.0
pyls-spyder 0.4.0
pyobjc-core 9.0
pyobjc-framework-Cocoa 9.0
pyobjc-framework-CoreServices 9.0
pyobjc-framework-FSEvents 9.0
pyodbc 4.0.34
pyOpenSSL 23.2.0
pyparsing 3.0.9
PyQt5-sip 12.11.0
pyrsistent 0.18.0
PySocks 1.7.1
pytest 7.4.0
pytest-cov 4.1.0
python-dateutil 2.8.2
python-dotenv 0.21.0
python-json-logger 2.0.7
python-lsp-black 1.2.1
python-lsp-jsonrpc 1.0.0
python-lsp-server 1.7.2
python-multipart 0.0.9
python-slugify 5.0.2
python-snappy 0.6.1
pytoolconfig 1.2.5
pytube 15.0.0
pytz 2023.3.post1
pyviz-comms 2.3.0
PyWavelets 1.4.1
PyYAML 6.0
pyzmq 23.2.0
QDarkStyle 3.0.2
qstylizer 0.2.2
QtAwesome 1.2.2
qtconsole 5.4.2
QtPy 2.2.0
queuelib 1.5.0
regex 2022.7.9
requests 2.31.0
requests-file 1.5.1
requests-toolbelt 1.0.0
responses 0.13.3
rfc3339-validator 0.1.4
rfc3986-validator 0.1.1
rich 13.6.0
rope 1.7.0
Rtree 1.0.1
ruamel.yaml 0.17.21
ruamel-yaml-conda 0.17.21
ruff 0.5.1
s3fs 2023.4.0
safetensors 0.4.3
scikit-image 0.20.0
scikit-learn 1.3.0
scipy 1.13.1
Scrapy 2.8.0
seaborn 0.12.2
semantic-version 2.10.0
Send2Trash 1.8.0
sentencepiece 0.1.99
service-identity 18.1.0
setuptools 68.0.0
shellingham 1.5.4
simpful 2.12.0
sip 6.6.2
six 1.16.0
smart-open 5.2.1
smmap 5.0.1
sniffio 1.2.0
snowballstemmer 2.2.0
sortedcontainers 2.4.0
sounddevice 0.4.6
soundfile 0.12.1
soupsieve 2.4
soxr 0.3.7
spacy 3.7.5
spacy-legacy 3.0.12
spacy-loggers 1.0.5
Sphinx 5.0.2
sphinxcontrib-applehelp 1.0.2
sphinxcontrib-devhelp 1.0.2
sphinxcontrib-htmlhelp 2.0.0
sphinxcontrib-jsmath 1.0.1
sphinxcontrib-qthelp 1.0.3
sphinxcontrib-serializinghtml 1.1.5
spyder 5.4.3
spyder-kernels 2.4.4
SQLAlchemy 1.4.39
srsly 2.4.8
stack-data 0.2.0
stanza 1.4.2
starlette 0.37.2
statsmodels 0.14.0
streamlit 1.36.0
sympy 1.11.1
tables 3.9.2
tabulate 0.8.10
tblib 1.7.0
tenacity 8.2.2
terminado 0.17.1
text-unidecode 1.3
textdistance 4.2.1
thinc 8.2.5
threadpoolctl 2.2.0
three-merge 0.1.1
tifffile 2023.4.12
tiktoken 0.5.1
tinycss2 1.2.1
tldextract 3.2.0
tokenizers 0.19.1
toml 0.10.2
tomlkit 0.12.0
toolz 0.12.0
torch 2.1.0
tornado 6.3.2
tqdm 4.65.0
traitlets 5.7.1
transformers 4.42.3
Twisted 22.10.0
typer 0.12.3
typing_extensions 4.12.2
typing-inspect 0.9.0
tzdata 2023.3
tzlocal 5.1
uc-micro-py 1.0.1
ujson 5.4.0
Unidecode 1.2.0
urllib3 2.2.2
uvicorn 0.23.2
uvloop 0.19.0
validators 0.22.0
w3lib 1.21.0
wasabi 1.1.3
watchdog 2.1.6
watchfiles 0.22.0
wcwidth 0.2.5
weasel 0.4.1
webencodings 0.5.1
websocket-client 0.58.0
websockets 11.0.3
Werkzeug 3.0.3
whatthepatch 1.0.2
wheel 0.38.4
whisper-ctranslate2 0.3.2
widgetsnbextension 4.0.5
wrapt 1.14.1
wurlitzer 3.0.2
xarray 2023.6.0
xlwings 0.29.1
xxhash 2.0.2
xyzservices 2022.9.0
y-py 0.5.9
yapf 0.31.0
yarl 1.8.1
ypy-websocket 0.8.2
zict 2.2.0
zipp 3.11.0
zope.interface 5.4.0
zstandard 0.19.0I haven't been able to replicate this issue. I ran the script on 2 different machines and it runs normally.
Could you create a new virtual environment and try again ?
stewartugelow
commented
2 months ago Tried again in a new vent. Same result. This is python 3.11 on an M1 MacBook Pro with 16 GB ram.
Request:
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/requests/models.py", line 971, in json
return complexjson.loads(self.text, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/stewart/anaconda3/lib/python3.11/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/stewart/anaconda3/lib/python3.11/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/stewart/anaconda3/lib/python3.11/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/stewart/Dropbox/dev/vmlx/llava-panda.py", line 14, in <module>
print(response.json())
^^^^^^^^^^^^^^^
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/requests/models.py", line 975, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)Server:
INFO: 127.0.0.1:53180 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
result = await app( # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
await super().__call__(scope, receive, send)
File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
await self.middleware_stack(scope, receive, send)
File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
raise exc
File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
await self.app(scope, receive, _send)
File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
raise exc
File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/routing.py", line 756, in __call__
await self.middleware_stack(scope, receive, send)
File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/routing.py", line 776, in app
await route.handle(scope, receive, send)
File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
await self.app(scope, receive, send)
File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
raise exc
File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/routing.py", line 72, in app
response = await func(request)
^^^^^^^^^^^^^^^^^^^
File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app
raw_response = await run_endpoint_function(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
return await dependant.call(**values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/fastmlx/fastmlx.py", line 135, in chat_completion
prompt = processor.apply_chat_template(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/transformers/processing_utils.py", line 926, in apply_chat_template
raise ValueError(
ValueError: No chat template is set for this processor. Please either set the `chat_template` attribute, or provide a chat template as an argument. See https://huggingface.co/docs/transformers/main/en/chat_templating for more information.PIP List:
Package Version
------------------------- -----------
aiofiles 23.2.1
altair 5.3.0
annotated-types 0.7.0
anyio 4.4.0
attrs 23.2.0
certifi 2024.7.4
charset-normalizer 3.3.2
click 8.1.7
contourpy 1.2.1
cycler 0.12.1
dnspython 2.6.1
email_validator 2.2.0
fastapi 0.111.0
fastapi-cli 0.0.4
fastmlx 0.1.0
ffmpy 0.3.2
filelock 3.15.4
fonttools 4.53.1
fsspec 2024.6.1
gradio 4.37.2
gradio_client 1.0.2
h11 0.14.0
httpcore 1.0.5
httptools 0.6.1
httpx 0.27.0
huggingface-hub 0.23.4
idna 3.7
importlib_resources 6.4.0
Jinja2 3.1.4
jsonschema 4.23.0
jsonschema-specifications 2023.12.1
kiwisolver 1.4.5
markdown-it-py 3.0.0
MarkupSafe 2.1.5
matplotlib 3.9.1
mdurl 0.1.2
mlx 0.16.0
mlx-lm 0.15.2
mlx-vlm 0.0.11
numpy 1.26.4
orjson 3.10.6
packaging 24.1
pandas 2.2.2
pillow 10.4.0
pip 23.2.1
protobuf 5.27.2
pydantic 2.8.2
pydantic_core 2.20.1
pydub 0.25.1
Pygments 2.18.0
pyparsing 3.1.2
python-dateutil 2.9.0.post0
python-dotenv 1.0.1
python-multipart 0.0.9
pytz 2024.1
PyYAML 6.0.1
referencing 0.35.1
regex 2024.5.15
requests 2.32.3
rich 13.7.1
rpds-py 0.19.0
ruff 0.5.1
safetensors 0.4.3
scipy 1.13.1
semantic-version 2.10.0
setuptools 65.5.0
shellingham 1.5.4
six 1.16.0
sniffio 1.3.1
starlette 0.37.2
tokenizers 0.19.1
tomlkit 0.12.0
toolz 0.12.1
tqdm 4.66.4
transformers 4.42.4
typer 0.12.3
typing_extensions 4.12.2
tzdata 2024.1
ujson 5.10.0
urllib3 2.2.2
uvicorn 0.30.1
uvloop 0.19.0
watchfiles 0.22.0
websockets 11.0.3Could you delete and re-download this model: mlx-community/llava-1.5-7b-4bit?
stewartugelow
commented
2 months ago Done, but no luck.
Request
Traceback (most recent call last):
File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/requests/models.py", line 974, in json
return complexjson.loads(self.text, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/stewart/anaconda3/lib/python3.11/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/stewart/anaconda3/lib/python3.11/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/stewart/anaconda3/lib/python3.11/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/stewart/Dropbox/dev/vmlx/llava.py", line 14, in <module>
print(response.json())
^^^^^^^^^^^^^^^
File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/requests/models.py", line 978, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)SERVER:
500 Internal Server Error
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
result = await app( # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
await super().__call__(scope, receive, send)
File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
await self.middleware_stack(scope, receive, send)
File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
raise exc
File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
await self.app(scope, receive, _send)
File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
raise exc
File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/routing.py", line 756, in __call__
await self.middleware_stack(scope, receive, send)
File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/routing.py", line 776, in app
await route.handle(scope, receive, send)
File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
await self.app(scope, receive, send)
File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
raise exc
File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/starlette/routing.py", line 72, in app
response = await func(request)
^^^^^^^^^^^^^^^^^^^
File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app
raw_response = await run_endpoint_function(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
return await dependant.call(**values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/fastmlx/fastmlx.py", line 135, in chat_completion
prompt = processor.apply_chat_template(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/stewart/Dropbox/dev/vmlx/rbuild/lib/python3.11/site-packages/transformers/processing_utils.py", line 926, in apply_chat_template
raise ValueError(
ValueError: No chat template is set for this processor. Please either set the `chat_template` attribute, or provide a chat template as an argument. See https://huggingface.co/docs/transformers/main/en/chat_templating for more information.
pip list
Package Version
------------------------- -----------
aiofiles 23.2.1
altair 5.3.0
annotated-types 0.7.0
anyio 4.4.0
attrs 23.2.0
certifi 2024.7.4
charset-normalizer 3.3.2
click 8.1.7
contourpy 1.2.1
cycler 0.12.1
dnspython 2.6.1
email_validator 2.2.0
fastapi 0.111.0
fastapi-cli 0.0.4
fastmlx 0.1.0
ffmpy 0.3.2
filelock 3.15.4
fonttools 4.53.1
fsspec 2024.6.1
gradio 4.37.2
gradio_client 1.0.2
h11 0.14.0
httpcore 1.0.5
httptools 0.6.1
httpx 0.27.0
huggingface-hub 0.23.4
idna 3.7
importlib_resources 6.4.0
Jinja2 3.1.4
jsonschema 4.23.0
jsonschema-specifications 2023.12.1
kiwisolver 1.4.5
markdown-it-py 3.0.0
MarkupSafe 2.1.5
matplotlib 3.9.1
mdurl 0.1.2
mlx 0.16.0
mlx-lm 0.15.2
mlx-vlm 0.0.11
numpy 1.26.4
orjson 3.10.6
packaging 24.1
pandas 2.2.2
pillow 10.4.0
pip 23.2.1
protobuf 5.27.2
pydantic 2.8.2
pydantic_core 2.20.1
pydub 0.25.1
Pygments 2.18.0
pyparsing 3.1.2
python-dateutil 2.9.0.post0
python-dotenv 1.0.1
python-multipart 0.0.9
pytz 2024.1
PyYAML 6.0.1
referencing 0.35.1
regex 2024.5.15
requests 2.32.3
rich 13.7.1
rpds-py 0.19.0
ruff 0.5.1
safetensors 0.4.3
scipy 1.13.1
semantic-version 2.10.0
setuptools 65.5.0
shellingham 1.5.4
six 1.16.0
sniffio 1.3.1
starlette 0.37.2
tokenizers 0.19.1
tomlkit 0.12.0
toolz 0.12.1
tqdm 4.66.4
transformers 4.42.4
typer 0.12.3
typing_extensions 4.12.2
tzdata 2024.1
ujson 5.10.0
urllib3 2.2.2
uvicorn 0.30.1
uvloop 0.19.0
watchfiles 0.22.0
websockets 11.0.3Is there debugging I could add to the fastmlx.py file that would be helpful?
Blaizzy
commented
2 months ago Yes, there is.
You can print the processor and its attributes. And print the configuration.
Also, as a sanity you can comment out the chat template part and pass the formatted text directly.
Example:
USER: <image>\nWhat are these?\nASSISTANT:I can share the code later. I'm currently travelling, and I'm on my phone.
stewartugelow
commented
2 months ago Upon further investigation, I can confirm that this is an mlx-vlm bug, not a fastmlx bug.
I get the same chat template errors with all of the following:
models--mlx-community--llava-1.5-7b-4bit models--mlx-community--llava-llama-3-8b-v1_1-8bit models--mlx-community--llava-phi-3-mini-4bit models--mlx-community--llava-v1.6-mistral-7b-8bit
Moving this to https://github.com/Blaizzy/mlx-vlm/issues/51
Getting this message:
File "/anaconda3/lib/python3.11/site-packages/transformers/processing_utils.py", line 926, in apply_chat_template raise ValueError( ValueError: No chat template is set for this processor. Please either set the
chat_templateattribute, or provide a chat template as an argument.Happened with:
models--mlx-community--llava-1.5-7b-4bit models--mlx-community--llava-llama-3-8b-v1_1-8bit