Closed stewartugelow closed 3 months ago
This happens in the 0.1 release as well.
The image is a bytes io object so if you add ?raw=true
to the end it should work.
Example:
import requests
import json
url = "http://localhost:8000/v1/chat/completions"
headers = {"Content-Type": "application/json"}
data = {
"model": "mlx-community/nanoLLaVA-1.5-4bit",
"image": "https://upload.wikimedia.org/wikipedia/commons/thumb/3/39/Giant_Pandas_playing.jpg/1024px-Giant_Pandas_playing.jpg?raw=true",
"messages": [{"role": "user", "content": "What are these"}],
"max_tokens": 100
}
response = requests.post(url, headers=headers, data=json.dumps(data))
print(response.json())
I've never heard of BytesIO!
Unfortunately your suggestion didn't work...
Code:
import requests
import json
url = "http://localhost:8000/v1/chat/completions"
headers = {"Content-Type": "application/json"}
data = {
"model": "mlx-community/nanoLLaVA-1.5-4bit",
"image": "https://upload.wikimedia.org/wikipedia/commons/thumb/3/39/Giant_Pandas_playing.jpg/1024px-Giant_Pandas_playing.jpg?raw=true",
"messages": [{"role": "user", "content": "What are these"}],
"max_tokens": 100
}
response = requests.post(url, headers=headers, data=json.dumps(data))
print(response.json())
Traceback from client:
Traceback (most recent call last):
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/requests/models.py", line 971, in json
return complexjson.loads(self.text, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/stewart/anaconda3/lib/python3.11/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/stewart/anaconda3/lib/python3.11/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/stewart/anaconda3/lib/python3.11/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/stewart/Dropbox/dev/temp-fastmlx/nano-panda.py", line 14, in <module>
print(response.json())
^^^^^^^^^^^^^^^
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/requests/models.py", line 975, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Traceback from server:
INFO: 127.0.0.1:52534 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 426, in run_asgi
result = await app( # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
await super().__call__(scope, receive, send)
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
await self.middleware_stack(scope, receive, send)
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
raise exc
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
await self.app(scope, receive, _send)
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
raise exc
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/routing.py", line 756, in __call__
await self.middleware_stack(scope, receive, send)
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/routing.py", line 776, in app
await route.handle(scope, receive, send)
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
await self.app(scope, receive, send)
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
raise exc
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/routing.py", line 72, in app
response = await func(request)
^^^^^^^^^^^^^^^^^^^
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app
raw_response = await run_endpoint_function(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
return await dependant.call(**values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/fastmlx/fastmlx.py", line 167, in chat_completion
output = vlm_generate(
^^^^^^^^^^^^^
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/mlx_vlm/utils.py", line 828, in generate
input_ids, pixel_values, mask = prepare_inputs(
^^^^^^^^^^^^^^^
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/mlx_vlm/utils.py", line 694, in prepare_inputs
image = load_image(image)
^^^^^^^^^^^^^^^^^
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/transformers/image_utils.py", line 363, in load_image
image = PIL.Image.open(BytesIO(requests.get(image, timeout=timeout).content))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/stewart/anaconda3/lib/python3.11/site-packages/PIL/Image.py", line 3283, in open
raise UnidentifiedImageError(msg)
PIL.UnidentifiedImageError: cannot identify image file <_io.BytesIO object at 0x13c1bd3f0>
What version of FastMLX are you running?
I tried it with 0.01 first and then reproduced it with the 0.10 version.
I can't replicate it.
Could you share your environment details:
Wait I think I managed to replicate if 🤓
I'm sorry @stewartugelow
It seems like this is a wikimedia issue or this particular link you have.
When I downloaded the image to my machine and ran it, it worked fine.
Additionally, I have tested fastmlx support with remote images and it works fine.
Examples:
New Pandas image Wikimedia:
import requests
import json
url = "http://localhost:8000/v1/chat/completions"
headers = {"Content-Type": "application/json"}
data = {
"model": "mlx-community/llava-1.5-7b-4bit",
"image": "https://upload.wikimedia.org/wikipedia/commons/thumb/3/39/Giant_Pandas_playing.jpg/640px-Giant_Pandas_playing.jpg",
"messages": [{"role": "user", "content": "What are these?"}],
"max_tokens": 100,
"temperature": 0.7
}
response = requests.post(url, headers=headers, data=json.dumps(data))
print(response.json())
Bear image from Google:
import requests
import json
url = "http://localhost:8000/v1/chat/completions"
headers = {"Content-Type": "application/json"}
data = {
"model": "mlx-community/nanoLLaVA-1.5-4bit",
"image": "https://cdn.britannica.com/71/149571-050-C33E9F0F/Brown-bear-Finland.jpg?w=920&h=518&c=crop",
"messages": [{"role": "user", "content": "What are these?"}],
"max_tokens": 500,
"temperature": 0.2
}
response = requests.post(url, headers=headers, data=json.dumps(data))
print(response.json())
These should work. I have just tested them.
Let me know if the issue persists :)
This request works perfectly (with version 0.01):
This request, which is identical except for the image URL, breaks horribly:
Traceback from the request:
Traceback from the server: