search

Blaizzy / fastmlx

FastMLX is a high performance production ready API to host MLX models.
Other
159 stars 12 forks source link

Weird image URL bug with Wikimedia #11

Closed stewartugelow closed 2 months ago

stewartugelow commented 2 months ago

This request works perfectly (with version 0.01):

import requests
import json

url = "http://localhost:8000/v1/chat/completions"
headers = {"Content-Type": "application/json"}
data = {
    "model": "mlx-community/nanoLLaVA-1.5-4bit",
    "image": "https://th-thumbnailer.cdn-si-edu.com/vEjiMlcfuvZCgV0FJ1e3-nbbt3E=/1000x750/filters:no_upscale()/https://tf-cmsv2-smithsonianmag-media.s3.amazonaws.com/filer/0a/95/0a9509db-ff91-409d-8af5-5decfe0f661b/42-67183501.jpg",
    "messages": [{"role": "user", "content": "What are these"}],
    "max_tokens": 100
}

response = requests.post(url, headers=headers, data=json.dumps(data))
print(response.json())

This request, which is identical except for the image URL, breaks horribly:

import requests
import json

url = "http://localhost:8000/v1/chat/completions"
headers = {"Content-Type": "application/json"}
data = {
    "model": "mlx-community/nanoLLaVA-1.5-4bit",
    "image": "https://upload.wikimedia.org/wikipedia/commons/thumb/3/39/Giant_Pandas_playing.jpg/1024px-Giant_Pandas_playing.jpg",
    "messages": [{"role": "user", "content": "What are these"}],
    "max_tokens": 100
}

response = requests.post(url, headers=headers, data=json.dumps(data))
print(response.json())

Traceback from the request:

Traceback (most recent call last):
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/requests/models.py", line 971, in json
    return complexjson.loads(self.text, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stewart/anaconda3/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stewart/anaconda3/lib/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stewart/anaconda3/lib/python3.11/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/stewart/Dropbox/dev/temp-fastmlx/nano-panda.py", line 14, in <module>
    print(response.json())
          ^^^^^^^^^^^^^^^
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/requests/models.py", line 975, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Traceback from the server:

INFO:     127.0.0.1:51280 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 426, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/routing.py", line 756, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/routing.py", line 776, in app
    await route.handle(scope, receive, send)
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
    await self.app(scope, receive, send)
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/routing.py", line 72, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/fastmlx/fastmlx.py", line 126, in chat_completion
    output = vlm_generate(
             ^^^^^^^^^^^^^
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/mlx_vlm/utils.py", line 828, in generate
    input_ids, pixel_values, mask = prepare_inputs(
                                    ^^^^^^^^^^^^^^^
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/mlx_vlm/utils.py", line 694, in prepare_inputs
    image = load_image(image)
            ^^^^^^^^^^^^^^^^^
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/transformers/image_utils.py", line 363, in load_image
    image = PIL.Image.open(BytesIO(requests.get(image, timeout=timeout).content))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/PIL/Image.py", line 3283, in open
    raise UnidentifiedImageError(msg)
PIL.UnidentifiedImageError: cannot identify image file <_io.BytesIO object at 0x2b4b6ecf0>
stewartugelow commented 2 months ago

This happens in the 0.1 release as well.

Blaizzy commented 2 months ago

The image is a bytes io object so if you add ?raw=true to the end it should work.

Example:

import requests
import json

url = "http://localhost:8000/v1/chat/completions"
headers = {"Content-Type": "application/json"}
data = {
    "model": "mlx-community/nanoLLaVA-1.5-4bit",
    "image": "https://upload.wikimedia.org/wikipedia/commons/thumb/3/39/Giant_Pandas_playing.jpg/1024px-Giant_Pandas_playing.jpg?raw=true",
    "messages": [{"role": "user", "content": "What are these"}],
    "max_tokens": 100
}

response = requests.post(url, headers=headers, data=json.dumps(data))
print(response.json())
stewartugelow commented 2 months ago

I've never heard of BytesIO!

Unfortunately your suggestion didn't work...

Code:

import requests
import json

url = "http://localhost:8000/v1/chat/completions"
headers = {"Content-Type": "application/json"}
data = {
    "model": "mlx-community/nanoLLaVA-1.5-4bit",
    "image": "https://upload.wikimedia.org/wikipedia/commons/thumb/3/39/Giant_Pandas_playing.jpg/1024px-Giant_Pandas_playing.jpg?raw=true",
    "messages": [{"role": "user", "content": "What are these"}],
    "max_tokens": 100
}

response = requests.post(url, headers=headers, data=json.dumps(data))
print(response.json())

Traceback from client:

Traceback (most recent call last):
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/requests/models.py", line 971, in json
    return complexjson.loads(self.text, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stewart/anaconda3/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stewart/anaconda3/lib/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stewart/anaconda3/lib/python3.11/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/stewart/Dropbox/dev/temp-fastmlx/nano-panda.py", line 14, in <module>
    print(response.json())
          ^^^^^^^^^^^^^^^
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/requests/models.py", line 975, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Traceback from server:

INFO:     127.0.0.1:52534 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 426, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/routing.py", line 756, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/routing.py", line 776, in app
    await route.handle(scope, receive, send)
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
    await self.app(scope, receive, send)
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/starlette/routing.py", line 72, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/fastmlx/fastmlx.py", line 167, in chat_completion
    output = vlm_generate(
             ^^^^^^^^^^^^^
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/mlx_vlm/utils.py", line 828, in generate
    input_ids, pixel_values, mask = prepare_inputs(
                                    ^^^^^^^^^^^^^^^
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/mlx_vlm/utils.py", line 694, in prepare_inputs
    image = load_image(image)
            ^^^^^^^^^^^^^^^^^
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/transformers/image_utils.py", line 363, in load_image
    image = PIL.Image.open(BytesIO(requests.get(image, timeout=timeout).content))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/stewart/anaconda3/lib/python3.11/site-packages/PIL/Image.py", line 3283, in open
    raise UnidentifiedImageError(msg)
PIL.UnidentifiedImageError: cannot identify image file <_io.BytesIO object at 0x13c1bd3f0>
Blaizzy commented 2 months ago

What version of FastMLX are you running?

stewartugelow commented 2 months ago

I tried it with 0.01 first and then reproduced it with the 0.10 version.

Blaizzy commented 2 months ago

I can't replicate it.

Could you share your environment details:

Blaizzy commented 2 months ago

Wait I think I managed to replicate if 🤓

Blaizzy commented 2 months ago

I'm sorry @stewartugelow

It seems like this is a wikimedia issue or this particular link you have.

When I downloaded the image to my machine and ran it, it worked fine.

Additionally, I have tested fastmlx support with remote images and it works fine.

Examples:

New Pandas image Wikimedia:

import requests
import json

url = "http://localhost:8000/v1/chat/completions"
headers = {"Content-Type": "application/json"}
data = {
    "model": "mlx-community/llava-1.5-7b-4bit",
    "image": "https://upload.wikimedia.org/wikipedia/commons/thumb/3/39/Giant_Pandas_playing.jpg/640px-Giant_Pandas_playing.jpg",
    "messages": [{"role": "user", "content": "What are these?"}],
    "max_tokens": 100,
    "temperature": 0.7
}
response = requests.post(url, headers=headers, data=json.dumps(data))
print(response.json())

Bear image from Google:

import requests
import json

url = "http://localhost:8000/v1/chat/completions"
headers = {"Content-Type": "application/json"}
data = {
    "model": "mlx-community/nanoLLaVA-1.5-4bit",
    "image": "https://cdn.britannica.com/71/149571-050-C33E9F0F/Brown-bear-Finland.jpg?w=920&h=518&c=crop",
    "messages": [{"role": "user", "content": "What are these?"}],
    "max_tokens": 500,
    "temperature": 0.2
}

response = requests.post(url, headers=headers, data=json.dumps(data))
print(response.json())
Blaizzy commented 2 months ago

These should work. I have just tested them.

Let me know if the issue persists :)