OpenedAI Vision

An OpenAI API compatible vision server, it functions like gpt-4-vision-preview and lets you chat about the contents of an image.

• Compatible with the OpenAI Vision API (aka "chat with images")
• Does not connect to the OpenAI API and does not require an OpenAI API Key
• Not affiliated with OpenAI in any way

Model support

Can't decide which to use? See the OpenVLM Leaderboard

If you can't find your favorite model, you can open a new issue and request it.

Recent updates

Version 0.39.2

• performance: use float16 with Qwen2 AWQ, small performance improvement
• fix: handle Ubuntu 24 / Python 3.12 a little better, thanks @Lissanro
• old code in the last docker, github worker problems again?

Version 0.39.1

• fix: the github docker package build seems to have been broken a while

Version 0.39.0

• new model support: rhymes-ai/Aria
• improved support for multi-image in various models.
• docker package: The latest release will now be tagged with :latest, rather than latest commit.
• ⚠️ docker: docker will now run as a user instead of root. Your hf_home volume may need the ownership fixed, you can use this command: sudo chown $(id -u):$(id -g) -R hf_home

Version 0.38.2

• Fix: multi-image for ovis 1.6

Version 0.38.1

• Fix: add wandb to requirements.txt so wandb is upgraded on manual install

Version 0.38.0

• new model support: AIDC-AI/Ovis1.6-Gemma2-9B

Version 0.37.0

• new model support: nvidia/NVLM-D-72B

Version 0.36.0

• new model support: BAAI/Emu3-Chat
• Experimental support for fancyfeast/joy-caption-alpha-two with multiple images (see: backend for more details)

Version 0.35.0

• Update Molmo (tensorflow-cpu no longer required), and added autocast for faster, smaller types than float32.
• New option: --use-double-quant to enable double quantization with --load-in-4bit, a little slower for a little less VRAM.
• Molmo 72B will now run in under 48GB of vram using --load-in-4bit --use-double-quant.
• Add completion_tokens counts and logged T/s for streamed results, other compatibility improvements
• Include sample tokens/s data (A100) in vision.sample.env

Version 0.34.0

• new model support: Meta-llama: Llama-3.2-11B-Vision-Instruct, Llama-3.2-90B-Vision-Instruct
• new model support: Ai2/allenai Molmo family of models
• new model support: stepfun-ai/GOT-OCR2_0, this is an OCR only model, all chat is ignored.
• Support moved to alt image: Bunny-Llama-3-8B-V, Bunny-v1_1-Llama-3-8B-V, Mantis-8B-clip-llama3, Mantis-8B-siglip-llama3, omchat-v2.0-13B-single-beta_hf, qihoo360/360VL-8B

Installation

Docker support (tested, recommended)

⚠️ Docker support requires NVIDIA CUDA container support installed and correctly configured

See Linux Installation or Windows Installation (use WSL2 with docker and nvidia drivers)

1) Edit the vision.env or vision-alt.env file to suit your needs. See: vision.sample.env for an example.

cp vision.sample.env vision.env
# OR for alt the version
cp vision-alt.sample.env vision-alt.env

2) You can run the server via docker compose like so:

# for OpenedAI Vision Server
docker compose up
# for OpenedAI Vision Server (alternate)
docker compose -f docker-compose.alt.yml up

Add the -d flag to daemonize and run in the background as a service.

3) To update your setup (or download the image before running the server), you can pull the latest version of the image with the following command:

# for OpenedAI Vision Server
docker compose pull
# for OpenedAI Vision Server (alternate)
docker compose -f docker-compose.alt.yml pull

Manual Installation instructions

Tested wth Python 3.10 & 3.11. AWQ and GPTQ are known to have issues with Python 3.12 (the default in Ubuntu 24) and require special attention, see below.

# Create & activate a new virtual environment (optional but recommended)
python -m venv .venv
source .venv/bin/activate
# upgrade pip
pip install --upgrade pip
# install the python dependencies
pip install -U -r requirements.txt "transformers>=4.45.2"
# OR install the python dependencies for the alt version
pip install -U -r requirements.txt "transformers==4.41.2"
# optional - export variables for your environment, see vision.sample.env.
#export HF_HOME=hf_home
# run the server with your chosen model, see vision.sample.env for known working configurations.
python vision.py --model vikhyatk/moondream2

Additional steps are required for some models (Mantis, Dragonfly, Emu3, etc.), see the Dockerfile for the latest installation instructions.

Python 3.12

The following additional steps are required to support AWQ and GPTQ models with Python 3.12. Perform these steps after other requirements have been installed.

1. Install the CUDA toolkit

2. Compile and install AWQ and GPTQ from source.

pip install git+https://github.com/AutoGPTQ/AutoGPTQ.git --no-build-isolation
INSTALL_KERNELS=1 pip install git+https://github.com/casper-hansen/AutoAWQ.git --no-build-isolation

Usage

usage: vision.py [-h] -m MODEL [-b BACKEND] [-f FORMAT] [-d DEVICE] [--device-map DEVICE_MAP] [--max-memory MAX_MEMORY] [--no-trust-remote-code] [-4]
                 [--use-double-quant] [-8] [-F] [-A {sdpa,eager,flash_attention_2}] [-T MAX_TILES] [--preload] [-L {DEBUG,INFO,WARNING,ERROR,CRITICAL}] [-H HOST]
                 [-P PORT]

OpenedAI Vision API Server

options:
  -h, --help            show this help message and exit
  -m MODEL, --model MODEL
                        The model to use, Ex. llava-hf/llava-v1.6-mistral-7b-hf (default: None)
  -b BACKEND, --backend BACKEND
                        Force the backend to use (phi3, idefics2, llavanext, llava, etc.) (default: None)
  -f FORMAT, --format FORMAT
                        Force a specific chat format. (vicuna, mistral, chatml, llama2, phi15, etc.) (doesn't work with all models) (default: None)
  -d DEVICE, --device DEVICE
                        Set the torch device for the model. Ex. cpu, cuda:1 (default: auto)
  --device-map DEVICE_MAP
                        Set the default device map policy for the model. (auto, balanced, sequential, balanced_low_0, cuda:1, etc.) (default: auto)
  --max-memory MAX_MEMORY
                        (emu2 only) Set the per cuda device_map max_memory. Ex. 0:22GiB,1:22GiB,cpu:128GiB (default: None)
  --no-trust-remote-code
                        Don't trust remote code (required for many models) (default: False)
  -4, --load-in-4bit    load in 4bit (doesn't work with all models) (default: False)
  --use-double-quant    Used with --load-in-4bit for an extra memory savings, a bit slower (default: False)
  -8, --load-in-8bit    load in 8bit (doesn't work with all models) (default: False)
  -F, --use-flash-attn  DEPRECATED: use --attn_implementation flash_attention_2 or -A flash_attention_2 (default: False)
  -A {sdpa,eager,flash_attention_2}, --attn_implementation {sdpa,eager,flash_attention_2}
                        Set the attn_implementation (default: sdpa)
  -T MAX_TILES, --max-tiles MAX_TILES
                        Change the maximum number of tiles. [1-55+] (uses more VRAM for higher resolution, doesn't work with all models) (default: None)
  --preload             Preload model and exit. (default: False)
  -L {DEBUG,INFO,WARNING,ERROR,CRITICAL}, --log-level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
                        Set the log level (default: INFO)
  -H HOST, --host HOST  Host to listen on, Ex. localhost (default: 0.0.0.0)
  -P PORT, --port PORT  Server tcp port (default: 5006)

API Documentation

• OpenAI Vision guide

Sample API Usage

chat_with_image.py has a sample of how to use the API.

Usage

usage: chat_with_image.py [-h] [-s SYSTEM_PROMPT] [--openai-model OPENAI_MODEL] [-S START_WITH] [-m MAX_TOKENS] [-t TEMPERATURE] [-p TOP_P] [-u] [-1] [--no-stream] image_url [questions ...]

Test vision using OpenAI

positional arguments:
  image_url             URL or image file to be tested
  questions             The question to ask the image (default: None)

options:
  -h, --help            show this help message and exit
  -s SYSTEM_PROMPT, --system-prompt SYSTEM_PROMPT
                        Set a system prompt. (default: None)
  --openai-model OPENAI_MODEL
                        OpenAI model to use. (default: gpt-4-vision-preview)
  -S START_WITH, --start-with START_WITH
                        Start reply with, ex. 'Sure, ' (doesn't work with all models) (default: None)
  -m MAX_TOKENS, --max-tokens MAX_TOKENS
                        Max tokens to generate. (default: None)
  -t TEMPERATURE, --temperature TEMPERATURE
                        Temperature. (default: None)
  -p TOP_P, --top_p TOP_P
                        top_p (default: None)
  -u, --keep-remote-urls
                        Normally, http urls are converted to data: urls for better latency. (default: False)
  -1, --single          Single turn Q&A, output is only the model response. (default: False)
  --no-stream           Disable streaming response. (default: False)

Example:

$ python chat_with_image.py -1 https://images.freeimages.com/images/large-previews/cd7/gingko-biloba-1058537.jpg "Describe the image."
The image presents a single, large green leaf with a pointed tip and a serrated edge. The leaf is attached to a thin stem, suggesting it's still connected to its plant. The leaf is set against a stark white background, which contrasts with the leaf's vibrant green color. The leaf's position and the absence of other objects in the image give it a sense of isolation. There are no discernible texts or actions associated with the leaf. The relative position of the leaf to the background remains constant as it is the sole object in the image. The image does not provide any information about the leaf's size or the type of plant it belongs to. The leaf's serrated edge and pointed tip might suggest it's from a deciduous tree, but without additional context, this is purely speculative. The image is a simple yet detailed representation of a single leaf.

$ python chat_with_image.py https://images.freeimages.com/images/large-previews/e59/autumn-tree-1408307.jpg
Answer: The image captures a serene autumn scene. The main subject is a deciduous tree, standing alone on the shore of a calm lake. The tree is in the midst of changing colors, with leaves in shades of orange, yellow, and green. The branches of the tree are bare, indicating that the leaves are in the process of falling. The tree is positioned on the left side of the image, with its reflection visible in the still water of the lake.

The background of the image features a mountain range, which is partially obscured by a haze. The mountains are covered in a dense forest, with trees displaying a mix of green and autumnal colors. The sky above is clear and blue, suggesting a calm and sunny day.

The overall composition of the image places the tree as the focal point, with the lake, mountains, and sky serving as a picturesque backdrop. The image does not contain any discernible text or human-made objects, reinforcing the natural beauty of the scene. The relative positions of the objects in the image create a sense of depth and perspective, with the tree in the foreground, the lake in the middle ground, and the mountains and sky in the background. The image is a testament to the tranquil beauty of nature during the autumn season.

Question: What kind of tree is it?
Answer: Based on the image, it is not possible to definitively identify the species of the tree. However, the tree's characteristics, such as its broad leaves and the way they change color in the fall, suggest that it could be a type of deciduous tree commonly found in temperate regions. Without more specific details or a closer view, it is not possible to provide a more precise identification of the tree species.

Question: Is it a larch?
Answer: The tree in the image could potentially be a larch, which is a type of deciduous conifer. Larches are known for their needle-like leaves that turn yellow and orange in the fall before falling off. However, without a closer view or more specific details, it is not possible to confirm whether the tree is indeed a larch. The image does not provide enough detail to make a definitive identification of the tree species.

Question: ^D

FAQ, Known Problems & Workarounds

1. Related to cuda device split, If you get:

   RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1! (when checking argument for argument tensors in method wrapper_CUDA_cat)

   Try to specify a single cuda device with CUDA_VISIBLE_DEVICES=1 (or # of your GPU) before running the script. or set the device via --device-map cuda:0 (or --device cuda:0 in the alt image!) on the command line.

2. 4bit/8bit quantization and flash attention 2 don't work for all the models. No workaround, see: sample.env for known working configurations.

3. The default --device-map auto doesn't always work well with these models. If you have issues with multiple GPU's, try using sequential and selecting the order of your CUDA devices, like so:

# Example for reversing the order of 2 devices.
CUDA_VISIBLE_DEVICES=1,0 python vision.py -m llava-hf/llava-v1.6-34b-hf --device-map sequential

You can also use the environment variable: OPENEDAI_DEVICE_MAP="sequential" to specify the --device-map argument.

4. "My Nvidia GPU isn't detected when using docker.", using Nvidia CUDA with docker.
   • Linux: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html
   • Windows: Use WSL2 with docker and nvidia drivers: https://docs.nvidia.com/cuda/wsl-user-guide/index.html