search

vllm-project / vllm

A high-throughput and memory-efficient inference and serving engine for LLMs
https://docs.vllm.ai
Apache License 2.0
28.09k stars 4.15k forks source link

[Bug]: InternVL bounding box prediction does not work #9153

Open MoritzLaurer opened 1 day ago

MoritzLaurer commented 1 day ago

Your current environment

The output of `python collect_env.py` ```text python collect_env.py Collecting environment information... PyTorch version: 2.4.0+cu121 Is debug build: False CUDA used to build PyTorch: 12.1 ROCM used to build PyTorch: N/A OS: Ubuntu 20.04.6 LTS (x86_64) GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0 Clang version: Could not collect CMake version: Could not collect Libc version: glibc-2.31 Python version: 3.9.5 (default, Jun 4 2021, 12:28:51) [GCC 7.5.0] (64-bit runtime) Python platform: Linux-5.10.192-183.736.amzn2.x86_64-x86_64-with-glibc2.31 Is CUDA available: True CUDA runtime version: 12.5.82 CUDA_MODULE_LOADING set to: LAZY GPU models and configuration: GPU 0: NVIDIA A10G Nvidia driver version: 535.161.07 cuDNN version: Probably one of the following: /usr/lib/x86_64-linux-gnu/libcudnn.so.9.2.1 /usr/lib/x86_64-linux-gnu/libcudnn_adv.so.9.2.1 /usr/lib/x86_64-linux-gnu/libcudnn_cnn.so.9.2.1 /usr/lib/x86_64-linux-gnu/libcudnn_engines_precompiled.so.9.2.1 /usr/lib/x86_64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.2.1 /usr/lib/x86_64-linux-gnu/libcudnn_graph.so.9.2.1 /usr/lib/x86_64-linux-gnu/libcudnn_heuristic.so.9.2.1 /usr/lib/x86_64-linux-gnu/libcudnn_ops.so.9.2.1 HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True CPU: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 48 bits physical, 48 bits virtual CPU(s): 48 On-line CPU(s) list: 0-47 Thread(s) per core: 2 Core(s) per socket: 24 Socket(s): 1 NUMA node(s): 1 Vendor ID: AuthenticAMD CPU family: 23 Model: 49 Model name: AMD EPYC 7R32 Stepping: 0 CPU MHz: 3110.497 BogoMIPS: 5599.58 Hypervisor vendor: KVM Virtualization type: full L1d cache: 768 KiB L1i cache: 768 KiB L2 cache: 12 MiB L3 cache: 96 MiB NUMA node0 CPU(s): 0-47 Vulnerability Gather data sampling: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Retbleed: Mitigation; untrained return thunk; SMT enabled with STIBP protection Vulnerability Spec rstack overflow: Mitigation; safe RET Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Retpolines, IBPB conditional, STIBP always-on, RSB filling Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch topoext ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru wbnoinvd arat npt nrip_save rdpid Versions of relevant libraries: [pip3] numpy==1.26.4 [pip3] nvidia-cublas-cu12==12.1.3.1 [pip3] nvidia-cuda-cupti-cu12==12.1.105 [pip3] nvidia-cuda-nvrtc-cu12==12.1.105 [pip3] nvidia-cuda-runtime-cu12==12.1.105 [pip3] nvidia-cudnn-cu12==9.1.0.70 [pip3] nvidia-cufft-cu12==11.0.2.54 [pip3] nvidia-curand-cu12==10.3.2.106 [pip3] nvidia-cusolver-cu12==11.4.5.107 [pip3] nvidia-cusparse-cu12==12.1.0.106 [pip3] nvidia-ml-py==12.560.30 [pip3] nvidia-nccl-cu12==2.20.5 [pip3] nvidia-nvjitlink-cu12==12.6.77 [pip3] nvidia-nvtx-cu12==12.1.105 [pip3] pyzmq==26.2.0 [pip3] torch==2.4.0 [pip3] torchvision==0.19.0 [pip3] transformers==4.45.2 [pip3] triton==3.0.0 [conda] numpy 1.26.4 pypi_0 pypi [conda] nvidia-cublas-cu12 12.1.3.1 pypi_0 pypi [conda] nvidia-cuda-cupti-cu12 12.1.105 pypi_0 pypi [conda] nvidia-cuda-nvrtc-cu12 12.1.105 pypi_0 pypi [conda] nvidia-cuda-runtime-cu12 12.1.105 pypi_0 pypi [conda] nvidia-cudnn-cu12 9.1.0.70 pypi_0 pypi [conda] nvidia-cufft-cu12 11.0.2.54 pypi_0 pypi [conda] nvidia-curand-cu12 10.3.2.106 pypi_0 pypi [conda] nvidia-cusolver-cu12 11.4.5.107 pypi_0 pypi [conda] nvidia-cusparse-cu12 12.1.0.106 pypi_0 pypi [conda] nvidia-ml-py 12.560.30 pypi_0 pypi [conda] nvidia-nccl-cu12 2.20.5 pypi_0 pypi [conda] nvidia-nvjitlink-cu12 12.6.77 pypi_0 pypi [conda] nvidia-nvtx-cu12 12.1.105 pypi_0 pypi [conda] pyzmq 26.2.0 pypi_0 pypi [conda] torch 2.4.0 pypi_0 pypi [conda] torchvision 0.19.0 pypi_0 pypi [conda] transformers 4.45.2 pypi_0 pypi [conda] triton 3.0.0 pypi_0 pypi ROCM Version: Could not collect Neuron SDK Version: N/A vLLM Version: 0.6.1.dev238+ge2c6e0a82 vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled GPU Topology: GPU0 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X 0-47 0 N/A Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks```

Model Input Dumps

No response

๐Ÿ› Describe the bug

I'm starting a vllm container with OpenGVLab/InternVL2 and vLLM version 0.6.2 with vllm serve OpenGVLab/InternVL2-1B --trust-remote-code

As a prompt, I'm specifically using the object detection prompt: Please provide the bounding box coordinates of the region this sentence describes: <ref>{}</ref>. This prompt is documented here, here, or here. This prompt is designed to make the model output bounding box coordinates.

This prompt works correctly when used in transformers: 

path = 'OpenGVLab/InternVL2-1B'
model = AutoModel.from_pretrained(
    path,
    torch_dtype=torch.bfloat16,
    low_cpu_mem_usage=True,
    use_flash_attn=True,
    trust_remote_code=True
).eval().cuda()

tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True, use_fast=False)

# set the max number of tiles in `max_num`
pixel_values = load_image('./data/image.png', max_num=12).to(torch.bfloat16).cuda()
generation_config = dict(max_new_tokens=1024, do_sample=True)

# single-image single-round conversation (ๅ•ๅ›พๅ•่ฝฎๅฏน่ฏ)
question = "<image>\nPlease provide the bounding box coordinate of the region this sentence describes: <ref>the top right cloud</ref>"
response = model.chat(tokenizer, pixel_values, question, generation_config)
print(f'User: {question}\nAssistant: {response}')
#output: 
User: <image>
Please provide the bounding box coordinate of the region this sentence describes: <ref>the top right cloud</ref>
Assistant: the top right cloud[[619, 165, 958, 412]]

When I use the same standard bounding box prompt with vLLM and the OAI client, however, the output is gibberish: 

from openai import OpenAI
import os
from time import time

# initialize the client but point it to TGI
client = OpenAI(
    base_url="http://localhost:8000/v1", 
    api_key="EMPTY"
)

generation_parameters = {
    "temperature": 0.2,
    "max_tokens": 128,
    "top_p": 0.7,
    #"max_input_tiles": 12,
    #"repetition_penalty": 1.2,
    "stream": False,
}

def predict(messages):
    return client.chat.completions.create(
        model="OpenGVLab/InternVL2-1B", # needs to be /repository since there are the model artifacts stored
        messages=messages,
        **generation_parameters
    ).choices[0].message.content

messages = [
    {"role": "system", 
     "content": "You are a helpful assistant",
    },
    {"role": "user", 
     "content": [
        {
            "type": "image_url",
            "image_url": {
                "url": "https://unsplash.com/photos/ZVw3HmHRhv0/download?ixid=M3wxMjA3fDB8MXxhbGx8NHx8fHx8fDJ8fDE3MjQ1NjAzNjl8&force=true&w=1920"
            }
        },
        {
            "type": "text",
            #"text": "What type of animal is in the image above? What color does it have?"
            "text": "\nPlease provide the bounding box coordinate of the region this sentence describes: <ref>the bird</ref>"  
        }
    ]},
]

response = predict(messages)
print(f"LLM output: {response}")
LLM output:  erรณtka exilis

The expected output would be something like the bird[[123, 234, 123, 234]] with bounding box coordinates for the object specified in the prompt.

The vLLM container does successfully generate text when prompted without this bounding-box prompt. e.g. the prompt "What type of animal is in the image above? What color does it have?" leads to this output with this image: The animal in the image is a species of bird. The bird in the image is a Blue Tit. The bird has a distinctive and colorful plumage, with a blue head and a yellow and a bit of a red or orange throat.

I have tested this with several images and prompt variants and generation parameters. The bounding box prompt always outputs some form of gibberish, while other prompts output correct text. I also noticed that non-bounding box prompts produce correct text, but it also seems a bit suboptimal with repetitions and small errors.

I have a suspicion that there is something wrong in the chat template application done by vLLM for OpenGVLab/InternVL2. My assumption is that, because the model has only been trained with this bounding box prompt with the very specific chat template, it fully fails when there is a small deviation from the chat template. For normal prompts, the negative effect of the chat template issue might be less visible, because it probably has seen a wider variety of input formats, so it still produces correct, but suboptimal outputs. But this is just an assumption and the issue might come from somewhere else. Optimal InternVL outputs can also be created via this official HF Space demo for comparison. I found the PR that added InternVL to vLLM, but I'm not sure which part of the code implements the preprocessor with the chat template.

Before submitting a new issue...

DarkLight1337 commented 1 day ago

Currently, we only support inserting image tokens (<image> etc.) per image, and hardcode it by model. This type of custom chat template where other tokens are inputted per image isn't supported in vLLM yet.

DarkLight1337 commented 1 day ago

@ywang96 together with https://github.com/vllm-project/vllm/pull/7905#issuecomment-2399659561, this is another case where we need to override how the image tokens are inserted per image input. We may have to work on this soon.

Isotr0py commented 22 hours ago

This is odd, because InternVL2-2B can create correct bounding box result:

$ python examples/offline_inference_vision_language.py -m internvl_chat --num-prompts 1
WARNING 10-09 16:50:14 config.py:352] Async output processing is only supported for CUDA or TPU. Disabling it for other platforms.
INFO 10-09 16:50:14 llm_engine.py:237] Initializing an LLM engine (v0.6.1.post3.dev210+g83caf35e.d20241003) with config: model='/data/LLM-model/InternVL2-2B', speculative_config=None, tokenizer='/data/LLM-model/InternVL2-2B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cpu, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=/data/LLM-model/InternVL2-2B, use_v2_block_manager=True, num_scheduler_steps=1, chunked_prefill_enabled=False multi_step_stream_outputs=True, enable_prefix_caching=False, use_async_output_proc=False, use_cached_outputs=False, mm_processor_kwargs={'max_dynamic_patch': 1})
WARNING 10-09 16:50:14 tokenizer.py:166] Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead.
WARNING 10-09 16:50:14 cpu_executor.py:328] CUDA graph is not supported on CPU, fallback to the eager mode.
WARNING 10-09 16:50:14 cpu_executor.py:354] Environment variable VLLM_CPU_KVCACHE_SPACE (GB) for CPU backend is not set, using 4 by default.
INFO 10-09 16:50:14 selector.py:183] Cannot use _Backend.FLASH_ATTN backend on CPU.
INFO 10-09 16:50:14 selector.py:128] Using Torch SDPA backend.
INFO 10-09 16:50:17 selector.py:183] Cannot use _Backend.FLASH_ATTN backend on CPU.
INFO 10-09 16:50:17 selector.py:128] Using Torch SDPA backend.
Loading safetensors checkpoint shards:   0% Completed | 0/2 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:00<00:00, 71.83it/s]

INFO 10-09 16:50:19 cpu_executor.py:212] # CPU blocks: 2730
<s><|im_start|>user
<image>
Please provide the bounding box coordinate of the region this sentence describes: <ref>the bird</ref><|im_end|>
<|im_start|>assistant

WARNING 10-09 16:50:21 tokenizer.py:166] Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead.
Processed prompts: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 1/1 [00:12<00:00, 12.95s/it, est. speed input: 22.63 toks/s, output: 1.47 toks/s]
the bird[[44, 409, 474, 936]]

However, both InternVL-1B and InternVL-4B indeed create gibberish outputs for box bounding.

MoritzLaurer commented 21 hours ago

Thanks for your quick responses!

@DarkLight1337 , I'm not sure if this really requires a special logic/implementation. The prompt Please provide the bounding box coordinate of the region this sentence describes: <ref>the bird</ref> is just a string and these <ref> tokens are just special tokens which the the tokenizer can handle. So as long as these string are inputted in the same way the model has seen them during training, it should just work natively afaik (but maybe I'm wrong, I lack knowledge of the internals of vLLM).

@Isotr0py , that's very interesting. I had only tried InternVL-1B, because deploying 2B gave me this error (although 1B had accepted DISABLE_SLIDING_WINDOW=true): 

Exit code: 1. Reason:         ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/config.py", line 207, in __init__
    self.max_model_len = _get_and_verify_max_len(
                         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/config.py", line 1738, in _get_and_verify_max_len
    raise NotImplementedError(
NotImplementedError: Disabling sliding window is not supported for models with rope_scaling. Please raise an issue so we can investigate.
Process SpawnProcess-1:
Traceback (most recent call last):
  File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 388, in run_mp_engine
    engine = MQLLMEngine.from_engine_args(engine_args=engine_args,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 134, in from_engine_args
    engine_config = engine_args.create_engine_config()
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/engine/arg_utils.py", line 874, in create_engine_config
    model_config = self.create_model_config()
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/engine/arg_utils.py", line 811, in create_model_config
    return ModelConfig(
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/config.py", line 207, in __init__
    self.max_model_len = _get_and_verify_max_len(
                         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/config.py", line 1738, in _get_and_verify_max_len
    raise NotImplementedError(
NotImplementedError: Disabling sliding window is not supported for models with rope_scaling. Please raise an issue so we can investigate.
MoritzLaurer commented 21 hours ago

Interesting, I confirm that I also get bounding boxes with the 2B model. (when successfully deployed with DISABLE_SLIDING_WINDOW=false)

MoritzLaurer commented 21 hours ago

I tested whether the issue comes from DISABLE_SLIDING_WINDOW, but I still get gibberish with the 1B model when changing to DISABLE_SLIDING_WINDOW=false.

(I noticed that internvl tokenizes images to many tokens (3k+ tokens for just one image and a short text prompt), so thought that it could maybe be linked to the sliding window making the model only pay attention to parts of the input)

Isotr0py commented 20 hours ago

Seems that InternVL2-4B can also output the correct bounding box on GPU, previous gibberish outputs may relate to the ipex on CPU backend:

$ python examples/offline_inference_vision_language.py -m internvl_chat
WARNING 10-09 11:04:13 config.py:1646] Casting torch.bfloat16 to torch.float16.
INFO 10-09 11:04:24 llm_engine.py:237] Initializing an LLM engine (v0.1.dev2938+gdc4aea6) with config: model='OpenGVLab/InternVL2-4B', speculative_config=None, tokenizer='OpenGVLab/InternVL2-4B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=4096, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=OpenGVLab/InternVL2-4B, use_v2_block_manager=True, num_scheduler_steps=1, chunked_prefill_enabled=False multi_step_stream_outputs=True, enable_prefix_caching=False, use_async_output_proc=True, use_cached_outputs=False, mm_processor_kwargs=None)
INFO 10-09 11:04:25 selector.py:217] Cannot use FlashAttention-2 backend for Volta and Turing GPUs.
INFO 10-09 11:04:25 selector.py:116] Using XFormers backend.
/opt/conda/envs/vllm/lib/python3.10/site-packages/xformers/ops/fmha/flash.py:211: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
  @torch.library.impl_abstract("xformers_flash::flash_fwd")
/opt/conda/envs/vllm/lib/python3.10/site-packages/xformers/ops/fmha/flash.py:344: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
  @torch.library.impl_abstract("xformers_flash::flash_bwd")
INFO 10-09 11:04:26 model_runner.py:1051] Starting to load model OpenGVLab/InternVL2-4B...
INFO 10-09 11:04:26 selector.py:217] Cannot use FlashAttention-2 backend for Volta and Turing GPUs.
INFO 10-09 11:04:26 selector.py:116] Using XFormers backend.
INFO 10-09 11:04:26 weight_utils.py:242] Using model weights format ['*.safetensors']
Loading safetensors checkpoint shards:   0% Completed | 0/2 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  50% Completed | 1/2 [00:03<00:03,  3.96s/it]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:07<00:00,  3.50s/it]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:07<00:00,  3.57s/it]

INFO 10-09 11:04:34 model_runner.py:1062] Loading model weights took 7.7740 GB
INFO 10-09 11:04:38 gpu_executor.py:122] # GPU blocks: 810, # CPU blocks: 682
INFO 10-09 11:04:38 gpu_executor.py:126] Maximum concurrency for 4096 tokens per request: 3.16x
INFO 10-09 11:04:40 model_runner.py:1385] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
INFO 10-09 11:04:40 model_runner.py:1389] CUDA graphs can take additional 1~3 GiB memory per GPU. If you are running out of memory, consider decreasing `gpu_memory_utilization` or enforcing eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
INFO 10-09 11:04:55 model_runner.py:1513] Graph capturing finished in 14 secs.
Processed prompts: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 4/4 [00:10<00:00,  2.56s/it, est. speed input: 1312.32 toks/s, output: 10.15 toks/s]
the bird[[39, 407, 497, 909]]
the bird[[46, 411, 503, 914]]
the bird[[42, 406, 497, 935]]
the bird[[41, 408, 495, 935]]
MoritzLaurer commented 18 hours ago

I also just got it to work with the 4b model