[Bug]: mismatch between multimodal tokens and placeholders for Llava-Next (4 GPUs)

sayakpaul commented 6 days ago

Your current environment

The output of `python collect_env.py`

```text Collecting environment information... PyTorch version: 2.4.0+cu121 Is debug build: False CUDA used to build PyTorch: 12.1 ROCM used to build PyTorch: N/A OS: Ubuntu 20.04.6 LTS (x86_64) GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0 Clang version: Could not collect CMake version: version 3.27.7 Libc version: glibc-2.31 Python version: 3.10.14 (main, May 6 2024, 19:42:50) [GCC 11.2.0] (64-bit runtime) Python platform: Linux-5.15.0-1049-aws-x86_64-with-glibc2.31 Is CUDA available: False CUDA runtime version: 12.1.105 CUDA_MODULE_LOADING set to: N/A GPU models and configuration: Could not collect Nvidia driver version: Could not collect cuDNN version: Could not collect HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True CPU: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian Address sizes: 48 bits physical, 48 bits virtual CPU(s): 64 On-line CPU(s) list: 0-63 Thread(s) per core: 2 Core(s) per socket: 32 Socket(s): 1 NUMA node(s): 2 Vendor ID: AuthenticAMD CPU family: 25 Model: 1 Model name: AMD EPYC 7R13 Processor Stepping: 1 CPU MHz: 2649.998 BogoMIPS: 5299.99 Hypervisor vendor: KVM Virtualization type: full L1d cache: 1 MiB L1i cache: 1 MiB L2 cache: 16 MiB L3 cache: 128 MiB NUMA node0 CPU(s): 0-15,32-47 NUMA node1 CPU(s): 16-31,48-63 Vulnerability Gather data sampling: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Mitigation; safe RET Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch topoext invpcid_single ssbd ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr rdpru wbnoinvd arat npt nrip_save vaes vpclmulqdq rdpid Versions of relevant libraries: [pip3] numpy==1.26.4 [pip3] nvidia-cublas-cu12==12.1.3.1 [pip3] nvidia-cuda-cupti-cu12==12.1.105 [pip3] nvidia-cuda-nvrtc-cu12==12.1.105 [pip3] nvidia-cuda-runtime-cu12==12.1.105 [pip3] nvidia-cudnn-cu12==9.1.0.70 [pip3] nvidia-cufft-cu12==11.0.2.54 [pip3] nvidia-curand-cu12==10.3.2.106 [pip3] nvidia-cusolver-cu12==11.4.5.107 [pip3] nvidia-cusparse-cu12==12.1.0.106 [pip3] nvidia-ml-py==12.560.30 [pip3] nvidia-nccl-cu12==2.20.5 [pip3] nvidia-nvjitlink-cu12==12.6.68 [pip3] nvidia-nvtx-cu12==12.1.105 [pip3] onnx==1.16.2 [pip3] onnxruntime==1.19.2 [pip3] pyzmq==26.2.0 [pip3] torch==2.4.0 [pip3] torchvision==0.19.0 [pip3] transformers==4.44.2 [pip3] triton==3.0.0 [conda] numpy 1.26.4 pypi_0 pypi [conda] nvidia-cublas-cu12 12.1.3.1 pypi_0 pypi [conda] nvidia-cuda-cupti-cu12 12.1.105 pypi_0 pypi [conda] nvidia-cuda-nvrtc-cu12 12.1.105 pypi_0 pypi [conda] nvidia-cuda-runtime-cu12 12.1.105 pypi_0 pypi [conda] nvidia-cudnn-cu12 9.1.0.70 pypi_0 pypi [conda] nvidia-cufft-cu12 11.0.2.54 pypi_0 pypi [conda] nvidia-curand-cu12 10.3.2.106 pypi_0 pypi [conda] nvidia-cusolver-cu12 11.4.5.107 pypi_0 pypi [conda] nvidia-cusparse-cu12 12.1.0.106 pypi_0 pypi [conda] nvidia-ml-py 12.560.30 pypi_0 pypi [conda] nvidia-nccl-cu12 2.20.5 pypi_0 pypi [conda] nvidia-nvjitlink-cu12 12.6.68 pypi_0 pypi [conda] nvidia-nvtx-cu12 12.1.105 pypi_0 pypi [conda] pyzmq 26.2.0 pypi_0 pypi [conda] torch 2.4.0 pypi_0 pypi [conda] torchvision 0.19.0 pypi_0 pypi [conda] transformers 4.44.2 pypi_0 pypi [conda] triton 3.0.0 pypi_0 pypi ROCM Version: Could not collect Neuron SDK Version: N/A vLLM Version: 0.6.1@5a60699c452c0b9b8086a978d8572c257c2c3cc4 vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled GPU Topology: Could not collect ```

Running my original script through SLURM so that is why the output above doesn't have any GPUs. I am on 4 H100s.

Model Input Dumps

No response

🐛 Describe the bug

Similar to https://github.com/vllm-project/vllm/issues/7996, I am running into when using Llava-NexT:

[rank0]: ValueError: Attempted to assign 2340 + 2144 + 1850 + 2160 + 2832 + 2438 + 2340 + 2830 + 2536 + 1948 = 23418 multimodal tokens to 23516 placeholders

All the code is here: https://github.com/sayakpaul/simple-image-recaptioning

This is why I launch it:

# full CC3M training set
python main.py \
    --data_path="pipe:curl -s -f -L https://huggingface.co/datasets/pixparse/cc3m-wds/resolve/main/cc3m-train-{0000..0575}.tar" --batch_size=48

Before submitting a new issue...

[X] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

DarkLight1337 commented 6 days ago

Can you post the images in the batch that is causing the error?

sayakpaul commented 5 days ago

Strangely, even if I wrap the infer() call here under a try-except to serialize the faulty images, it won't do it and just error out with original error message.

Is this expected?

DarkLight1337 commented 5 days ago

I think if there is an internal failure inside the model, the whole vLLM engine needs to be started anew. You can try to narrow down the batch number that causes the error and post the corresponding images.

sayakpaul commented 5 days ago

Hmm quite strangely, it doesn't happen when using a single GPU. Does that sound similar?

DarkLight1337 commented 5 days ago

Hmm quite strangely, it doesn't happen when using a single GPU. Does that sound similar?

I haven't heard of such issues resulting from using multiple GPUs.

Another thing you can try is to increase max_model_len.

sayakpaul commented 5 days ago

It is already at 32k.

DarkLight1337 commented 5 days ago

It would greatly help debugging if you could identify which batch is consistently causing this error.

sayakpaul commented 3 days ago

Yeah I am trying. I am unable to find any outputs I am capturing with print() in my logs. Do I have to configure any special logging primitives?

DarkLight1337 commented 3 days ago

A few things you can try:

Set the environment variable according to here and set enforce_eager=True to disable cuda graph.
Catch the ValueError inside the model itself and inspect the pixel_values.

DarkLight1337 commented 3 days ago

Can you check whether #8496 fixes the issue for you?

sayakpaul commented 3 days ago

Thanks, will try when I get a moment.

sayakpaul commented 2 days ago

Seems to be working.

sayakpaul commented 7 hours ago

Will close this issue as https://github.com/vllm-project/vllm/pull/8496 seems to be working beautifully. I wanted to ask a silly question hence not opening a new issue.

I have the following simple script:

Code

```python import os import queue from concurrent.futures import ThreadPoolExecutor from tqdm import tqdm import fire from data_processing import initialize_dataloader from model import load_vllm_engine, infer from utils import save_results def main( data_path: str, batch_size: int = 48, dataloader_num_workers: int = 8, output_dir: str = "sample_outputs", max_tokens: int = 120, detect_watermarks: bool = False, ): vllm_engine, sampling_params = load_vllm_engine(max_tokens=max_tokens) dataloader = initialize_dataloader( data_path=data_path, batch_size=batch_size, dataloader_num_workers=dataloader_num_workers, output_dir=output_dir, detect_watermarks=detect_watermarks, ) output_queue = queue.Queue() save_thread = ThreadPoolExecutor(max_workers=dataloader_num_workers) os.makedirs(output_dir, exist_ok=True) save_future = save_thread.submit(save_results, output_queue, output_dir) try: print("Starting the generation process.") for batch in tqdm(dataloader): batch["sampling_params"] = sampling_params try: outputs = infer(vllm_engine, batch) if outputs is not None: original_captions = batch["original_captions"] img_bytes = batch["img_bytes"] img_hashes = batch["img_hashes"] output_queue.put((original_captions, outputs, img_bytes, img_hashes)) except: continue finally: output_queue.put(None) save_thread.shutdown(wait=True) save_future.result() print("All processes completed. Captions generation and saving done.") if __name__ == "__main__": fire.Fire(main) ```

Once it finishes execution on multiple GPUs successfully, I get:

All processes completed. Captions generation and saving done.
ERROR 09-19 02:37:04 multiproc_worker_utils.py:120] Worker VllmWorkerProcess pid 404188 died, exit code: -15
INFO 09-19 02:37:04 multiproc_worker_utils.py:123] Killing local vLLM worker processes
[rank0]:[W919 02:37:09.827306477 CudaIPCTypes.cpp:16] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]
/fsx/sayak/vllm/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown

Warnings could likely be ignored but I see an "ERROR". Should vllm engine be closed in a different way?

vllm-project / vllm