[Bug] Qwen2-1.5B Q4F16_0 - libc++abi: terminating due to uncaught exception of type std::length_error: vector

[digisomni]

🐛 Bug

I believe when the final token completes or is about to be completed in a request, the entire app crashes with libc++abi: terminating due to uncaught exception of type std::length_error: vector

To Reproduce

Steps to reproduce the behavior:

1. Load an MLC Engine in Python with Qwen2-1.5B Q4F16_0
2. Inference it with a structured request (JSON), use streaming so you can see how all the tokens do complete but before/around when the request finalizes, it crashes.

Expected behavior

MLC should not crash, which in turn crashes my whole Python application.

Environment

• Platform (e.g. WebGPU/Vulkan/IOS/Android/CUDA): Metal
• Operating system (e.g. Ubuntu/Windows/MacOS/...): macOS Sonoma Version 14.6.1 (23G93)
• Device (e.g. iPhone 12 Pro, PC+RTX 3090, ...) Macbook Air M2 (24GB RAM, 512GB)
• How you installed MLC-LLM (conda, source): python -m pip install --pre -U -f https://mlc.ai/wheels mlc-llm-nightly mlc-ai-nightly
• How you installed TVM-Unity (pip, source): Was already installed with the mlc-nightly pip install
• Python version (e.g. 3.10): 3.11.9
• GPU driver version (if applicable): N/A
• CUDA/cuDNN version (if applicable): N/A
• TVM Unity Hash Tag (python -c "import tvm; print('\n'.join(f'{k}: {v}' for k, v in tvm.support.libinfo().items()))", applicable if you compile models): N/A
• Any other relevant information: N/A

[Ubospica]

Hi @digisomni, thanks for reporting the error! Could you provide the complete error message and the script to reproduce the error so we can better identify the problem? I failed to reproduce this error on my device, but not certain if this is related the environment.