search

vllm-project / vllm

A high-throughput and memory-efficient inference and serving engine for LLMs
https://docs.vllm.ai
Apache License 2.0
26.9k stars 3.95k forks source link

Bug when using your LLM API with prompt instances.. #529

Closed MM-IR closed 6 months ago

MM-IR commented 1 year ago
image

Hi, when using the model.generate API, I just run into this issue (200 dataset samples, stuck at 30%). If possible, could you provide any suggestions? This is really weird, as a rare case to this particular NL prompt, where your api runs smoothly with other prompt cases in my current tests.

Thanks in advance!

MM-IR commented 1 year ago

To be specific, I am testing on some sentiment transfer tasks, especially in positive transfer tasks with OPT models, I frequently find such issues occurred with several prompts (several positive prompts fail, but I see the same prompt does not fail with other LMs).

zhuohan123 commented 1 year ago

Hi! Thanks for reporting this! This seems like an issue caused by exceptionally long sequences. Can you share the code to reproduce this error so we can look into this?

MM-IR commented 1 year ago

Hi, I work with OPT-IML-1.3b models, and in addition to such phenomena, I find your results are inconsistent with a lot of other implementations, such as initial Huggingface API, and the DeepSpeed wrappers.

I do suggest you to experiment with OPT-IML series.

For the code, I am less likely to think it matters a lot, as I also experiment these prompts (yeah, they are long with100~200 tokens) with other model instances, such as OpenASSISTANT-pythia-12b, LLaMa-7B, etc. They all run smoothly...

MM-IR commented 1 year ago

Could you check again for OPT-IML instances?

zhuohan123 commented 1 year ago

Could you check again for OPT-IML instances?

Sorry to bother again, can you provide a concrete prompt example that can reproduce the error on opt-iml-1.3b? We tried some simple examples before but did not find errors.

hmellor commented 6 months ago

Closing this issue as stale as there has been no discussion in the past 3 months.

If you are still experiencing the issue you describe, feel free to re-open this issue.