OptimalScale / LMFlow

An Extensible Toolkit for Finetuning and Inference of Large Foundation Models. Large Models for All.
https://optimalscale.github.io/LMFlow/
Apache License 2.0
8.11k stars 819 forks source link

[Feature] Add vllm inference example #863

Closed wheresmyhair closed 1 week ago

wheresmyhair commented 1 week ago

Description

  1. Add vllm inference example
  2. Change detokenize to decode_inference_result to make it clearer

Tests

  1. MemorySafeVLLMInference test image

  2. Example sh test image Note that the fetal python error comes from the killing signal. This will not affect the inference, as the results are already saved: image

wheresmyhair commented 1 week ago

Changes made, tests passed.

Tests

  1. Example sh test image

  2. MemorySafeVLLMInference test image