Inference with `vllm` - Githubissues

THUDM / AgentTuning

AgentTuning: Enabling Generalized Agent Abilities for LLMs

https://thudm.github.io/AgentTuning/

1.36k stars 95 forks source link

Inference with `vllm` #29

Closed yc1999 closed 1 year ago

yc1999 commented 1 year ago

Hi, thanks for ur great work! I notice that you use Text-Generation-Inference to accelerate the evaluation process. Does your model work with vllm?

Btlmd commented 1 year ago

I haven't tested the checkpoints in vLLM yet. However, as we made no modifications to the model architecture of Llama-2, AgentLM should be able to run at any platform that supports Llama-2.