search

stanfordnlp / dspy

DSPy: The framework for programming—not prompting—foundation models
https://dspy.ai
MIT License
18.76k stars 1.44k forks source link

VLLM support not yet merged #225

Closed meditans closed 11 months ago

meditans commented 12 months ago

Just mentioning this here to avoid anyone being confused by error messages like:

AttributeError: module 'dspy' has no attribute 'HFClientVLLM'. Did you mean: 'HFClientTGI'?

while following the documentation. See https://github.com/stanfordnlp/dspy/pull/93

okhat commented 11 months ago

Yeah it's probably ready to merge. We just need to test it out with a recent vllm version.

Have you given it a shot?

meditans commented 11 months ago

I have written a quick version for myself that works with current vLLM, adapting the idea of that commit. I can post a PR if that helps! (Edit: done)