search

microsoft / DeepSpeed-MII

MII makes low-latency and high-throughput inference possible, powered by DeepSpeed.
Apache License 2.0
1.91k stars 175 forks source link

Please clarify structured output support #544

Open MRYingLEE opened 1 week ago

MRYingLEE commented 1 week ago

Deepspeed supports lm-format-enforcer(https://github.com/noamgat/lm-format-enforcer) through prefix_allowed_tokens_fn. An example is https://github.com/noamgat/lm-format-enforcer?tab=readme-ov-file#basic-tutorial. So, I am sure DeepSpeed supports structured output.

So far, I didn't find such a kind of example with DeepSpeed-MII.

Can someone clarify whether DeepSpeed-MII has structured output support built-in?

Thank you in advance,