[FEATURE] Access to logits and final hidden layer

microsoft / DeepSpeed-MII

MII makes low-latency and high-throughput inference possible, powered by DeepSpeed.

Apache License 2.0

1.91k stars 175 forks source link

[FEATURE] Access to logits and final hidden layer #463

Open lshamis opened 7 months ago

lshamis commented 7 months ago

Hello, I don't believe there is any way to access logits or the final hidden layer embedding in the current API and would like to request this feature. Thank you

SulRash commented 6 months ago

This would definitely make the framework infinitely better than vLLM, which suffers the same issue.