[FEATURE] Access to logits and final hidden layer

microsoft / DeepSpeed-MII

MII makes low-latency and high-throughput inference possible, powered by DeepSpeed.

Apache License 2.0

1.76k stars 163 forks source link

[FEATURE] Access to logits and final hidden layer #463

Open lshamis opened 2 months ago

lshamis commented 2 months ago

Hello, I don't believe there is any way to access logits or the final hidden layer embedding in the current API and would like to request this feature. Thank you

SulRash commented 1 month ago

This would definitely make the framework infinitely better than vLLM, which suffers the same issue.