Modifies the existing template for trt-llm to enable forward compatibility from the bridge. Additionally, enables non-streaming responses from the truss with temporary hack on "stream" value in the TRT LLM integration
:computer: How
:microscope: Testing
Tested the instantiation with my own truss, sent it additional fields, got back 200
stream value behavior has been validated in previous work
vshulman
commented
3 weeks ago
:rocket: What
Modifies the existing template for trt-llm to enable forward compatibility from the bridge. Additionally, enables non-streaming responses from the truss with temporary hack on "stream" value in the TRT LLM integration
:computer: How
:microscope: Testing
Tested the instantiation with my own truss, sent it additional fields, got back 200
stream value behavior has been validated in previous work