Update MPT FT handler to work with go server

mosaicml / examples

Fast and flexible reference benchmarks

Apache License 2.0

441 stars 125 forks source link

Closed asfandyarq closed 1 year ago

asfandyarq commented 1 year ago

Main changes:

Update predict to work with new batched input format (adapted from #341).
Better kwarg type conversion, to handle issues with json passed into this
FT checkpoint reuse and more logging (helps with debugging)
Explicitly import FasterTransformer.examples to avoid confusion with mosaicml/examples

Tested with latest inference images.

dskhudia commented 1 year ago

Looks great. Thank you for the improvements. Feel free to merge once CI is green.