Add http server to JetStream - Githubissues

google / JetStream

JetStream is a throughput and memory optimized engine for LLM inference on XLA devices, starting with TPUs (and GPUs in future -- PRs welcome).

Apache License 2.0

194 stars 24 forks source link

Add http server to JetStream #115

Closed JoeZijunZhou closed 1 month ago

JoeZijunZhou commented 2 months ago

Add Http server entrypoint to JetStream
default: http://localhost:8080/
API docs: http://localhost:8080/docs
generate API: POST http://localhost:8080/v1/generate, DecodeRequest JSON
health check API: GET http://localhost:8080/v1/health
Tested with mock engine
TODO: Integrate with MaxText and PyTorch engines after #94