search

hltcoe / sandle

Run a large language modeling SANDbox in your Local Environment
Other
7 stars 1 forks source link

Ray Serve #81

Closed ccmaymay closed 1 year ago

ccmaymay commented 1 year ago

https://www.anyscale.com/blog/serving-pytorch-models-with-fastapi-and-ray-serve https://www.anyscale.com/blog/how-hutom-io-uses-ray-and-pytorch-to-scale-surgical-video-analysis-and

Integrates with FastAPI (#66)

ccmaymay commented 1 year ago

too low-level for our purposes; other frameworks provide more LLM-specific features