Refactor generators and add ct2fast support

LucienShui / huggingface-vscode-endpoint-server

starcoder server for huggingface-vscdoe custom endpoint

Apache License 2.0

166 stars 55 forks source link

Open piratos opened 1 year ago

piratos commented 1 year ago

Hello, in my fork I:

refactored the generators
Added support loading ctranslate2 based models (starcoderct2fast) which are incredibly fast on consumer hardware
Added support to finding the model type and returning the correct class in the main function (from local or HF hub)
support for websocket streaming

it is a WIP but if you want any of these features I ll be happy to create a proper PR

LucienShui commented 1 year ago

Sorry for taking so long to reply to you. In fact, I encourage everyone to submit pull requests directly, and I will carefully review each one.

thanhnew2001 commented 8 months ago

Anything happened to this ticket?