Performance tests ctransformers

su77ungr / CASALIOY

♾️ toolkit for air-gapped LLMs on consumer-grade hardware

Apache License 2.0

230 stars 31 forks source link

Performance tests ctransformers #46

Open su77ungr opened 1 year ago

su77ungr commented 1 year ago

@hippalectryon-0 introduced HF text embeddings with #45.

May you - if it fits you well - elaborate how this performs?

Edit: missing embeddings port

hippalectryon-0 commented 1 year ago

Not sure what the benchmark would be though ?

su77ungr commented 1 year ago

lacking understanding here but this should be seen as a llamacpp port competitor ig... the repo was created a few hours ago so there's still missing a huge chunk. also might support mosaic in the future

hippalectryon-0 commented 1 year ago

Implemented in #transfor.

However there are several drawbacks:

No stop word for now https://github.com/marella/ctransformers/issues/3 leading to very long generated messages
No streaming, therefore we don't see anything until the end of the generation

su77ungr commented 1 year ago

streaming added now https://github.com/marella/ctransformers/releases/tag/v0.1.2