search

llm-tools / embedJs

A NodeJS RAG framework to easily work with LLMs and embeddings
https://llm-tools.mintlify.app/get-started/introduction
Apache License 2.0
335 stars 40 forks source link

Consideration - Compress LLM template/query optimization #31

Closed converseKarl closed 7 months ago

converseKarl commented 7 months ago

there is another git project from Microsoft for LLM Prompt Query compression thatr reduces query prompt by 20% and optimizes it but also reduces the LLM costing and speeds up performance.

https://github.com/microsoft/LLMLingua

Could this be something useful to hook in as an optimization on the LLM template and query?

adhityan commented 7 months ago

Yes, this is certainly useful.

I checked this out when they put it out. I want to enable it as a simple flag you set to activate during build. Unfortunately Microsoft has only provided a Python implementation for this library atm. I am keeping track of this and we can reopen this thread once there is support for it in JS/TS.