sobelio / llm-chain

`llm-chain` is a powerful rust crate for building chains in large language models allowing you to summarise text and complete complex tasks
https://llm-chain.xyz
MIT License
1.3k stars 128 forks source link

llama: add Embeddings for llama #245

Closed danbev closed 8 months ago

danbev commented 9 months ago

This commit adds the ability to generate embeddings using the llama.

The motivation for this is to be able to use llama for embeddings in combination with a vector store, like Qdrant.

This commit also adds an example that demonstrates how to use the llm-chain-llama crate for generating embeddings and then use the Qdrant vector store for storing and searching for similar documents.


Example of running simliarity_search_llama:

env LLM_CHAIN_MODEL=~/work/ai/llama.cpp/models/llama-2-7b-chat.Q4_0.gguf cargo r --release --example similarity_search_llama
    Finished release [optimized] target(s) in 0.14s
     Running `/home/danielbevenius/work/ai/llm-chain/target/release/examples/similarity_search_llama`
llama_model_loader: loaded meta data with 19 key-value pairs and 291 tensors from
 ...
llama_new_context_with_model: compute buffer total size = 159.07 MiB
Documents stored under IDs: ["14081b4a-6690-4731-b64e-4058450fa428", "ee0687be-952c-4d30-845e-73807ffe74f1", "fad54a60-51db-4494-bcb8-8c80a12a151c"]
Retrieved stored documents: [Document { page_content: "Sound for the concert was engineered by sound engineer Bill Hanley. \"It worked very well\", he says of the event. \"I built special speaker columns on the hills and had 16 loudspeaker arrays in a square platform going up to the hill on 70-foot [21 m] towers. We set it up for 150,000 to 200,000 people. Of course, 500,000 showed up.\"[48] ALTEC designed marine plywood cabinets that weighed half a ton apiece and stood 6 feet (1.8 m) tall, almost 4 feet (1.2 m) deep, and 3 feet (0.91 m) wide. Each of these enclosures carried four 15-inch (380 mm) JBL D140 loudspeakers. The tweeters consisted of 4×2-Cell & 2×10-Cell Altec Horns. Behind the stage were three transformers providing 2,000 amperes of current to power the amplification setup.[49][page needed] For many years this system was collectively referred to as the Woodstock Bins.[50] The live performances were captured on two 8-track Scully recorders in a tractor trailer back stage by Edwin Kramer and Lee Osbourne on 1-inch Scotch recording tape at 15 ips, then mixed at the Record Plant studio in New York.[51]", metadata: Some(EmptyMetadata) }]
Juzov commented 8 months ago

lgtm @williamhogman thoughts?

williamhogman commented 8 months ago

Yeah let's merge