search

sobelio / llm-chain

`llm-chain` is a powerful rust crate for building chains in large language models allowing you to summarise text and complete complex tasks
https://llm-chain.xyz
MIT License
1.3k stars 128 forks source link

Updated llama.cpp #199

Closed andychenbruce closed 1 year ago

andychenbruce commented 1 year ago

llama.cpp upstream had tensor_split converted from an array to a *const f32 which makes it a little more complicated since it doesn't have the trait Sync so any structures using it also aren't Sync.

https://github.com/ggerganov/llama.cpp/commit/ae178ab46bfd6ecb2422da5dad441a4e2fef8b7e

I think it is safe to explicitly mark them as so

unsafe impl Sync for ContextParams {}
unsafe impl Send for ContextParams {}

as long as the data at the pointer is never mutated by llama.cpp but not sure.

Also had to remove the Serialize, Deserialize for the *const f32 from being derived but that didn't break anything so idk why they were there in the first place.

Juzov commented 1 year ago

Hi andy, looks good but you need to upgrade the bindings.rs after you build the package. As per the bindgen userguide.

The resulting bindings will be written to $OUT_DIR/bindings.rs where $OUT_DIR is chosen by cargo and is something like ./target/debug/build/llm-chain-llama-sys-23ea4c1322828090/out/.

so copy the bindings.rs from there into to src/bindings.rs in llama-sys. Thanks!

andychenbruce commented 1 year ago

👌

Juzov commented 1 year ago

Looks like there's some build problems. Merge with main first before debugging, there were some problems in the main branch as well.