search

utilityai / llama-cpp-rs

132 stars 42 forks source link

Impelement sync / send for llama structs #481

Open thewh1teagle opened 2 weeks ago

thewh1teagle commented 2 weeks ago

So I can use it across threads with Mutex.

MarcusDunn commented 2 weeks ago

I don't know if it is safe to implement either of those.

A PR would be welcome if there's both a problem that cannot be solved without implementing them and an argument for why Send and Sync are safe to implement (I'm not nearly a good enough rust or c++ developer to be confident)

I've had success with static's + message passing while writing a web server based on this library, it's not super elegant, but it certainly gets the job done. I'd be interested to know the usecase you're running into.