Add llm-rs-python to haystack-integrations

[anakin87]

Hey @LLukas22!

I'm Stefano and I work on the Haystack project.

I found llm-rs-python by chance and started experimenting with it on my laptop and was really impressed! 💣

So I would like you to add your project to haystack-integrations: if you open a PR there, your project will also appear in the Haystack website.

What do you think? Does this sound good to you?

[LLukas22]

Glad you enjoyed playing around with llm-rs-python a bit.

I already thought about adding it to the haystack-integrations and posting a short message + example in your discords #show-and-tell channel but decided against it until the new GGUF file format is implemented.

I don't know how much you know about the GGML ecosystem but there will soon (maybe 1-3 weeks) be a change from the current GGJTv3 file format to a new GGUF format which will be a breaking change with all executers (llama.cpp, rustformers etc. ) and existing converted models.

I see Haystack as a more stable and production ready alternative to LangChain and i don't want to publish an integration that will break in 1-3 weeks. These reoccuring file format changes were one of the reasons why i started contibruting to the rustformers project, which aimed to provide a more stable alternative to llama.cpp.

I will probably wait until GGUF and embedding generation with BERT-like models is finished and i can switch one of my Retrieval Augmented QA pipelines from pytorch over to rustformers before adding it to your integrations.

[TuanaCelik]

Hey @LLukas22 - It's great to hear that you are making sure you're building a project that is maintained and are conscious of people being able to use it 🙏 Some thoughts from me, it's totally up to you when and how you do this:

• The Discord is a lot more informal, I think people would really appreciate you posting about this already. If you tell them about a breaking change you know is coming and that you're making sure your project handles that, even better :)
• When you're ready, we can help you add it to the integrations page too. If you'd like to wait to make sure that package is more stable, let's do that 👍

[LLukas22]

Alright, i'll try to post about this via discord when i get back home from work and i'll probably add a little disclaimer hinting that there will be breaking changes soon.

Concerning the integrations page, i could create a draft but i would like to wait until GGUF lands in the ggml, llama.cpp and rustformers main branches.

[TuanaCelik]

Sounds good @LLukas22 Feel free to create a PR when you're ready, we could also leave it in a PR for you as long as tou need, to be merged when ready.

[TuanaCelik]

Hey @LLukas22 - I just wanted to check back in here and see if you'd like to create a page for Haystack integrations. I'm happy to help here. You were mentioning that there was a new GGUF format coming, any updates there? This article made me think about it: https://medium.com/@fvanlitsenburg/building-a-private-gpt-with-haystack-part-3-using-llama-2-with-ggml-c2d994da40da

[TuanaCelik]

Disregard me :D - You're already in touch with @anakin87 on Discord 🤗

[LLukas22]

@TuanaCelik I haven't forgotten about this, and i'm still planning on adding this after GGUF is finalized. But we still need to integrate full GGUF support into rustformers. And since Philpax (the original GGUF creator) and I are currently quite busy that could take some time. llama.cpp currently supports GGUF in it's beta releases and it's planned to deprecate the older GGJT format soon.

As already stated we will probably need more time on the rust side of things as we are also working on candle a bit and plan to integrate it in the near future, especially for embedding purposes.