search

socketteer / loom

Multiversal tree writing interface for human-AI collaboration
1k stars 72 forks source link

Integration with `transformers` #13

Open deepdiffuser opened 1 year ago

deepdiffuser commented 1 year ago

It would be interesting to be able to use loom with open source LLMs such as GPT-Neo-X, FLAN-UL2, and LLaMA. The transformers library by Huggingface has support for almost every open source LLM through a standardized interface.

One approach to accomplish this could be direct integration. Another approach, to keep the loom client thin, could be to develop (maybe this already exists?) a shim that adapts the OpenAI API shape to a transformers backend

mirrorcult commented 1 year ago

https://github.com/oobabooga/text-generation-webui, which just runs transformers (and has support for LLaMA etc) has an API which is pretty similar to the openAI shape--probably not hard to adapt loom to post requests to the local API, I was planning on doing this

deepdiffuser commented 1 year ago

I just discovered https://github.com/hyperonym/basaran which should accomplish this!

mirrorcult commented 1 year ago

interesting, that looks much better

kirill-fedyanin commented 1 year ago

@mirrorcult Have you succeed to launch loom on llama?

Technologicat commented 2 months ago

I'd like to second this.

Having read the blog posts on Loom, this thing borders on the legendary... so would be very nice to get it working with a local LLM. Even though oobabooga's text-generation-webui nowadays provides an OpenAI-compatible API, Loom seems hardcoded to call OpenAI's servers.

Most LLM tools are missing the multiverse features, which are kind of the point of this.

SillyTavern with its Timelines extension comes close, for chat mode, but it'd be really nice to play around with base models with a tool like this.