KoboldAI-Client is not a "client"?

[ghost]

I think that the documentation / readme may need significant improvement to clear up a possible misunderstanding.

First I attempted to learn how can I generate text with the KoboldAI Horde REST API in python, by using the "Full Documentation", but I failed to figure it out.

Then I thought I would install the "client" program and somehow intercept/debug it how it uses the KoboldAI Horde REST API.

At https://koboldai.net/ this github repo is recommended to GNU/Linux users, so I came here, and downloaded the stuff. (I am using MX Linux Wildflower)

Then after the download (I don't use git), I extracted "KoboldAI-united", I opened a terminal in the folder, and simply run ./play.sh in the hope it will start.

I would have preferred to do a simple

"pip install KoboldAI-client"

instead, but because in the readme it is explained, that this thing has its own self-contained runtime (conda?), so I choose to first just give it a try simply running "play.sh".

I seriously had my doubts at the point when I saw in the readme, that this (client?) has GPU requirements, and so many dependencies, but regardless I just gave it a try anyway.

The result was as I expected a complete tragedy. The automatic script downloaded 17.6 GB dependencies and the "Installation" failed. Moreover it automatically opened up the UI in my web browser (chromium), but the UI was not functional for obvious reasons. I had to manually kill the running process in the terminal with CTRL+C.

I only intended to get some simple (client) program that would act as a gateway by giving access to the KoboldAI Horde REST API, so I could get a working API URL, what I can paste into the TaverAI UI, to use the UI with the Kobold Horde Workers provided models.

Anyway, I just attempted to figure out a way I could use the processing power of the Kobold Horde with the TavernUI for text generation.

I did not intend to start a "server" or "worker" node. I only have a low-end laptop. There is no way I could run anything demanding on it. And I also do not wish to use google colab. I intended to use the processing power of the Kobold Horde, by using the public API combined with the TavernAI interface.

So something like https://lite.koboldai.net/ but with the TavernUI interface.

If this is NOT a simple "client" but something omnipotent combined

Server + Worker + Config Interface + KoboldAI lite user interface + Whatever magic,

then the "KoboldAI-Client" name is very confusing for people who are just looking for a "Client" program to gain access to the "Horde".

As an extra I attached the error log of the installation, but I opened this "issue" for pointing out that it should be somehow better mentioned or clarified in the documentation, that this is NOT a simple "client".

error.log

[henk717]

I agree, the -client is actually a server. The -client stems from the days colabs had seperate servers for the AI part. In my repo and anything based on it I have changed the name of the repo quite a while ago. But the official account I have limited access to and can't change it there.

The program isn't designed to run as a pip package and also has dependencies outside python.

[henk717]

Also to answer your question regarding Tavern, the KoboldAI server can't be used as an intermediate for the Horde and this is not planned because we do not want to give people the power to host the horde on the horde and disrupt the system. TavernAI will need to add horde support itself.

[ghost]

I see. Thank you very much.

So the (public) API is just for getting general info about the horde statistics?

Can this somehow be communicated to the community to improve the info displayed on the https://koboldai.net/ website?

I mean that currently a simple user visits the website, finds "KoboldAI Client", goes happy download without thinking, then gets confused. xD

[henk717]

I understand it can be quite confusing so ill summerize the current situation.

There is the real KoboldAI which repo was traditionally named KoboldAI-Client, but in reality its a bit of a middle ground since its both the client and the server in one. If you use it as a desktop application its going to present you with the KoboldAI Client while the server runs in the background. It makes it behave much closer to a traditional desktop program in that sense. If you run the KoboldAI software on another PC that would become the server, and you can then navigate to the browser page of the server for the Client. They are very tightly integrated so the only way to obtain the real KoboldAI Client is to run a copy of the server somewhere.

The KoboldAI server got API support, so next to the official client it also serves an API that other things can hook in to. TavernAI is one of the alternative clients that uses that API. So you can have the KoboldAI server running somewhere, and then instead of using the official client you use the API with TavernAI. This API is accessible by adding /api to the KoboldAI link (And by removing /new_ui if you use the new UI).

Another thing using the API is the KoboldAI Horde, the workers are KoboldAI servers connecting to the Horde and the Horde interacts with the servers API to send and receive information.

KoboldAI Horde (koboldai.net) is basically a standalone project that can interact with the KoboldAI API and Stable Diffusion models (Where it is called Stable Horde). It has its own APi as well, we call the KoboldAI Horde API which is available at https://koboldai.net/api .

The KoboldAI Horde has its own clients, KoboldAI itself can act as both a worker (server) or as a client for the horde. But the horde part is entirely optional for it. The server can run models on its own, connect to the Horde, or to another API. It can even connect to another instance of a KoboldAI server.

Then there is KoboldAI Lite, which is a browser based client for the Horde and very recently KoboldAI Servers. And KoboldAI Lite runs entirely in the browser.

---

So as far as clients go its all pretty interconnected and it depends on what you want a client for which one is best suited for you. Either a full copy of KoboldAI running locally or in the cloud, KoboldAI Lite, Tavern or something else.

[ghost]

Thank you very much!

So, if I understand it right now:

• The Client and the Server are tightly integrated.
• The KoboldAI server has its own API.
• TavernAI can use the API of the KoboldAI server.
• There is a part of the KoboldAI API especially related to Horde Workers.
• koboldai.net is an other project which has its own API, but does interact with the servers by using the KoboldAI API.
• KoboldAI Lite is a client for the koboldai.net Horde, and by default it is using the API of the koboldai.net Horde, but it is also capable of using the API of the KoboldAI servers.

1. Did I understood it correctly? ^

So if I want to use the koboldai.net Horde with TavernAI, then I must modify the TavernAI source code, to make it work with the koboldai.net Horde API.

Because TavernAI can only use the API of KoboldAI servers, but not the API of https://koboldai.net/api.

2. Right? ^

So I must be looking at the source code of KoboldAI Lite to figure it out how does the https://koboldai.net/api Horde API work, and then I must modify the TavernAI source code in order to to make it work with the Horde API. Because TavernAI doesn't work with the Horde API, but only with the KoboldAI server API.

3. Correct? ^

The official and up-to date source code of KoboldAI Lite is:

https://github.com/henk717/lite.koboldai.net

So I must read it and somehow figure it out how it works with the https://koboldai.net/api, and then implement this mechanism in TavernAI.

4. Right? ^

Sorry for the many questions. I just really wish to have fun by using TavernAI without running my own KoboldAI server, and using instead the free public workers of the Horde, which are currently only accessible through KoboldAI Lite.

[henk717]

Mostly correct, point 4 is wrong, we do not have anything specific to workers they use normal API calls like everything else.

The lite.koboldai.net on my account is an archive copy the real one in use is the one I forked from.

Everything else you got right.

[ghost]

Thank you very much.

I am still a little bit confused though.

You say that I shouldn't really care about the API of the Horde (koboldai.net/api) and instead figure out some way I could pick out one specific worker from the Horde and then I am ready to hook that worker up with Tavern?

Also, is there no active git repo for the development of lite.koboldai.net? Only an archive?

I downloaded the current newest version with

wget "https://lite.koboldai.net/"

but I feel kind of sad about that there is no way I could track its active development.

[henk717]

No, there are two API's one is the KoboldAI Local API which is what clients for KoboldAI can use (The Horde uses this to request generations from a worker, and this is just the regular KoboldAI under the hood, there is nothing specific to the horde in it).

Then there is the Horde API which the clients of the Horde can use and that one is an entirely different API available at https://koboldai.net/api

The page is hosted on this github pages from the developer of KoboldAI Lite, mine is just a fork for archival reasons: https://github.com/kaihordewebui/kaihordewebui.github.io

So for your purpose of getting TavernAI to work on the Horde you only need to care about the Horde API, the regular API is not relevant and already supported in Tavern.

[ghost]

Oh, my god, my head hurts. xD

Thank you very much for all of this explanation and help, and everything.

I think finally I understand everything now.

You are awesome.

:)