API example doesn't work

[im-not-tom]

Hello,

I was trying to add option to use backend of text-generation-webui to Tavern AI and noticed that your API example is outdated.

Trying to use it will generate rather cryptic IndexError in server output, as there are parameters missing.

2nd, much larger issue is that not only is collab version of the project using different number of expected parameters, it also generates only HTML. Plain text (markdown) output, which is generated when running text-generation-webui locally is missing.

Would it be possible to fix at least 2nd issue?

[oobabooga]

For using the API, make sure to start the server with these flags only:

python server.py --model model --listen --no-stream

That is, don't include --cai-chat. It should work as expected like this.

Let me know if you can get Tavern AI to work with this backend. That would be pretty cool and I would be interested in trying it.

[oobabooga]

Here is a notebook that you can use for this:

https://colab.research.google.com/github/oobabooga/AI-Notebooks/blob/main/API-notebook.ipynb

While calling the API, make sure to remove the 7860 port and replace HTTP with HTTPS:

server = "e44d97c7-ee3b-46e1.gradio.live"

response = requests.post(f"https://{server}/run/textgen", json={
...

[im-not-tom]

Thanks.

I was also able to use collab I linked above simply by removing lines

if cai_chat:
  params.add('--cai-chat')
else:
  params.add('--chat')

from 3rd code block. With this and some minor modifications of Tavern, I was able to use your backend. It seems like Tavern expects ony two API endpoins in the end. One to generate text and one to return name of currently selected model. I hacked together way to parse 2nd information in rather ugly way.

I'll check whether it would be possible to add real API endpoint, but to be honest, so far, gradio is all greek to me.

Let me know if you can get Tavern AI to work with this backend. That would be pretty cool and I would be interested in trying it.

I have it basically done, but now gradio.live went down. Hopefully not thanks to me.

Once it gets back online, I'll make PR to Tavern AI project. Meanwhile, my code is in last two commits here: https://github.com/im-not-tom/TavernAI/

---

Thanks for your help.

[im-not-tom]

Just adding link to PR in TaverAI project. Thank you again for your help

https://github.com/TavernAI/TavernAI/pull/63

[oobabooga]

Very nice work @im-not-tom! I hope it gets approved.

[alkeryn]

sorry for the bump but quick question, if you change the preset in the textgen webui, will that affect the api when tavern is set to use the gui setting ?