Cannot find oobabooga server on either port 7861 or backup port 7860

[CrazicalCZ]

[DeSinc]

You need to enable API access on Oobabooga. The API only works when Chat Mode is OFF and no-stream is enabled.

In summary, in the launch batch file for Oobabooga, make sure you:

1. REMOVE the --chat or --cai-chat arguments (API does not function with these enabled at the time of writing this)
2. ADD the --no-stream argument

Example: --no-stream --listen-port 7861 <-- I coded it so that it is expecting chat server on this port since stable diffusion uses 7860 by default too

[Shizzkizz]

Could give us the whole batch file or code for windows to launch it? I was trying this but still no luck.

python server.py --load-in-8bit --no-stream --listen-port 7861

No sure how you remove the --chat commands if they arent in the launch command. Really appreciate any help in simplifying this part as it feels like the last hurdle to get it working.

[DeSinc]

Ah yeah I meant remove --cai-chat or --chat if it was already present, but if it's not there already then that's fine.

I'm just running the default .bat file it gives you to run the program, but with --model pointing to my model name and I'm also setting --wbits 4 but that's because I want to quantise the model smaller. You might want to do the same thing or you might not. Also I think some models can't do --wbits 4, iirc the model has to be designed for it.

Here's my FULL .bat file that I tested with this bot, the launch args are at the very bottom:

@echo off

@echo Starting the web UI...

cd /D "%~dp0"

set MAMBA_ROOT_PREFIX=%cd%\installer_files\mamba
set INSTALL_ENV_DIR=%cd%\installer_files\env

if not exist "%MAMBA_ROOT_PREFIX%\condabin\micromamba.bat" (
  call "%MAMBA_ROOT_PREFIX%\micromamba.exe" shell hook >nul 2>&1
)
call "%MAMBA_ROOT_PREFIX%\condabin\micromamba.bat" activate "%INSTALL_ENV_DIR%" || ( echo MicroMamba hook not found. && goto end )
cd text-generation-webui

call python server.py --auto-devices --model alpaca-native-4bit --wbits 4 --groupsize 128 --no-stream --listen-port 7861

:end
pause

[DeSinc]

Edit: closing because this is an oobabooga config change not to do with my bot, you gotta do the above steps to fix it.

[DeSinc]

if you still have an issue - does the oobabooga server actually launch and can you access it? try removing that --load-in-8bit and try my model with my config to see if it helps. Run the download-model.bat in oobabooga folder and choose option L none of the above, then type this model: ozcur/alpaca-native-4bit then use it with my above bat file.