Overcomplicated and unexplained usage for beginners

[Avroboros]

Hi, i wanted to ask if its possible to do a simple tutorial or a simple detailed guide how to make this work, since i've never used WSL before and all i see in the main page is just code gibberish and lines of script that i don't know what to do, where to execute them... etc

I installed wsl and aphrodite engine itself, umamba.exe doesn't work and when i run runtime.cmd it creates this environment :

"** Visual Studio 2019 Developer Command Prompt v16.11.27 Copyright (c) 2021 Microsoft Corporation

---

[vcvarsall.bat] Environment initialized for: 'x64' (windows) C:\AI\Aphrodite Engine (PygmalionAI)\aphrodite-engine-main>"

But i don't know what to do with it, help would be appreciated. I'm trying to use the API key to use with SillyTavern (ooba instead of kobold ai if possible).

[AlpinDale]

Hi. At the moment, native windows support is still a WIP. You will need to run the engine in WSL. You need to run this command inside the WSL terminal window:

git clone https://github.com/PygmalionAI/aphrodite-engine && cd aphrodite-engine && ./runtime.sh python -m aphrodite.endpoints.api_server_ooba --model TheBloke/Mythalion-13B-AWQ -q awq --dtype float16 --host 127.0.0.1

It will ask if you want to install from source, which you can safely type n to skip and install the pip package instead. This will launch a Ooba-compatible API that you can use with SillyTavern. In the example command, I specified Mythalion-13B, but you can switch it out with any other model you want. Make sure you set the -q command to either gptq or awq if you're using quantized models, and completely omit it if you're running a 16bit model.

P.S. This is mentioned in the README, but the default API key is EMPTY for the ooba endpoint. Make sure the Aphrodite API type is selected in the API tab.

[Avroboros]

i ran the wsl.exe window and ran the command you gave me but it gave me this error :

"tar (grandchild): bzip2: Cannot exec: No such file or directory tar (grandchild): Error is not recoverable: exiting now tar: Child died with signal 13 tar: Error is not recoverable: exiting now ./update-runtime.sh: line 5: bin/micromamba: No such file or directory ./update-runtime.sh: line 7: bin/micromamba: No such file or directory ./update-runtime.sh: line 8: bin/micromamba: No such file or directory ./update-runtime.sh: line 9: bin/micromamba: No such file or directory"

By the looks of it i need to install micromamba, so i tried inserting the line "conda install -c conda-forge micromamba" that the anaconda website gave me inside the wsl terminal and also inside the CMD and both gave me the error saying that the command "conda" is unrecognizable

[AlpinDale]

Looks like you don't have bzip2 installed. Please run sudo apt update && apt install bzip2, then run the command again. You may need to delete the aphrodite-engine folder. You can find it by opening your windows file explorer, and navigating to the Ubuntu drive. Look inside the home folder.

[Avroboros]

i did what you asked me and it downloaded everything perfectly, saying that it installed a bunch of modules and everything successfully. However, all the wsl did was install the packages, but didn't ask me if i wanted to install from source (so that i could press the N key), neither did it launch any API for now. (the aphrodite folder is in the root folder right now so maybe that affects it somehow? since i got a notification saying that folders in the root folder could have problems related to permissions and etc)

[Avroboros]

Fixed the problem, i was forced to create a username and change the directory from root to home.

i did it again and now its giving this error :

/home/user/aphrodite-engine/conda/envs/linux/bin/python: Error while finding module specification for 'aphrodite.endpoints.api_server_ooba' (ImportError: cannot import name 'cuda_utils' from partially initialized module 'aphrodite' (most likely due to a circular import) (/home/user/aphrodite-engine/aphrodite/init.py))

[AlpinDale]

Fixed the problem, i was forced to create a username and change the directory from root to home.

i did it again and now its giving this error :

/home/user/aphrodite-engine/conda/envs/linux/bin/python: Error while finding module specification for 'aphrodite.endpoints.api_server_ooba' (ImportError: cannot import name 'cuda_utils' from partially initialized module 'aphrodite' (most likely due to a circular import) (/home/user/aphrodite-engine/aphrodite/init.py))

Looks like you've encountered a bug/oversight. Did you decide to not install from source when running ./runtime.sh python -m ....? If yes, please build from source instead.

[Avroboros]

I ran with source this time, and still got the error :

Do you want to install aphrodite from source? (y/n): y Obtaining file:///home/user/aphrodite-engine Installing build dependencies ... done Checking if build backend supports build_editable ... done Getting requirements to build editable ... error error: subprocess-exited-with-error

× Getting requirements to build editable did not run successfully. │ exit code: 1 ╰─> [26 lines of output] Traceback (most recent call last): File "/home/user/aphrodite-engine/conda/envs/linux/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in main() File "/home/user/aphrodite-engine/conda/envs/linux/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main json_out['return_val'] = hook(hook_input['kwargs']) File "/home/user/aphrodite-engine/conda/envs/linux/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 132, in get_requires_for_build_editable return hook(config_settings) File "/tmp/pip-build-env-tn6sr4l8/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 468, in get_requires_for_build_editable return self.get_requires_for_build_wheel(config_settings) File "/tmp/pip-build-env-tn6sr4l8/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 355, in get_requires_for_build_wheel return self._get_build_requires(config_settings, requirements=['wheel']) File "/tmp/pip-build-env-tn6sr4l8/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 325, in _get_build_requires self.run_setup() File "/tmp/pip-build-env-tn6sr4l8/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 341, in run_setup exec(code, locals()) File "", line 80, in File "", line 37, in get_nvcc_cuda_version File "/home/user/aphrodite-engine/conda/envs/linux/lib/python3.10/subprocess.py", line 421, in check_output return run(popenargs, stdout=PIPE, timeout=timeout, check=True, File "/home/user/aphrodite-engine/conda/envs/linux/lib/python3.10/subprocess.py", line 503, in run with Popen(popenargs, kwargs) as process: File "/home/user/aphrodite-engine/conda/envs/linux/lib/python3.10/subprocess.py", line 971, in init self._execute_child(args, executable, preexec_fn, close_fds, File "/home/user/aphrodite-engine/conda/envs/linux/lib/python3.10/subprocess.py", line 1863, in _execute_child raise child_exception_type(errno_num, err_msg, err_filename) FileNotFoundError: [Errno 2] No such file or directory: '/root/aphrodite-engine/conda/envs/linux/bin/nvcc' [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. error: subprocess-exited-with-error

× Getting requirements to build editable did not run successfully. │ exit code: 1 ╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip. /home/user/aphrodite-engine/conda/envs/linux/bin/python: Error while finding module specification for 'aphrodite.endpoints.api_server_ooba' (ImportError: cannot import name 'cuda_utils' from partially initialized module 'aphrodite' (most likely due to a circular import) (/home/user/aphrodite-engine/aphrodite/init.py)) root@Donbascaust:/home/user/aphrodite-engine#

[Trustmaster554]

I've got the same error. I also tried it with and without installing from source.

python -m aphrodite.endpoints.api_server_kobold --help /usr/bin/python: Error while finding module specification for 'aphrodite.endpoints.api_server_kobold' (ImportError: cannot import name 'cuda_utils' from partially initialized module 'aphrodite' (most likely due to a circular import) (/mnt/c/Users/*****/Documents/Repo/Aphrodite/aphrodite-engine/aphrodite/init.py))

[Trustmaster554]

FYI, I fixed it moving the import to the only function where it is used:

in the file ../aphrodite/common/utils.py I removed line 9 "from aphrodite import cuda_utils" and moved it after line 30, with the correct indentation:

def get_max_shared_memory_bytes(gpu: int = 0) -> int:
    from aphrodite import cuda_utils
    """Returns the maximum shared memory per thread block in bytes."""

Also I got a follow-up error, which I solved by switching to the parent directory and staring aphrodite from there: $ ls -lisa total 0 17732923532795739 0 drwxrwxrwx 1 trustmaster trustmaster 4096 Oct 19 06:43 . 13229323905431723 0 drwxrwxrwx 1 trustmaster trustmaster 4096 Oct 19 06:35 .. 4503599627449309 0 drwxrwxrwx 1 trustmaster trustmaster 4096 Oct 19 08:20 aphrodite-engine $ python -m aphrodite.endpoints.api_server_kobold --model TheBloke/airochronos-33B-AWQ -q awq --dtype float16 --host 127.0.0.1

[Avroboros]

If only i could understand this.

[perkel666]

please add that this is linux only.

At the moment this software doesn't even achieve basics of software development to inform users what OS they need to use in order for it to work. Windows users will just see:

ERROR: Could not find a version that satisfies the requirement aphrodite-engine (from versions: none)
ERROR: No matching distribution found for aphrodite-engine

[AlpinDale]

@perkel666 https://github.com/PygmalionAI/aphrodite-engine?tab=readme-ov-file#requirements

[CamiloMM]

If only i could understand this

Kinda old post but I feel bad for you and wanna reassure you you're not stupid - most Python projects are kinda like this; Docker mostly exists as a solution to Works in My Machine!™ nature of Python stuff in particular. You gotta ship the whole computer for it to "just work". It's not the dev's fault here, as most people who do Python for a long time get used to it.

Some projects do the impossible and make it so Python mostly works on Windows most of the time, like TextGen WebUI (think SD WebUI but for text), or single executables like Kobold. When those don't work you can usually reinstall and it should generally fix them, as well. I suggest you try these.