Issue with openai extension in docker container for text-generation-webui => Python 3.10 required.

[ark626]

When running the docker container for text-generation-webui there is an issue with the openai extension due to an incompatibility with the syntax of the typing.py file.

Basically they use the newest syntax instead of the Optionals and Union typing package removing any downwards compatibility of it. This can be fixed by editing the following file:

/opt/text-generation-webui/extensions/openai/typing.py with the content of this text i.E. via nano.

To inject into the docker container and edit files check the number via docker ps and run docker exec -it <containerid> bash.

I know this fix is only temporal as it gets refreshed when the container is deleted but for now it seems to work.

typing.txt

[dusty-nv]

Thanks @ark626 - trying to think of a good solution for this, unless these can just be patched via sed commands in the dockerfile on JetPack 5 (although that would not be robust to changes upstream). Unfortunately they seem to have made Python 3.10 a hard requirement now. For now, you could use docker commit to save your changes to a different container tag - then everytime you start that new container tag, your changes will be intact.

[ark626]

I checked the stable diffusion. Most of the "Need" for python 3.10 stems from using things like the new Typing. This is easily exchangeable and gives downwards compatibility to 3.9. I currently tried this quick and dirty but will do a fork of the repo later on.

Seems to affect only 3-5 files in total.

Changes in the files like exchanging @cache to @functools.lru_cache(maxsize=None) or exchange dict tuple list etc for Typing.Dict Typing.List etc.

[dusty-nv]

OK yes, some other containers I do that patching of the typing hints on (this was predominantly during JetPack 5, which didn't come with Python 3.10 by default). Admittedly it is frustrating to keep up with the changes when the only things used in newer python are these typings.

[ark626]

Honestly as a Support Person i dont understand why they immediatly switch to this newer kind of syntax while the old isnt even deprecated. In fact the new typings where introduced with Python 3.10 and this is rather new. So this change might be a little to soon after the new syntax was introduced.

If you want i can create a fork of the stable diffusion repo with the downport done. Then you could implement it into the docker container. Currently i simply modified the files i needed but i can take a deeper look and patch all the files to 3.9 this way its maybe easier to have a Docker image for Jetpack 5 users.

[dusty-nv]

If you want i can create a fork of the stable diffusion repo with the downport done.

I wouldn't fork it because then it'll just perpetually be out of date with upstream. Instead I find patching these with sed in the dockerfile to be okay, but hopefully in this case it isn't too many patches. Agree about the typing syntaxes though, for some reason this keeps popping up around projects. And the Python version I build around on JetPack 5 is Python 3.8, because that's what Ubuntu 20.04 comes with.

[GuardMoony]

Hi all,

sorry to barg in on this topic/issue, don't want to hijack this. But @ark626 can you confirm the API is usable by an external tool after patching? Because it tried this patch and got: "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error

[TadayukiOkada]

@GuardMoony there have been some changes to typing.py it seems. I've merged recent changes to ark626's file and it works for me: typing.txt

[GuardMoony]

@TadayukiOkada seems ok now. Thx!

[ark626]

Hi all,

sorry to barg in on this topic/issue, don't want to hijack this. But @ark626 can you confirm the API is usable by an external tool after patching? Because it tried this patch and got: "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error

Yes but the completions endpoint in the old text-generation-webui had a bug where it was completions and not completion or so. Some older external tools did concider it but maybe you have to check also their older versions to see how they fixed this typo.

[amasciotta]

I'm facing the same problem. The workaround I've tried is rebuilding the container and manually overriding the Python version, as indicated here and then here. I've run the command

PYTHON_VERSION=3.10 jetson-containers build text-generation-webui

The build fails with the following error:

+ apt-get clean
+ curl -sS https://bootstrap.pypa.io/get-pip.py
+ python3.10
Looking in indexes: http://jetson.webredirect.org/jp5/cu114
ERROR: Could not find a version that satisfies the requirement pip (from versions: none)
ERROR: No matching distribution found for pip
+ curl -sS https://bootstrap.pypa.io/pip/3.6/get-pip.py
+ python3.6
/tmp/install_python.sh: line 27: python3.6: command not found
curl: (23) Failed writing body (0 != 16384)
The command '/bin/sh -c /tmp/install_python.sh' returned a non-zero code: 127
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/devel/jetson-containers/jetson_containers/build.py", line 103, in <module>
    build_container(args.name, args.packages, args.base, args.build_flags, args.simulate, args.skip_tests, args.test_only, args.push, args.no_github_api)
  File "/home/devel/jetson-containers/jetson_containers/container.py", line 143, in build_container
    status = subprocess.run(cmd.replace(_NEWLINE_, ' '), executable='/bin/bash', shell=True, check=True)  
  File "/usr/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'DOCKER_BUILDKIT=0 docker build --network=host --tag text-generation-webui:r35.5.0-cp310-python --file /home/devel/jetson-containers/packages/build/python/Dockerfile --build-arg BASE_IMAGE=text-generation-webui:r35.5.0-cp310-cudnn --build-arg PYTHON_VERSION_ARG="3.10" /home/devel/jetson-containers/packages/build/python 2>&1 | tee /home/devel/jetson-containers/logs/20240411_121344/build/text-generation-webui_r35.5.0-cp310-python.txt; exit ${PIPESTATUS[0]}' returned non-zero exit status 127.

I'm using a Jetson Xavier with Jetpack 5.1.3. Any help would be much appreciated!

[dusty-nv]

Hi @amasciotta, looking into it now - it would be nice to get this all on Python 3.10 for JetPack 5, but will take time to rebuild everything (not just oobabooga, but pytorch and all the LLM APIs too). For the time being you may want to patch it like https://github.com/dusty-nv/jetson-containers/issues/364#issuecomment-1977582655

EDIT: was able to build python 3.10 on JP5 without issue...moving onto pytorch

[dusty-nv]

OK, here is the Python 3.10 oogabooga container for JetPack 5 to try (daec892):

dustynv/text-generation-webui:r35.4.1-cp310

I verified the llama.cpp loader is working fine, exllamav2 is not working though - I can't build newer than exllamav2 0.0.14 on JetPack 5, because newer exllama requires CUDA 12. And text-generation-webui imports stuff from exllamav2 that is not in 0.0.14. So we may still need to go back and pin the oogabooga version to an older tag on JetPack 5.