Is there a way to run pytriton on glibc2.32?

[DZ9]

I'm running my program on a gpu cluster with dockers. The default docker image has glibc 2.32 installed and its hard to upgrade it to 2.35. Is there any way I can running pytriton on glibc2.32 to serve my reward model？

[DZ9]

has been stucked here for sometime, please help

[piotrm-nvidia]

From your inquiry, it seems you are looking to run PyTriton on an environment with glibc 2.32 to serve your reward model, but are facing difficulties due to version compatibility issues with the Triton Inference Server. Given that glibc 2.32 is hard to upgrade in your setup, here's a possible approach to resolve this:

1. Identify the Compatible Versions:

   • The Triton Inference Server v2.33.0 supports glibc 2.31. If you can downgrade to glibc 2.31, you can use PyTriton with Triton Inference Server v2.33.0.
   • PyTriton version 0.2.5 is compatible with Triton Inference Server v2.33.0, as per the version details provided.
   • You can install PyTriton version 0.2.5 using the following command:

     pip install nvidia-pytriton==0.2.5

   • Ensure that you are using Python 3.8 or later, as PyTriton requires Python 3.8+.

2. Building PyTriton with Specific Triton Server Version:

   • You can build PyTriton targeting a specific version of the Triton Inference Server that is known to be compatible with glibc 2.31. Utilize the Docker image nvcr.io/nvidia/tritonserver:23.04-pyt-python-py3 tagged for Triton v2.33.0. See build guide for details.

   • You can build latest version of PyTriton agains 23.04 Triton Inference Server image with the following commands:

     # Assuming you are in the root directory of the PyTriton project
     make install-dev
     make dist -e TRITONSERVER_IMAGE_VERSION=23.04

Please provide more details about the operating system you are targeting with glibc 2.32. This information will help in understanding the specific challenges you are facing and enable us to provide a more tailored solution. Additionally, could you share any error messages or specific issues you encountered while trying to run PyTriton on glibc 2.32? This will help in diagnosing the problem more effectively. You can run pip install with verbose logging to capture detailed error messages.

[DZ9]

@piotrm-nvidia Thank you for your reply. I could install nvidia-pytriton==0.2.5 on my docker image, but 0.2.5 only support python3.8, while my program need to run with python 3.10+. Begin with 0.2.6+, pytriton supports python3.10, but unfortunately, glibc 2.35+ is required with 0.2.6+ because triton server in it is changed to 2.36.0.

I will try the building method, but I wonder weather this python3.10 support is deeply binded with triton server 2.36.0? Thanks again for your help!

[piotrm-nvidia]

Thank you for providing more details on your requirements. Since you need Python 3.10 support, but are constrained by the older glibc version, here's an adjusted approach that might work for you:

1. Custom Build of PyTriton:

   • You can indeed attempt to build PyTriton against the Triton Inference Server version that's compatible with glibc 2.31. Since PyTriton v0.3.0 includes build scripts that support Python 3.8 to 3.11, you can use these scripts to compile a Python 3.10 backend. Here’s how you can set this up:

     # Build PyTriton using Docker with Triton Server version that supports glibc 2.31
     make install-dev
     make dist -e TRITONSERVER_IMAGE_VERSION=23.04 -e PYTHON_VERSIONS="3.10"

   • This command specifies the Triton Inference Server Docker image and Python version. The build process will use the Docker image corresponding to Triton v2.33.0 (which is compatible with your glibc version) as the base, and compile the PyTriton backend for Python 3.10.

2. Testing and Validation:

   • It's important to note that while this setup isn't officially tested, it theoretically aligns with supported configurations. After building PyTriton, you should rigorously test your model deployment to ensure stability and performance.
   • Look for any runtime issues or incompatibilities that might arise due to the custom build environment. Pay close attention to errors related to linking or missing libraries.

3. Monitoring and Adjustments:

   • Monitor the performance and behavior of your application in the new setup. If you encounter specific errors or performance issues, adjustments might be necessary, possibly involving further customizations or optimizations in the build process.

Please let us know how this solution works for you, and if there are any other challenges you face. Your feedback will be invaluable in enhancing support for such configurations in future releases.

[DZ9]

@piotrm-nvidia Thanks for your help! I followed your instructions to build the wheel，but It failed at the last step, and I didn't get the wheel file. Error response from daemon: bad parameter: pathcannot be empty

The full error log is below:

[notice] To update, run: pip install --upgrade pip
+ docker cp /mnt/pytriton/scripts/auditwheel_patched.py b5f669ec719c72dbccc55670ed469b7222ab7e1580bc805bac1291b0c7c3bde7:/tmp/
Successfully copied 3.07kB to b5f669ec719c72dbccc55670ed469b7222ab7e1580bc805bac1291b0c7c3bde7:/tmp/
+ docker exec b5f669ec719c72dbccc55670ed469b7222ab7e1580bc805bac1291b0c7c3bde7 bash -c 'LD_LIBRARY_PATH=/pytriton/tritonserver/external_libs /tmp/auditwheel_patched.py -vvvv repair --plat manylinux_2_35_x86_64 /mnt/pytriton/dist/nvidia_pytriton-0.5.1.dev0+g8de96ba.d20240423-py3-none-linux_x86_64.whl'
INFO:auditwheel.main_repair:Repairing nvidia_pytriton-0.5.1.dev0+g8de96ba.d20240423-py3-none-linux_x86_64.whl
INFO:auditwheel.main_repair:This does not look like a platform wheel, no ELF executable or shared library file (including compiled Python C extension) found in the wheel archive
++ dirname /mnt/pytriton/dist
+ WHEELHOUSE_DIR=/mnt/pytriton/wheelhouse
++ docker exec b5f669ec719c72dbccc55670ed469b7222ab7e1580bc805bac1291b0c7c3bde7 bash -c 'find /mnt/pytriton/wheelhouse -type f -name *.whl'
+ RESULT_WHEEL_PATH=
+ docker cp b5f669ec719c72dbccc55670ed469b7222ab7e1580bc805bac1291b0c7c3bde7: /mnt/pytriton/dist
Error response from daemon: bad parameter: pathcannot be empty
find ./dist -iname *-linux*.whl -type f -delete
ls -lh dist
total 1.1M
-rw-r--r-- 1 root root 1.1M Apr 23 16:43 nvidia_pytriton-0.5.1.dev0+g8de96ba.d20240423.tar.gz
twine check dist/*
Checking dist/nvidia_pytriton-0.5.1.dev0+g8de96ba.d20240423.tar.gz: PASSED

[github-actions[bot]]

This issue is stale because it has been open 21 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[jkosek]

Hi @DZ9. We have updated our scripts to automatically select the glibc version based on used container. Could you try the build process again?

make install-dev
make dist -e TRITONSERVER_IMAGE_VERSION=23.04 -e PYTHON_VERSIONS="3.10"

[github-actions[bot]]

This issue is stale because it has been open 21 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 7 days with no activity.