piotrm-nvidia
commented
6 months ago I tried replicating your issue with Python 3.11.7 and PyTriton 0.4.2 in a Docker container based on Ubuntu 22.04 but couldn't reproduce the error. Here's a concise summary of my setup process and testing, which you might find helpful for cross-checking against your setup:
-
Docker Initialization:
- Command:
docker run -ti ... ubuntu:22.04 bash - This sets up a standard Ubuntu 22.04 environment with necessary configurations for PyTriton.
- Command:
-
Miniconda & Python Setup:
- Installed necessary dependencies (
git, Python libraries). - Installed Miniconda (latest version) and set up a Python 3.11.7 environment.
- Ensure to follow the PyTriton installation guide, which has additional steps compared to the standard Miniconda documentation.
- Installed necessary dependencies (
-
PyTriton Installation:
- Command:
pip install "nvidia-pytriton==0.4.2" - Make sure your
LD_LIBRARY_PATHincludes the Conda libraries.
- Command:
-
Testing the Model:
- Created a sample Python script (
script.py) with a basic Triton server setup and a mock inference function. - Ran the script in the background and monitored the Triton server's output.
- Created a sample Python script (
-
Client Setup & Test:
- Installed Pillow and created a test image.
- Used a Python script to send an inference request to the Triton server and received the expected output.
Could you please confirm if your setup process matches the above? Additionally, it would be helpful if you could provide:
- The exact steps you followed to set up your environment and run the model.
- Any additional configurations or dependencies specific to your setup.
- The complete code of the model and client if they differ from the standard samples.
This information will help in pinpointing the cause of the error you're encountering.
Full installation guide
Please find below full installation guide for PyTriton with miniconda I used to run successful test.
Start docker:
docker run -ti --network=host --platform linux/amd64 --ulimit core=-1 --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864 --cap-add=SYS_PTRACE --shm-size 2G ubuntu:22.04 bashInside docker install conda deps:
apt-get update && apt-get install -y gitInstall additional Python libraries necessary for PyTriton C++ backend (ADDED from guide):
apt install -y python3 python3-distutils python-is-python3 curlSet conda version and machine:
export CONDA_VERSION=latest
export TARGET_MACHINE=x86_64Download conda:
curl "https://repo.anaconda.com/miniconda/Miniconda3-${CONDA_VERSION}-Linux-${TARGET_MACHINE}.sh" --output miniconda.shRun miniconda installation. Approve all licenses and path as defaults:
bash miniconda.shAdd conda to path:
export PATH=~/miniconda3/bin/:$PATHInit bash configuration with conda:
conda init bashRun new bash interpreter to load profile:
bashCreate virtual environment with Python 3.11.7
conda create -c conda-forge -n venv python=3.11.7Activate new environment:
conda activate venvModify library paths to include conda folders:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CONDA_PREFIX/lib Install PyTriton:
pip install "nvidia-pytriton==0.4.2"Create script.py
import numpy as np
from pytriton.decorators import batch
@batch
def infer_fn(image):
assert image.shape == (1, 224, 224, 3), f"Expected image shape (224, 224, 3), got {image.shape}"
# This can implement actual inference logic for image classification
return [np.char.encode(["cat"], "utf-8")]
from pytriton.model_config import ModelConfig, Tensor
from pytriton.triton import Triton
# Connecting inference callable with Triton Inference Server
triton = Triton()
# Load model into Triton Inference Server
triton.bind(
model_name="ImageNet",
infer_func=infer_fn,
inputs=[
Tensor(name="image", dtype=np.uint8, shape=(224, 224, 3)),
],
outputs=[
Tensor(name="class", dtype=np.bytes_, shape=(-1,)),
],
config=ModelConfig(max_batch_size=2)
)
triton.serve()Run script.py with & to put it in the background and observe output from Triton process:
# python3 /home/piotrm/src/script.py &
[1] 3630
I0108 10:10:10.248543 3474 pinned_memory_manager.cc:241] Pinned memory pool is created at '0x7f8ecc000000' with size 268435456
I0108 10:10:10.248804 3474 cuda_memory_manager.cc:107] CUDA memory pool is created on device 0 with size 67108864
I0108 10:10:10.249335 3474 server.cc:592]
+------------------+------+
| Repository Agent | Path |
+------------------+------+
+------------------+------+
I0108 10:10:10.249350 3474 server.cc:619]
+---------+------+--------+
| Backend | Path | Config |
+---------+------+--------+
+---------+------+--------+
I0108 10:10:10.249360 3474 server.cc:662]
+-------+---------+--------+
| Model | Version | Status |
+-------+---------+--------+
+-------+---------+--------+
I0108 10:10:10.292109 3474 metrics.cc:817] Collecting metrics for GPU 0: NVIDIA RTX A6000
I0108 10:10:10.292324 3474 metrics.cc:710] Collecting CPU metrics
I0108 10:10:10.292460 3474 tritonserver.cc:2458]
+----------------------------------+------------------------------------------+
| Option | Value |
+----------------------------------+------------------------------------------+
| server_id | triton |
| server_version | 2.39.0 |
| server_extensions | classification sequence model_repository |
| | model_repository(unload_dependents) sch |
| | edule_policy model_configuration system_ |
| | shared_memory cuda_shared_memory binary_ |
| | tensor_data parameters statistics trace |
| | logging |
| model_repository_path[0] | /root/.cache/pytriton/workspace_5le8muxi |
| model_control_mode | MODE_EXPLICIT |
| strict_model_config | 0 |
| rate_limit | OFF |
| pinned_memory_pool_byte_size | 268435456 |
| cuda_memory_pool_byte_size{0} | 67108864 |
| min_supported_compute_capability | 6.0 |
| strict_readiness | 1 |
| exit_timeout | 30 |
| cache_enabled | 0 |
+----------------------------------+------------------------------------------+
I0108 10:10:10.294234 3474 grpc_server.cc:2513] Started GRPCInferenceService at 0.0.0.0:8001
I0108 10:10:10.294421 3474 http_server.cc:4497] Started HTTPService at 0.0.0.0:8000
I0108 10:10:10.338401 3474 http_server.cc:270] Started Metrics Service at 0.0.0.0:8002
I0108 10:10:11.437711 3474 model_lifecycle.cc:461] loading: ImageNet:1
I0108 10:10:12.754948 3474 python_be.cc:2199] TRITONBACKEND_ModelInstanceInitialize: ImageNet_0_0 (CPU device 0)
I0108 10:10:13.020888 3474 model_lifecycle.cc:818] successfully loaded 'ImageNet'
Install Pillow to prepare input for client:
pip install PillowStart python interpreter
pythonPaste in interpreter code to create random image:
import numpy as np
from PIL import Image
# Create a new images with the same size as the input data
test_img = Image.new("RGB", (224, 224))
# Get the pixels object of the image
pixels = test_img.load()
test_img.save("cat.jpg")Paste in intepreter code to send inference request
import numpy as np
from PIL import Image
from pytriton.client import ModelClient
# Create some input data as a numpy array of an image
img = Image.open("cat.jpg")
img = img.resize((224, 224))
input_data = np.array(img)
# Create a ModelClient object with the server address and model name
client = ModelClient("localhost:8000", "ImageNet")
# Call the infer_sample method with the input data
result_dict = client.infer_sample(input_data)Paste in interpreter code to print result:
result_dictExpected output:
{'class': b'cat'}
dogky123
github-actions[bot]
Description
Bug when upgrade to python 3.11.7. 3.11.5 is good.
Log
Environment