search

triton-inference-server / server

The Triton Inference Server provides an optimized cloud and edge inferencing solution.
https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/index.html
BSD 3-Clause "New" or "Revised" License
8.31k stars 1.48k forks source link

Python backend doesn't load model.py in the model directory properly when trailing slashes(/) present in --backend-directory option #6730

Open twjang opened 10 months ago

twjang commented 10 months ago

Description The Python backend does not properly load the model.py file in the model directory when trailing slashes (/) are present in the --backend-directory option.

Triton Information What version of Triton are you using? r23.12

Are you using the Triton container or did you build it yourself? container

To Reproduce Follow the instructions in https://github.com/triton-inference-server/python_backend?tab=readme-ov-file#quick-start

  1. Launch the Triton Docker image: 
    
docker run --shm-size=1g --ulimit memlock=-1 -p 8000:8000 -p 8001:8001 -p 8002:8002 --ulimit stack=67108864 -ti nvcr.io/nvidia/tritonserver:23.12-py3
2. Inside the docker container, clone `python_backend` repository and prepare the model repository directory:

git clone https://github.com/triton-inference-server/python_backend -b r23.12 cd python_backend mkdir -p models/add_sub/1/ cp examples/add_sub/model.py models/add_sub/1/model.py cp examples/add_sub/config.pbtxt models/add_sub/config.pbtxt


3. Start Triton server with the following command:

tritonserver --model-repository pwd/models --backend-directory /opt/tritonserver/backends/ --log-verbose 1

According to the log, `triton_python_backend_stub` is launched with following arguments:

I1222 03:27:07.578630 363 stub_launcher.cc:253] Starting Python backend stub: exec /opt/tritonserver/backends/python/triton_python_backend_stub /opt/tritonserver/python_backend/models/add_sub/1/model.py triton_python_backend_shm_region_1 1048576 1048576 363 /opt/tritonserver/backends/python 336 add_sub /opt/tritonserver/backends/python 

However, the last argument should be `DEFAULT`.

4. Successfully run Triton server with the following command:
Please be aware that the trailing `/` has been removed from the `--backend-directory` option.

tritonserver --model-repository pwd/models --backend-directory /opt/tritonserver/backends --log-verbose 1

Now, everything works fine.

I1222 03:29:44.179683 387 stub_launcher.cc:253] Starting Python backend stub: exec /opt/tritonserver/backends/python/triton_python_backend_stub /opt/tritonserver/python_backend/models/add_sub/1/model.py triton_python_backend_shm_region_2 1048576 1048576 387 /opt/tritonserver/backends/python 336 add_sub_0_0 DEFAULT 



**Expected behavior**
The presence of a trailing / in the  `--backend-directory` option should not affect its behavior.
Tabrizian commented 10 months ago

Thanks for reporting this issue. I have filed a bug for this issue.