triton-inference-server / server

The Triton Inference Server provides an optimized cloud and edge inferencing solution.
https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/index.html
BSD 3-Clause "New" or "Revised" License
8.33k stars 1.48k forks source link

Can't load custom backend shared library from s3 (24.07) #7550

Open gerasim13 opened 2 months ago

gerasim13 commented 2 months ago

Description After updating my server to the new version (24.07-py3 tag), my custom backends have stopped working without a clear explanation of what's causing the issue.

Here is few lines from my log:

2024-08-20 13:21:03.116 | I0820 10:21:03.116129 1 model_lifecycle.cc:776] "failed to load 'vosk'" |  
2024-08-20 13:21:03.116 | E0820 10:21:03.116080 1 model_lifecycle.cc:641] "failed to load 'vosk' version 1: Not found: unable to load shared library: s3://https://xxx-xxx.xxx:9010/xxx-bucket-dev/models/xxx/vosk/1/libtriton_vosk.so: cannot open shared object file: No such file or directory"

Triton Information I am using the official Triton server docker image with the 24.07-py3 tag.

To Reproduce

import tritonclient.grpc as client
with client.InferenceServerClient(
    url='xxx-xxx.xxx:32003',
) as triton_client:
    triton_client.load_model('vosk')

Expected behavior The custom backend 'vosk' should be successfully loaded without any issues.

gerasim13 commented 2 months ago

The same model and backend work well when I put them on a regular filesystem.

gerasim13 commented 2 months ago

The previous version I was using (23.12-py3) loads custom backend shared libraries well from my S3 storage (based on Ceph).