Open Niklas2501 opened 1 year ago
Can you try modifying the server resource further to add the needed envVar?, e.g. the triton container (for example) this adds requests but you could also add an envar.
Can you try modifying the server resource further to add the needed envVar?, e.g. the triton container (for example) this adds requests but you could also add an envar.
Can you elaborate what i should add as a env var?
I tried to use "/var/models/add20/conda-pack.tar.gz", i.e. the path in the PV, similar to to the model URI but rclone responded with an error message, see screenshot.
Can you check the triton container - is this an error from triton when its trying to use the EXECUTION_ENV_PATH? Here it says local folders are permitted: https://github.com/triton-inference-server/python_backend#important-notes. Can you verify the envVar has been added to the triton pod? If not can you show yoru latest server.yaml?
Before referring to your specific questions: In the link you posted the docs stated to used this to set the EXECUTION_ENV_PATH like this if it is in the model directory:
parameters: { key: "EXECUTION_ENV_PATH", value: {string_value: "$$TRITON_MODEL_DIRECTORY/conda-pack.tar.gz"} }
but this also leads to a "crash" (going into not ready) of the triton server with the same "describe model" output:
Spec:
Requirements:
python
pvc
Storage Uri: /var/models/add20
Status:
Conditions:
Last Transition Time: 2023-01-24T10:51:00Z
Reason: rpc error: code = InvalidArgument desc = load failed for model 'add20_1': version 1 is at UNAVAILABLE state: Internal: Failed to get the canonical path for /mnt/agent/models/add20_1/conda-pack.tar.gz.;
Status: False
Type: ModelReady
Last Transition Time: 2023-01-24T10:51:00Z
Reason: rpc error: code = InvalidArgument desc = load failed for model 'add20_1': version 1 is at UNAVAILABLE state: Internal: Failed to get the canonical path for /mnt/agent/models/add20_1/conda-pack.tar.gz.;
Status: False
Type: Ready
Replicas: 1
Events: <none>
This is the corresponding log of the triton container:
I0124 10:51:00.308943 1 grpc_server.cc:270] Process for RepositoryModelLoad, rpc_ok=1, 0 step START
I0124 10:51:00.308983 1 grpc_server.cc:225] Ready for RPC 'RepositoryModelLoad', 1
I0124 10:51:00.311559 1 model_config_utils.cc:646] Server side auto-completed config: name: "add20_1"
input {
name: "INPUT"
data_type: TYPE_FP32
dims: 4
}
output {
name: "OUTPUT"
data_type: TYPE_FP32
dims: 4
}
instance_group {
kind: KIND_CPU
}
default_model_filename: "model.py"
parameters {
key: "EXECUTION_ENV_PATH"
value {
string_value: "$$TRITON_MODEL_DIRECTORY/conda-pack.tar.gz"
}
}
backend: "python"
I0124 10:51:00.312534 1 model_lifecycle.cc:459] loading: add20_1:1
I0124 10:51:00.313322 1 backend_model.cc:308] Adding default backend config setting: default-max-batch-size,4
I0124 10:51:00.313369 1 shared_library.cc:108] OpenLibraryHandle: /opt/tritonserver/backends/python/libtriton_python.so
I0124 10:51:00.325310 1 python_be.cc:1612] 'python' TRITONBACKEND API version: 1.10
I0124 10:51:00.325347 1 python_be.cc:1634] backend configuration:
{"cmdline":{"auto-complete-config":"true","backend-directory":"/opt/tritonserver/backends","min-compute-capability":"6.000000","shm-default-byte-size":"16777216","default-max-batch-size":"4"}}
I0124 10:51:00.325392 1 python_be.cc:1764] Shared memory configuration is shm-default-byte-size=16777216,shm-growth-byte-size=67108864,stub-timeout-seconds=30
I0124 10:51:00.325480 1 python_be.cc:2010] TRITONBACKEND_GetBackendAttribute: setting attributes
I0124 10:51:00.325930 1 python_be.cc:1812] TRITONBACKEND_ModelInitialize: add20_1 (version 1)
I0124 10:51:00.327353 1 model_config_utils.cc:1838] ModelConfig 64-bit fields:
I0124 10:51:00.327361 1 model_config_utils.cc:1840] ModelConfig::dynamic_batching::default_queue_policy::default_timeout_microseconds
I0124 10:51:00.327363 1 model_config_utils.cc:1840] ModelConfig::dynamic_batching::max_queue_delay_microseconds
I0124 10:51:00.327365 1 model_config_utils.cc:1840] ModelConfig::dynamic_batching::priority_queue_policy::value::default_timeout_microseconds
I0124 10:51:00.327367 1 model_config_utils.cc:1840] ModelConfig::ensemble_scheduling::step::model_version
I0124 10:51:00.327368 1 model_config_utils.cc:1840] ModelConfig::input::dims
I0124 10:51:00.327370 1 model_config_utils.cc:1840] ModelConfig::input::reshape::shape
I0124 10:51:00.327372 1 model_config_utils.cc:1840] ModelConfig::instance_group::secondary_devices::device_id
I0124 10:51:00.327373 1 model_config_utils.cc:1840] ModelConfig::model_warmup::inputs::value::dims
I0124 10:51:00.327375 1 model_config_utils.cc:1840] ModelConfig::optimization::cuda::graph_spec::graph_lower_bound::input::value::dim
I0124 10:51:00.327377 1 model_config_utils.cc:1840] ModelConfig::optimization::cuda::graph_spec::input::value::dim
I0124 10:51:00.327379 1 model_config_utils.cc:1840] ModelConfig::output::dims
I0124 10:51:00.327381 1 model_config_utils.cc:1840] ModelConfig::output::reshape::shape
I0124 10:51:00.327383 1 model_config_utils.cc:1840] ModelConfig::sequence_batching::direct::max_queue_delay_microseconds
I0124 10:51:00.327386 1 model_config_utils.cc:1840] ModelConfig::sequence_batching::max_sequence_idle_microseconds
I0124 10:51:00.327388 1 model_config_utils.cc:1840] ModelConfig::sequence_batching::oldest::max_queue_delay_microseconds
I0124 10:51:00.327390 1 model_config_utils.cc:1840] ModelConfig::sequence_batching::state::dims
I0124 10:51:00.327392 1 model_config_utils.cc:1840] ModelConfig::sequence_batching::state::initial_state::dims
I0124 10:51:00.327393 1 model_config_utils.cc:1840] ModelConfig::version_policy::specific::versions
I0124 10:51:00.329043 1 python_be.cc:1503] Using Python execution env /mnt/agent/models/add20_1/conda-pack.tar.gz
I0124 10:51:00.329268 1 python_be.cc:1835] TRITONBACKEND_ModelFinalize: delete model state
E0124 10:51:00.329277 1 model_lifecycle.cc:597] failed to load 'add20_1' version 1: Internal: Failed to get the canonical path for /mnt/agent/models/add20_1/conda-pack.tar.gz.
I0124 10:51:00.330481 1 grpc_server.cc:270] Process for RepositoryModelLoad, rpc_ok=1, 0 step WRITEREADY
I0124 10:51:00.330604 1 grpc_server.cc:270] Process for RepositoryModelLoad, rpc_ok=1, 0 step COMPLETE
I0124 10:51:00.330617 1 grpc_server.cc:411] Done for RepositoryModelLoad, 0
I0124 10:51:04.463551 1 http_server.cc:3372] HTTP request: 0 /v2/health/ready
I0124 10:51:09.463787 1 http_server.cc:3372] HTTP request: 0 /v2/health/ready
I0124 10:51:09.463948 1 http_server.cc:3372] HTTP request: 0 /v2/health/live
Log of the agent container:
time="2023-01-24T10:50:59Z" level=info msg="Received operation" Name=Client
time="2023-01-24T10:50:59Z" level=info msg="calling load model" Name=Client
time="2023-01-24T10:50:59Z" level=info msg="Load model add20:1" Name=Client func=LoadModel
time="2023-01-24T10:50:59Z" level=debug msg="running with model add20_1:1 srcUri /var/models/add20" Name=V2ModelRepository func=DownloadModelVersion
time="2023-01-24T10:50:59Z" level=info msg="Copy from /var/models/add20 (original /var/models/add20) to /mnt/agent/rclone/3735392888" Source=RCloneClient
time="2023-01-24T10:50:59Z" level=info msg="Calling Rclone server: /sync/copy with {\"srcFs\":\"/var/models/add20\",\"dstFs\":\"/mnt/agent/rclone/3735392888\",\"createEmptySrcDirs\":true}" Source=RCloneClient
time="2023-01-24T10:51:00Z" level=info msg="rclone response: {}\n" Source=RCloneClient
time="2023-01-24T10:51:00Z" level=debug msg="Found model add20_1:1 artifactVersion 0 for /var/models/add20 at /mnt/agent/rclone/3735392888/1 " Name=V2ModelRepository func=DownloadModelVersion
time="2023-01-24T10:51:00Z" level=info msg="Calling Rclone server: /operations/purge with {\"fs\":\"/mnt/agent/rclone/3735392888\",\"remote\":\"\"}" Source=RCloneClient
time="2023-01-24T10:51:00Z" level=info msg="rclone response: {}\n" Source=RCloneClient
time="2023-01-24T10:51:00Z" level=info msg="Chose path /mnt/agent/rclone/3735392888/1 for model add20:1" Name=Client func=LoadModel
time="2023-01-24T10:51:00Z" level=debug msg="Loading model add20_1" Source=StateManager
time="2023-01-24T10:51:00Z" level=debug msg="model: add20_1, avail: 1073741824, required 0" Source=StateManager
time="2023-01-24T10:51:00Z" level=debug msg="Before memory update 1073741824, 0" Source=StateManager
time="2023-01-24T10:51:00Z" level=debug msg="After memory update 1073741824, 0" Source=StateManager
time="2023-01-24T10:51:00Z" level=debug msg="Before memory update 1073741824, 0" Source=StateManager
time="2023-01-24T10:51:00Z" level=debug msg="After memory update 1073741824, 0" Source=StateManager
time="2023-01-24T10:51:00Z" level=error msg="Failed to handle load model add20:1" Name=Client error="rpc error: code = InvalidArgument desc = load failed for model 'add20_1': version 1 is at UNAVAILABLE state: Internal: Failed to get the canonical path for /mnt/agent/models/add20_1/conda-pack.tar.gz.;\n"
2023/01/24 10:51:00 max retry time elapsed: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup seldon-collector on 10.43.0.10:53: server misbehaving"
It seems like adding EXECUTION_ENV_PATH as env var in the server.yaml also did not help - same problem. (Whitespace added by me)
NPP_VERSION=11.8.0.86
SHELL=/bin/bash
KUBERNETES_SERVICE_PORT_HTTPS=443
NVIDIA_VISIBLE_DEVICES=all
DALI_BUILD=5920076
SELDON_MESH_PORT_9003_TCP_PORT=9003
KUBERNETES_SERVICE_PORT=443
SELDON_MESH_PORT_80_TCP_PROTO=tcp
CUSOLVER_VERSION=11.4.1.48
SELDON_MESH_PORT_9003_TCP=tcp://10.43.220.150:9003
SELDON_SCHEDULER_SERVICE_HOST=10.43.251.151
SELDON_SCHEDULER_PORT_9055_TCP_PROTO=tcp
CUBLAS_VERSION=11.11.3.6
HOSTNAME=triton-pvc-0
DCGM_VERSION=2.2.9
SERVER_MODELS_DIR=/mnt/agent/models
SELDON_SCHEDULER_PORT_9005_TCP=tcp://10.43.251.151:9005
CUFFT_VERSION=10.9.0.58
NVIDIA_REQUIRE_CUDA=cuda>=9.0
SELDON_SCHEDULER_SERVICE_PORT_DATAFLOW=9008
SELDON_SCHEDULER_PORT_9008_TCP_ADDR=10.43.251.151
SELDON_SCHEDULER_PORT_9055_TCP_PORT=9055
SELDON_SCHEDULER_PORT_9004_TCP_ADDR=10.43.251.151
CUDA_CACHE_DISABLE=1
SELDON_SCHEDULER_PORT_9004_TCP=tcp://10.43.251.151:9004
NCCL_VERSION=2.15.5
SELDON_SCHEDULER_SERVICE_PORT_AGENT_MTLS=9055
CUSPARSE_VERSION=11.7.5.86
ENV=/etc/shinit_v2
PWD=/opt/tritonserver
SELDON_SCHEDULER_PORT_9002_TCP_PORT=9002
OPENUCX_VERSION=1.14.0
SELDON_SCHEDULER_PORT_9002_TCP=tcp://10.43.251.151:9002
NSIGHT_SYSTEMS_VERSION=2022.4.2.1
NVIDIA_DRIVER_CAPABILITIES=compute,utility,video
POLYGRAPHY_VERSION=0.42.1
TF_ENABLE_WINOGRAD_NONFUSED=1
TRT_VERSION=8.5.1.7
SELDON_SCHEDULER_PORT_9055_TCP_ADDR=10.43.251.151
SELDON_SCHEDULER_PORT_9044_TCP_PORT=9044
SELDON_SCHEDULER_PORT=tcp://10.43.251.151:9002
NVIDIA_PRODUCT_NAME=Triton Server
RDMACORE_VERSION=36.0
SELDON_SCHEDULER_SERVICE_PORT_XDS=9002
SELDON_SCHEDULER_PORT_9044_TCP_ADDR=10.43.251.151
HOME=/home/triton-server
SELDON_SCHEDULER_PORT_9005_TCP_PORT=9005
KUBERNETES_PORT_443_TCP=tcp://10.43.0.1:443
CUDA_VERSION=11.8.0.065
SELDON_SCHEDULER_SERVICE_PORT_SCHEDULER_MTLS=9044
SELDON_SCHEDULER_PORT_9044_TCP=tcp://10.43.251.151:9044
CURAND_VERSION=10.3.0.86
SELDON_SCHEDULER_PORT_9004_TCP_PROTO=tcp
SELDON_SCHEDULER_PORT_9055_TCP=tcp://10.43.251.151:9055
SELDON_MESH_SERVICE_PORT_DATA=80
CUTENSOR_VERSION=1.6.1.5
TRITON_SERVER_GPU_ENABLED=1
HPCX_VERSION=2.12.2tp1
SELDON_MESH_SERVICE_PORT_ADMIN=9003
SELDON_SCHEDULER_SERVICE_PORT_AGENT=9005
SELDON_SCHEDULER_SERVICE_PORT=9002
SERVER_GRPC_PORT=9500
SELDON_SCHEDULER_SERVICE_PORT_SCHEDULER=9004
SELDON_SCHEDULER_PORT_9008_TCP=tcp://10.43.251.151:9008
TERM=xterm-256color
TRITON_SERVER_VERSION=2.28.0
SELDON_MESH_PORT_80_TCP_PORT=80
GDRCOPY_VERSION=2.3
OPENMPI_VERSION=4.1.4
NVJPEG_VERSION=11.9.0.86
LIBRARY_PATH=/usr/local/cuda/lib64/stubs:
SELDON_SCHEDULER_PORT_9002_TCP_PROTO=tcp
SELDON_MESH_PORT_9003_TCP_PROTO=tcp
SHLVL=2
BASH_ENV=/etc/bash.bashrc
KUBERNETES_PORT_443_TCP_PROTO=tcp
TF_AUTOTUNE_THRESHOLD=2
SELDON_MESH_PORT=tcp://10.43.220.150:80
CUDNN_VERSION=8.7.0.80
KUBERNETES_PORT_443_TCP_ADDR=10.43.0.1
NSIGHT_COMPUTE_VERSION=2022.3.0.22
SELDON_MESH_PORT_9003_TCP_ADDR=10.43.220.150
SELDON_SCHEDULER_PORT_9004_TCP_PORT=9004
DALI_VERSION=1.18.0
NVIDIA_TRITON_SERVER_VERSION=22.11
LD_LIBRARY_PATH=/opt/tritonserver/backends/onnxruntime:/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
EXECUTION_ENV_PATH=$TRITON_MODEL_DIRECTORY/conda-pack.tar.gz
NVIDIA_BUILD_ID=48581224
SELDON_SCHEDULER_PORT_9005_TCP_PROTO=tcp
OMPI_MCA_coll_hcoll_enable=0
SELDON_SCHEDULER_PORT_9008_TCP_PROTO=tcp
OPAL_PREFIX=/opt/hpcx/ompi
KUBERNETES_SERVICE_HOST=10.43.0.1
CUDA_DRIVER_VERSION=520.61.05
SELDON_SCHEDULER_PORT_9044_TCP_PROTO=tcp
SELDON_MESH_PORT_80_TCP=tcp://10.43.220.150:80
KUBERNETES_PORT=tcp://10.43.0.1:443
KUBERNETES_PORT_443_TCP_PORT=443
_CUDA_COMPAT_PATH=/usr/local/cuda/compat
SELDON_SCHEDULER_PORT_9002_TCP_ADDR=10.43.251.151
NVIDIA_REQUIRE_JETPACK_HOST_MOUNTS=
PATH=/opt/tritonserver/bin:/usr/local/mpi/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/ucx/bin
TRITON_SERVER_USER=triton-server
SELDON_MESH_PORT_80_TCP_ADDR=10.43.220.150
SELDON_SCHEDULER_PORT_9005_TCP_ADDR=10.43.251.151
MOFED_VERSION=5.4-rdmacore36.0
TRTOSS_VERSION=22.10
SELDON_MESH_SERVICE_HOST=10.43.220.150
DEBIAN_FRONTEND=noninteractive
SELDON_MESH_SERVICE_PORT=80
TF_ADJUST_HUE_FUSED=1
TF_ADJUST_SATURATION_FUSED=1
TEST=test
SERVER_HTTP_PORT=9000
SELDON_SCHEDULER_PORT_9008_TCP_PORT=9008
_=/usr/bin/env
server.yaml used, with the env var added:
apiVersion: mlops.seldon.io/v1alpha1
kind: Server
metadata:
name: triton-pvc
namespace: seldon-mesh
spec:
serverConfig: triton
extraCapabilities:
- "pvc"
podSpec:
volumes:
- name: models-pvc
persistentVolumeClaim:
claimName: ml-models-pvc
containers:
- name: rclone
volumeMounts:
- name: models-pvc
mountPath: /var/models
- name: triton
env:
- name: TEST
value: "test"
- name: EXECUTION_ENV_PATH
value: "$$TRITON_MODEL_DIRECTORY/conda-pack.tar.gz"
Should you not try?:
- name: EXECUTION_ENV_PATH
value: "/var/models/conda-pack.tar.gz"
As said in their docs: If a non-$$TRITON_MODEL_DIRECTORY EXECUTION_ENV_PATH is used, only local file system paths are currently supported.
I don't think that would make sense. This would mean the triton server should look for the file at
(triton-container-root)/var/models/conda-pack.tar.gz
. I don't see a way the file would end up there.
While trying to find the root cause of the problem i noticed that unlike the other model files (i.e. the config.pbtxt and the model.py) the custom execution environment file is not copied / mounted to the same location by rclone
Content of the pv:
Content of the model directory in the triton container:
For normal model files rclone copies/mounts them from
(pv)/var/models/add20
to
(triton-container-root)/mnt/agent/models/add20_1/
Based on this, we should also see a file
(triton-container-root)/mnt/agent/models/add20_1/conda-pack.tar.gz
As one can see not only the custom execution environment file is not copied but also some other test files i included.
So i think there is an issue at the stage of copying/mounting the model files from the PV to the triton container and not an issue of the triton server not looking for the model at the correct location. Or am i missing something?
Might there be some kind of a filter or whitelist which files are copied by rclone that leads to conda-pack.tar.gz not being copied?
OK this makes sense as the issue. At present the logic is to copy the version folder and then just copy the config.pbtxt. But this seems a bug for above reason. Instead we should copy all files from top level folder and the version folder found.
So you also don't need the envVar as this is model specific and need to update the config.pbtxt as discussed in Triton docs.
We will need to look into a PR to change how files are copied across.
Ok, thanks for your update and of course for you patient help!
Describe the bug
Can't copy a custom execution environment file to the Triton server when trying to deploy a model via a PVC. How to set the EXECUTION_ENV_PATH parameter in the config.pbtxt?
To reproduce
Hey, I'm trying to deploy a python model to a Triton servery via a PVC
Now I can't figure out how to set the EXECUTION_ENV_PATH parameter in the config.pbtxt such that the custom execution environment file (e.g. conda-pack.tar.gz) can be copied by rclone and is also detected by the triton server?
As always, thank you very much for your help!
Expected behaviour
The custom execution environment file should be copied to the server and the model should be deployed successfully.
Environment
Proivder: Rancher Desktop Architekture: ARM64 / M1 Mac Kubernetes Cluster Version: Client Version: v1.25.2 Kustomize Version: v4.5.7 Server Version: v1.25.4+k3s1 Deployed Seldon System Images: image: docker.io/seldonio/seldonv2-controller:latest
Model Details
model.yaml
server.yaml