indrajit96
commented
6 months ago Hi @yutkin, Thanks for reaching out. I have filed a bug https://jirasw.nvidia.com/browse/DLIS-6362 for the same.
Open yutkin opened 6 months ago
indrajit96
commented
6 months ago Hi @yutkin, Thanks for reaching out. I have filed a bug https://jirasw.nvidia.com/browse/DLIS-6362 for the same.
Description We have a PyTorch model that we serve on the CPU using the triton inference server. We use the POLL approach to monitor the new model versions in the GCS bucket. We also use the gRPC protocol and proxyless load balancing with the Traffic Director to balance the load across triton pods.
When triton loads a new model from the GCS bucket, it sometimes fails with a Segmentation fault error. Example log:
The
assertion failed: prior > 0line comes from thesync.cc:105file which is probably the file from gRPC.Triton Information
nvcr.io/nvidia/tritonserver:24.02-pyt-python-py3To Reproduce Unfortunately, we don't know how to reproduce it.
The model config
```pbtxt name: "combined_model_new" platform: "pytorch_libtorch" input [ { name: "A" data_type: TYPE_INT64 dims: -1 }, { name: "B" data_type: TYPE_INT64 dims: -1 }, { name: "C" data_type: TYPE_INT64 dims: -1 }, { name: "D", data_type: TYPE_INT64 dims: -1 }, { name: "E", data_type: TYPE_INT64 dims: -1 }, { name: "F", data_type: TYPE_FP64 dims: -1 }, { name: "G" data_type: TYPE_STRING dims: -1 }, { name: "H" data_type: TYPE_STRING dims: -1 }, { name: "I" data_type: TYPE_STRING dims: -1 }, { name: "J" data_type: TYPE_STRING dims: -1 }, { name: "K" data_type: TYPE_STRING dims: -1 }, { name: "L" data_type: TYPE_STRING dims: -1 } ] output { name: "OUT1" data_type: TYPE_FP32 dims: -1 } output { name: "OUT2" data_type: TYPE_FP32 dims: -1 } dynamic_batching { } parameters { key: "INFERENCE_MODE" value { string_value: "true" } } instance_group [ { count: 5 kind: KIND_CPU } ] backend: "pytorch" model_warmup [{ name: "random sample" count: 100 batch_size: 1 inputs { key: "A" value: { data_type: TYPE_INT64 dims: [1] random_data: True } } inputs { key: "B" value: { data_type: TYPE_INT64 dims: [1] random_data: True } } inputs { key: "C" value: { data_type: TYPE_INT64 dims: [1] random_data: True } } inputs { key: "D" value: { data_type: TYPE_INT64 dims: [1] random_data: True } } inputs { key: "E" value: { data_type: TYPE_INT64 dims: [1] random_data: True } } inputs { key: "F" value: { data_type: TYPE_FP64 dims: [1] random_data: True } } inputs { key: "G" value: { data_type: TYPE_STRING dims: [1] random_data: True } } inputs { key: "H" value: { data_type: TYPE_STRING dims: [1] random_data: True } } inputs { key: "I" value: { data_type: TYPE_STRING dims: [1] random_data: True } } inputs { key: "J" value: { data_type: TYPE_STRING dims: [1] random_data: True } } inputs { key: "K" value: { data_type: TYPE_STRING dims: [1] random_data: True } } inputs { key: "L" value: { data_type: TYPE_STRING dims: [1] random_data: True } } }] ```
Expected behavior The models should be updated flawlessly without segfaults.