Description
I encountered a UnicodeDecodeError when running the Triton Model Analyzer in a Docker container with GPU support. The issue seems to occur when retrieving the GPU device name. Here’s the traceback:
Traceback (most recent call last):
File "/usr/local/bin/model-analyzer", line 8, in
sys.exit(main())
File "/usr/local/lib/python3.10/dist-packages/model_analyzer/entrypoint.py", line 263, in main
gpus = GPUDeviceFactory().verify_requested_gpus(config.gpus)
File "/usr/local/lib/python3.10/dist-packages/model_analyzer/device/gpu_device_factory.py", line 39, in init
self.init_all_devices()
File "/usr/local/lib/python3.10/dist-packages/model_analyzer/device/gpu_device_factory.py", line 71, in init_all_devices
device_name = device_atrributes.deviceName
File "/usr/local/lib/python3.10/dist-packages/model_analyzer/monitor/dcgm/dcgm_structs.py", line 464, in getattribute
return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf8 in position 0: invalid start byte
Triton Information
Docker image: nvcr.io/nvidia/tritonserver:24.07-py3-sdk
To Reproduce
Running Triton Model Analyzer in Docker with GPU support (docker run --gpus all).
The error occurs when retrieving the GPU device name using dcgm_structs.py.
Solution that works for me:
Adding errors="ignore" or errors="replace" to the decoding logic temporarily fixes the crash but may cause data loss.
class _DcgmStructure(Structure):
def __getattribute__(self, key):
value = super().__getattribute__(key)
if isinstance(value, bytes):
try:
return value.decode("utf-8")
except UnicodeDecodeError as e:
print(f"Failed to decode bytes: {e}")
return value.decode("utf-8", errors="replace")
if isclass(value):
return _WrappedStructure(value)
return value
...
Description I encountered a
sys.exit(main())
File "/usr/local/lib/python3.10/dist-packages/model_analyzer/entrypoint.py", line 263, in main
gpus = GPUDeviceFactory().verify_requested_gpus(config.gpus)
File "/usr/local/lib/python3.10/dist-packages/model_analyzer/device/gpu_device_factory.py", line 39, in init
self.init_all_devices()
File "/usr/local/lib/python3.10/dist-packages/model_analyzer/device/gpu_device_factory.py", line 71, in init_all_devices
device_name = device_atrributes.deviceName
File "/usr/local/lib/python3.10/dist-packages/model_analyzer/monitor/dcgm/dcgm_structs.py", line 464, in getattribute
return value.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf8 in position 0: invalid start byte
UnicodeDecodeError
when running the Triton Model Analyzer in a Docker container with GPU support. The issue seems to occur when retrieving the GPU device name. Here’s the traceback: Traceback (most recent call last): File "/usr/local/bin/model-analyzer", line 8, inTriton Information Docker image: nvcr.io/nvidia/tritonserver:24.07-py3-sdk
To Reproduce
docker run --gpus all
).dcgm_structs.py
.Solution that works for me: Adding
errors="ignore"
orerrors="replace"
to the decoding logic temporarily fixes the crash but may cause data loss.