triton-inference-server / model_analyzer

Triton Model Analyzer is a CLI tool to help with better understanding of the compute and memory requirements of the Triton Inference Server models.
Apache License 2.0
423 stars 74 forks source link

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf8 in position 0: invalid start byte #931

Open Anas0x45 opened 3 weeks ago

Anas0x45 commented 3 weeks ago

Description I encountered a UnicodeDecodeError when running the Triton Model Analyzer in a Docker container with GPU support. The issue seems to occur when retrieving the GPU device name. Here’s the traceback: Traceback (most recent call last): File "/usr/local/bin/model-analyzer", line 8, in sys.exit(main()) File "/usr/local/lib/python3.10/dist-packages/model_analyzer/entrypoint.py", line 263, in main gpus = GPUDeviceFactory().verify_requested_gpus(config.gpus) File "/usr/local/lib/python3.10/dist-packages/model_analyzer/device/gpu_device_factory.py", line 39, in init self.init_all_devices() File "/usr/local/lib/python3.10/dist-packages/model_analyzer/device/gpu_device_factory.py", line 71, in init_all_devices device_name = device_atrributes.deviceName File "/usr/local/lib/python3.10/dist-packages/model_analyzer/monitor/dcgm/dcgm_structs.py", line 464, in getattribute return value.decode("utf-8") UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf8 in position 0: invalid start byte

Triton Information Docker image: nvcr.io/nvidia/tritonserver:24.07-py3-sdk

To Reproduce

Solution that works for me: Adding errors="ignore" or errors="replace" to the decoding logic temporarily fixes the crash but may cause data loss.

class _DcgmStructure(Structure):

    def __getattribute__(self, key):
        value = super().__getattribute__(key)
        if isinstance(value, bytes):
            try:
                return value.decode("utf-8")
            except UnicodeDecodeError as e:
                print(f"Failed to decode bytes: {e}")
                return value.decode("utf-8", errors="replace")
        if isclass(value):
            return _WrappedStructure(value)
        return value
...