search

InterDigitalInc / CompressAI

A PyTorch library and evaluation platform for end-to-end compression research
https://interdigitalinc.github.io/CompressAI/
BSD 3-Clause Clear License
1.17k stars 232 forks source link

'utf-8' UnicodeDecodeError: codec can't decode byte oxd6 in position 1: invalid continuation byte When export to ONNX #287

Open YuKDseele opened 5 months ago

YuKDseele commented 5 months ago

Hi, I want to convert the 'ga', 'gs', 'ha', 'hs', 'entropy_bottleneck.compress' and 'gaussian_conditional.compress' from a compressai model into ONNX for export. However, I encountered this problem: 'utf-8' UnicodeDecodeError: codec can't decode byte oxd6 in position 1: invalid continuation byte. I tried commenting out the ANS encoder, and it seems to work fine, but I need the bitstream output. How can I fix this problem?

YuKDseele commented 5 months ago

I have reviewed my code again and it appears that the automatic decoding of bytes during the ONNX export is a default behavior, which may cause the problem I would like to understand how to prevent this from happening.

StuDDXT1610 commented 1 week ago

I have reviewed my code again and it appears that the automatic decoding of bytes during the ONNX export is a default behavior, which may cause the problem I would like to understand how to prevent this from happening.

Hi, I had the same errors like yours ones, i tried to modify the output of function compress in Entropy Models class to make it return a Tensor instead of strings ( i use frombuffer function ), but the onnx model result was weird. Have you fix this problem, pls tell me

YuKDseele commented 6 days ago

I have reviewed my code again and it appears that the automatic decoding of bytes during the ONNX export is a default behavior, which may cause the problem I would like to understand how to prevent this from happening.

Hi, I had the same errors like yours ones, i tried to modify the output of function compress in Entropy Models class to make it return a Tensor instead of strings ( i use frombuffer function ), but the onnx model result was weird. Have you fix this problem, pls tell me

Unfortunately, I did not solve this issue, which may be due to onnx not supporting CPP code (the ans encoder is built using CPP). I suggest using this part as post-processing (i.e. generate inputs and indexes, and finally encoding directly using the ans encoder)