Open HongchuanZeng opened 1 month ago
Hi - thanks for flagging this, it's something that I've done a bit of patching for but haven't found a fully generalizing solution. I'm unfortunately not able to devote much more time to improving this library, but I'd be happy to accept PR requests if you're able to make them!
Currently, the tokens are decoded one by one and shown on the interface. However, for some utf-8 symbols, if the corresponding tokens are decoded one by one, the result will be some unreadable codes which can not be correctly shown on the panel.