Need support for byte-pair encoded utf-8 symbols

callummcdougall / sae_vis

Create feature-centric and prompt-centric visualizations for sparse autoencoders (like those from Anthropic's published research).

MIT License

128 stars 27 forks source link

Need support for byte-pair encoded utf-8 symbols #55

Open HongchuanZeng opened 1 month ago

HongchuanZeng commented 1 month ago

Currently, the tokens are decoded one by one and shown on the interface. However, for some utf-8 symbols, if the corresponding tokens are decoded one by one, the result will be some unreadable codes which can not be correctly shown on the panel.

callummcdougall commented 1 month ago

Hi - thanks for flagging this, it's something that I've done a bit of patching for but haven't found a fully generalizing solution. I'm unfortunately not able to devote much more time to improving this library, but I'd be happy to accept PR requests if you're able to make them!