New Model: Charformer: Fast Character Transformers via Gradient-based Subword Tokenization

[neel04]

🌟 New model addition

Model description

arXiv = https://arxiv.org/pdf/2106.12672.pdf (pre-print; under review)

In this paper, they introduce a soft gradient-based subword tokenization module (GBST) that automatically learns latent subword representations from characters in a data-driven fashion. More importantly, is the introduction of Charformer, a deep Transformer model that integrates GBST and operates on the byte level.

Via extensive experiments on English GLUE, multilingual, and noisy text datasets, we show that Charformer outperforms a series of competitive byte-level baselines while generally performing on par and sometimes outperforming subword-based models. Additionally, Charformer is fast, improving the speed of both vanilla byte-level and subword-level Transformers by 28%-100% while maintaining competitive quality. We believe this work paves the way for highly performant token-free models that are trained completely end-to-end.

Open source status

• [ ] the model implementation is available: ----] Implementation and weights
• [ ] the model weights are available: ------------] to be released soon here
• [x] who are the authors: Yi Tay, Vinh Q. Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara BahriZhen Qin, Simon Baumgartner, Cong Yu, Donald Metzler (Google and DeepMind-->``)

[stefan-it]

Code is out now:

https://github.com/google-research/google-research/tree/master/charformer (please note the different url - compared to paper url)

[mapmeld]

An unofficial PyTorch implementation for Charformer https://github.com/lucidrains/charformer-pytorch

[MartinXPN]

Thanks for the great work! Will charformer be supported in the near future?

[MilesQLi]

Still not supported yet