codecov[bot]
commented
11 months ago Codecov Report
Attention: 41 lines in your changes are missing coverage. Please review.
Comparison is base (
91a3fe0) 46.52% compared to head (a9bcc53) 60.89%.
Additional details and impacted files
```diff @@ Coverage Diff @@ ## 0.3 #161 +/- ## =========================================== + Coverage 46.52% 60.89% +14.37% =========================================== Files 85 85 Lines 4400 4547 +147 =========================================== + Hits 2047 2769 +722 + Misses 2353 1778 -575 ```:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.
This PR reworked the text encoder interface. The previous
TransformerTextEncoder,BertTextEncoder,GPT2TextEncoder, andT5TextEncoderare unified intoTrfTextEncoder.TrfTextEncoderhas multiple fields that can modify the encode/decode process:annotate(default toTextEncoders.annotate_strings): Annotate the input string for the tokenizer, e.g.Stringwould be treated as a single sentence, not a single word.process: The preprocess function applied to the tokenization results, e.g. adding specialend-of-sentencetoken, computing attention mask...onehot(default toTextEncoders.lookup_fist): Apply onehot encoding on the preprocess result, the default behavior takes the first element from the proprocess result and applies onehot encoding.decode(default toidentity): The function that converts each token id back to string. This can be used to handle some tokenizers that use a different set of vocabulary such as gpt2's byte-level vocabulary.textprocess(default toTextEncodeBase.join_text): the function that joins thedecode-d result in complete sentence(s).A new api
decode_textis also provided to simplify text generation. These designs allows us to unify the behavior difference between the old<X>TextEncoders, and extract the text decoder direclty from huggingface tokenizer file.