Move tokenizer_data_utils.py from /data to /utils with the rest of the utils.
Update imports so function calls change from tokenizer_data_utils.tokenizer_and_embedding_resize to tokenizer_and_embedding_resize.
Add 3 unit tests:
Ensure adding special tokens works correctly
Ensure not adding special tokens doesn't modify tokenizer
Ensure input and output embeddings are resized properly
How to verify the PR
tox -e py
Was the PR tested
[x] I have added >=1 unit test(s) for every new method I have added.
Description of the change
Move
tokenizer_data_utils.py
from/data
to/utils
with the rest of the utils. Update imports so function calls change fromtokenizer_data_utils.tokenizer_and_embedding_resize
totokenizer_and_embedding_resize
. Add 3 unit tests:How to verify the PR
tox -e py
Was the PR tested