foundation-model-stack / fms-hf-tuning

🚀 Collection of tuning recipes with HuggingFace SFTTrainer and PyTorch FSDP.
Apache License 2.0
28 stars 48 forks source link

refactor: move tokenizer_data_utils with the rest of utils, add further unit testing. #348

Closed willmj closed 1 month ago

willmj commented 2 months ago

Description of the change

Move tokenizer_data_utils.py from /data to /utils with the rest of the utils. Update imports so function calls change from tokenizer_data_utils.tokenizer_and_embedding_resize to tokenizer_and_embedding_resize. Add 3 unit tests:

How to verify the PR

tox -e py

Was the PR tested

github-actions[bot] commented 2 months ago

Thanks for making a pull request! 😃 One of the maintainers will review and advise on the next steps.

aluu317 commented 1 month ago

Can you resolve the conflicts? I think it might need some code changes