update pad token id to use tokenziers padding token - Githubissues

sambanova / generative_data_prep

Apache License 2.0

58 stars 8 forks source link

update pad token id to use tokenziers padding token #76

Closed snova-zoltanc closed 9 months ago

snova-zoltanc commented 10 months ago

Please refer to CONTRIBUTING.md for detailed instructions.

Summary

PR Checklist

[ ] My PR is less than 500 lines of code
[ ] I have added sufficient comment as docstrings in my code
[ ] I have made corresponding changes to the documentation
[ ] I have written unit-tests to test all of my code