fix: enable use of pretokenized dataset with padding_free

[HarikrishnanBalagopal]

Description of the change

Related issue number

How to verify the PR

Was the PR tested

• [ ] I have added >=1 unit test(s) for every new method I have added.
• [ ] I have ensured all unit tests pass

[github-actions[bot]]

Thanks for making a pull request! 😃 One of the maintainers will review and advise on the next steps.

[kmehant]

Cherry picked the fix