pytorch / torcharrow

High performance model preprocessing library on PyTorch

https://pytorch.org/torcharrow/beta/index.html

BSD 3-Clause "New" or "Revised" License

645 stars 79 forks source link

Create add_tokens UDF #384

Closed Nayef211 closed 2 years ago

Nayef211 commented 2 years ago

Description

Adding and registering UDF for add_tokens which can operate on list of int and str

Testing

pytest torcharrow/test/transformation/test_text_ops.py

facebook-github-bot commented 2 years ago

@Nayef211 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot commented 2 years ago

@Nayef211 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot commented 2 years ago

@Nayef211 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot commented 2 years ago

@Nayef211 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

facebook-github-bot commented 2 years ago

This pull request was exported from Phabricator. Differential Revision: D37315924

facebook-github-bot commented 2 years ago

This pull request was exported from Phabricator. Differential Revision: D37315924