search

Yoctol / strpipe

text preprocessing pipeline
Other
5 stars 0 forks source link

reversible add sos eos and return sentlen after add sos eos #49

Closed SoluMilken closed 5 years ago

stegben commented 5 years ago

sos and eos token should be in different op, since it's not relevant to padding

SoluMilken commented 5 years ago

Can I split it into two ops in the next pr?