search

lopenling / Requests

For managing requests and everything related with requests
0 stars 0 forks source link

[RFW0006] Translatable Segments to Tokens #6

Closed mikkokotila closed 1 year ago

mikkokotila commented 1 year ago

A tokenizer without settings that takes in a segment of text, and breaks it down into tokens.