search

alpheios-project / tokenizer

Alpheios Tokenizer Service
1 stars 0 forks source link

id handling #16

Closed balmas closed 3 years ago

balmas commented 3 years ago

we need to decide where token and segment ids are assigned (client, tokenization service, other service)

balmas commented 3 years ago

decision: tokenizer is not responsible for assigning PIDs for tokens it parses out of text.