tidymodels / textrecipes

Extra recipes for Text Processing
https://textrecipes.tidymodels.org/
Other
160 stars 14 forks source link

Documentation: in `step_tokenize()`, `udpipe` engine also takes `training_options` #237

Open petrbouchal opened 1 year ago

petrbouchal commented 1 year ago

The problem

This is purely about documentation.

The documentation for step_tokenize(), under the parameter training_options says

A list of options passed to the tokenizer when it is being trained. Only applicable for engine == "tokenizers.bpe".

It also applies to udpipe as per https://www.emilhvitfeldt.com/post/textrecipes-version-0-4-0/.

It would be great if this could be added, otherwise operating udpipe via step_tokenize() is a bit of a mystery.

EmilHvitfeldt commented 1 year ago

Hello @petrbouchal 👋

This is very useful feedback! Please feel free to add other issues if other things are unclear!