Sometimes, I don't want to discard the remaining output, might be useful :)
So, this pull request adds a new feature to the token counting command-line interface. The new -s or --split option allows users to split their input text into multiple files, each containing a specific number of tokens.
Changes
A new option (-s, --split) has been added to the click.command() decorator in the cli function.
The cli function has been updated to handle this new option. When the split and truncate options are used together, the script splits the input tokens into groups of the specified size and writes each group to a separate file.
A new helper function grouper has been added to split an iterable into fixed-length chunks.
The split function now decodes each chunk before writing to the respective output file.
Sometimes, I don't want to discard the remaining output, might be useful :) So, this pull request adds a new feature to the token counting command-line interface. The new -s or --split option allows users to split their input text into multiple files, each containing a specific number of tokens.
Changes
click.command()
decorator in thecli
function.cli
function has been updated to handle this new option. When thesplit
andtruncate
options are used together, the script splits the input tokens into groups of the specified size and writes each group to a separate file.grouper
has been added to split an iterable into fixed-length chunks.