Add token and char count to histogram stats

huggingface / datatrove

Freeing data processing from scripting madness by providing a set of platform-agnostic customizable pipeline processing blocks.

Apache License 2.0

1.97k stars 139 forks source link

Closed guipenedo closed 2 months ago

guipenedo commented 2 months ago

Adds 2 new stats per original stat when histogram stats are collected:

originalname__chars: number of chars
originalname__tokens: number of tokens (if metadata["token_count"] is defined)