Tracking dataset metrics - Githubissues

sambanova / generative_data_prep

Apache License 2.0

58 stars 7 forks source link

Tracking dataset metrics #37

Closed snova-zoltanc closed 11 months ago

snova-zoltanc commented 1 year ago

This library should be tracking statistics about the dataset.

Some of the most important statistics would be

Number of tokens
Number of sequences
Amount of text that was dropped because of truncation input packing config
Average prompt length
Average completion length
Number of padding tokens
Sequence utilization (what percent of the tokens are not padding tokens)
Sequence completion utilization (what percent of the tokens are completion tokens)

snova-zoltanc commented 11 months ago

Done in this PR