box-key / cyzil

A tool for fast and in-depth analysis of sequence generation model in Cython
Apache License 2.0
1 stars 0 forks source link

Computation on large corpus #9

Open box-key opened 4 years ago

box-key commented 4 years ago

Implement a function to compute each score for large corpus. It reads sentence pairs line by line, compute score for each pair, then aggregate the result.

box-key commented 4 years ago

If I can read file by chunk, I can use tqdm to show the progress.

Reference: