search

nttcslab-sp / kaldiio

A pure python module for reading and writing kaldi ark files
Other
249 stars 35 forks source link

Fix memory issue when using Python2 #14

Closed kamo-naoyuki closed 5 years ago

kamo-naoyuki commented 5 years ago

What: Memory issue occurs when using kaldiio.ReadHelper with large scp file.

Condition:

python==2

Repro:

import kaldiio

with kaldiio.ReadHelper('scp:feats.scp') as f:
    for k, a in f:
        pass

Why: I invoked collections.MutableMapping.items() method but it is not a generator function in Python2, thus all matrices are loaded.

codecov-io commented 5 years ago

Codecov Report

Merging #14 into master will not change coverage. The diff coverage is 100%.

Impacted file tree graph

@@           Coverage Diff           @@
##           master      #14   +/-   ##
=======================================
  Coverage   81.31%   81.31%           
=======================================
  Files           6        6           
  Lines        1001     1001           
=======================================
  Hits          814      814           
  Misses        187      187
Impacted Files Coverage Δ
kaldiio/highlevel.py 70.37% <100%> (ø) :arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 9eaebcd...4d54237. Read the comment docs.