search

yajiemiao / pdnn

PDNN: A Python Toolkit for Deep Learning. http://www.cs.cmu.edu/~ymiao/pdnntk.html
Apache License 2.0
224 stars 105 forks source link

MemoryError in extracting binary ark file #52

Open asadullah797 opened 5 years ago

asadullah797 commented 5 years ago

I have scp file with corresponding ark file, whenever I want to use class kaldi_feat.py, using read_next_utt() function I get the following error: MemoryError I backtraced the error and obtained the following information of error:

tmp_mat = numpy.frombuffer(ark_read_buffer.read(rows cols 4), dtype=numpy.float32) Traceback (most recent call last): File "", line 1, in MemoryError Please note values of rows and cols are (1254238158, 5784298), their * becomes very big number.

MaigoAkisame commented 5 years ago

It looks like the format of the scp or ark file is corrupted: the program has read very large numbers of rows and columns.

You may need to provide the scp and ark file for me to debug it...

asadullah797 commented 5 years ago

Please find the attached scp with ark file for your information. I have tried two scp files and two ark files but got big values of rows and columns.

Untitled Folder.tar.gz Untitled Folder.tar.gz

Thanks

MaigoAkisame commented 5 years ago

I see that your ark files are in the compressed format, which is not supported by PDNN. I would suggest that you dump the scp and ark files in uncompressed binary format, and try again.

asadullah797 commented 5 years ago

No no, actually I could not attach files here in github thread as usual then I compressed the files but for experiment I am using the original uncompressed files.

MaigoAkisame commented 5 years ago

I did decompress your zip archives. What I mean is, in your .ark files, the matrices are stored in a compressed format, which is not recognized by PDNN.

Here's the content of one of your .ark files: image

The highlighted four bytes are the header of a matrix. "\0BCM" means "binary compressed matrix". The format accepted by PDNN is "\0BFM", which stands for "binary float matrix".

asadullah797 commented 5 years ago

Alrigtht, you are right, But could you please suggest me how can I export Kaldi features to python (maybe tensorflow). As far as I know PDNN convert kaldi features to python and these files are exactly kaldi MFCC features. If there is any changes required, could you please suggest me. Many thanks

MaigoAkisame commented 5 years ago

You can use either the kaldi_feat.py script in PDNN, or the readArk / readScp functions in the following script: https://github.com/MaigoAkisame/fileutils/blob/master/kaldi.py to load Kaldi features into Python.

Afterwards, you can arrange them into any shapes required by Tensorflow.

Unfortunately neither script supports the compressed matrix format, so you'll have to dump the features into the BFM format first.

asadullah797 commented 5 years ago

Thanks for your information. usually: copy-feats ark:abc.ark ark,t:a.txt is used to convert ark binary file to correspoding text format but I don't know how to convert text ark to binary float matrix in order to use it in kaldi_feat.py (PDNN).

MaigoAkisame commented 5 years ago

copy-feats scp:1.scp ark,scp:2.ark,2.scp

This should do the job. If you don't specify the text format, by default the output will be in uncompressed binary format.