MycroftAI / mycroft-precise

A lightweight, simple-to-use, RNN wake word listener
Apache License 2.0
848 stars 226 forks source link

Empty metrics from precise-test #185

Open ljj7975 opened 4 years ago

ljj7975 commented 4 years ago

I have correctly set up the dataset but I am seeing 0s for the counts

what might be the cause of this?

Command : precise-test -t 0.85 trained_model.net dataset

Data: <TrainData wake_words=502 not_wake_words=24641 test_wake_words=54 test_not_wake_words=2504>
=== False Positives ===

=== False Negatives ===

=== Counts ===
False Positives: 0
True Negatives: 0
False Negatives: 0
True Positives: 0

=== Summary ===
0 out of 0
0.00%

0.00% false positives
0.00% false negatives
MatthewScholefield commented 4 years ago

Hmm, strange. I don't think this is the case, but can you check if the test files are the correct format? They should be 16 bit integer mono wav files.

ljj7975 commented 4 years ago

yea they are .wav with monochannel 16000 sample rates I generated them with librosa write file and for some datasets I generated, it works fine. It just randomly happens that I couldn't figure out why so I was asking here.

As you see in the log, it correctly locate the files Data: <TrainData wake_words=502 not_wake_words=24641 test_wake_words=54 test_not_wake_words=2504>

ljj7975 commented 4 years ago

Things still fail with the correct format

import soundfile as sf

data = (item.audio_data.numpy() * 32767).astype('int16')
sf.write(output_path, data, 16000, subtype='PCM_16', endian='LITTLE')
ljj7975 commented 4 years ago

I have debugged the issue. pyache was getting into the way.

I think others fall into the same trap. is it possible for u to add support for disabling the cache?

in my cases files might have the same name but contents can be different

MatthewScholefield commented 4 years ago

Ahh, makes sense. It shouldn't be too hard to disable the cache. Out of curiosity, are you working off the dev branch or off of the fork with tf2 support?