Loading model files outside the main Javascript

[Thread7]

I opened issue #86 on building with my own models. But I am opening this because I am trying an alternate method and am also having issues with that. I followed the docs but my .js model files are not attempting to load. I am modifying the demo live.html to try this. I have files like mdef.js, feat.params.js, etc in the folder /webapp/hub4js/ I try to load them in live.html like this.

console.log('Line 67 Just about to load');   
 var id;
recognizer.postMessage({command: 'load',
                    callbackId: id,
                    data: ["hub4js/feat.params.js",
                    "hub4js/mdef.js",
                    "hub4js/means.js",
                    "hub4js/noisedict.js",
                    "hub4js/sendump.js",
                    "hub4js/transition_matrices.js",
                    "hub4js/variances.js"]
                   });        
  };

When I load http://localhost/webapp/live.html I look in the console and see the "Line 67" message. But none of those .js are even trying to be retrieved by my local web server. I have a file system monitor running and no process even looks for those files. And I am also confident that the default model is still being used because the next line of the console says this: INFO: pocketsphinx.c(152): Parsed model-specific feature parameters from rm1_200/feat.params

I don't want to complicate matters by adding this following note but it may help. When I try to include the actual load command like this postRecognizerJob({command: 'initialize', data: [["-hmm", "hub4js"], ["-fwdflat", "no"]]},

the console gives the error: ERROR: "acmod.c", line 79: Folder 'hub4js' does not contain acoustic model definition 'mdef' That may suggest a path problem. But I really don't think it is a path problem. I have tried putting it in the parent directory, the parent of the parent, and the /js directory. Nothing helps. Plus remember I am monitoring the filesystem and no process is trying to access anything like hub4js Not sure how to proceed.

[nshmyrev]

You need to:

1) Share your files 2) Describe which exact doc you are following. Neither

https://github.com/syl22-00/pocketsphinx.js#i-embedding-the-files-into-one-large-javascript-file

Nor

https://github.com/syl22-00/pocketsphinx.js#ii-package-model-files-outside-the-main-javascript

Describe what you are doing currently. You need to follow documentation instead.

[Thread7]

Well I am trying item 2 (package model files outside the main javascript). I just used a shortcut which I think is okay. I took the acoustic model js files from this repository: https://github.com/syl22-00/pocketsphinx.js-en_US-hub4wsj_sc_8k/ It seemed an easier way to get a test going than build them myself.

I have put all my files up on Github here: https://github.com/Thread7/pocketsphinx.js.external

and you can test my solution directly here: https://rawgit.com/Thread7/pocketsphinx.js.external/master/webapp/live.html

Any help would be great.

[syl22-00]

I'd recommend lazy loading of files:

https://github.com/syl22-00/pocketsphinx.js#h2-lazy-loading-raw-files

So you do not need to package your files into JavaScript, and you do not need to rebuild pocketsphinx.js.

[Thread7]

Ok, that sounds easier. I tried it. This is what I did:

1. Put the acoustic model files in /webapp/hub4-8gau/

2. Change this line on line 45 of live.html recognizer.postMessage(''); to this:

   recognizer.postMessage({command: 'lazyLoad',
                   callbackId: id,
                   data: {folders: [["/", "hub4-8gau"]],
              files: [["/hub4-8gau", "means", "../hub4-8gau/means"],
                      ["/hub4-8gau", "variances", "../hub4-8gau/variances"],
                  ["/hub4-8gau", "transition_matrices", "../hub4-8gau/transition_matrices"],
                  ["/hub4-8gau", "sendump", "../hub4-8gau/sendump"],
                  ["/hub4-8gau", "mdef", "../hub4-8gau/mdef"],
                  ["/hub4-8gau", "feat.params", "../hub4-8gau/feat.params"],
                  ["/hub4-8gau", "mixture_weights", "../hub4-8gau/mixture_weights"],
                  ["/hub4-8gau", "noisedict", "../hub4-8gau/noisedict"]]
             }
          });

3. Change line 58 to this:

   postRecognizerJob({command: 'initialize', data: [["-hmm", "hub4-8gau"], ["-fwdflat", "no"]]},
                       function() {
                                   if (recorder) recorder.consumers = [recognizer];
                                   feedWords(wordList);});

But I get a very simple error. Uncaught ReferenceError: id is not defined Not sure how to use "id" but thought this might work. Just above my recognizer.postMessage statement I included this id = function(event) { console.log('Callback from recognizer.postMessage worked'); };

But with that statement included I get this error: Uncaught DOMException: Failed to execute 'postMessage' on 'Worker': An object could not be cloned.

I am stuck on how to handle this error. If you want to see my exact code I created lazy.html in my repo. It can be tested here:

https://rawgit.com/Thread7/pocketsphinx.js.external/master/webapp/lazy.html

Thanks!

[syl22-00]

id should be an id, not a function. Look at other postMessage calls, that's the we we do using callbackmanager.

[Thread7]

Ok, I cannot find a single example in the docs, or in these issues where someone is using the id in a postMessage call. Still I put the line: var id; one line above the the postMessage. It seems to get farther, but now I get the error in my console: ERROR: "acmod.c", line 79: Folder 'hub4-8gau' does not contain acoustic model definition 'mdef'

You can see for yourself by checking the link I updated: https://rawgit.com/Thread7/pocketsphinx.js.external/master/webapp/lazy.html

Thanks.

[syl22-00]

What I would suggest is that you spend some time understanding live.html, especially for the way we send commands to the recognizer via worker messages. Documentation of the README.md file and inside live.html should suffice.

You are posting your lazyLoad command to the worker right after it is instanciated, before it is ready. Also, if you base yourself on live.html, you might want to use the postRecognizerJob function:

// A convenience function to post a message to the recognizer and associate
// a callback to its response
function postRecognizerJob(message, callback) {

[Thread7]

Ok thanks. I'll work down that path and post my results. BTW, since it seems like lazyLoad is the consensus best way to load the audio model. Maybe even for the live.html demo it should replace packaging the rm1_200 model inside pocketsphinx.js. If there was just a complete working lazyLoad example in here it would probably save many of us a lot of time.

Thanks!

[Thread7]

Ok, so I got lazyLoad to work. It seems to be reading the files from the new acoustic model directory.
You can test it here: https://rawgit.com/Thread7/pocketsphinx.js.external/master/webapp/lazy.html and view the source here: https://github.com/Thread7/pocketsphinx.js.external

The problem now is that accuracy is still very bad. I'm sure the initial advice will be that I need to get a good model for my language/vocab/recording environment. That is my hunch, but I'm not so sure.

First, I probably tried 10 models. Most of them I found by browsing here: https://sourceforge.net/projects/cmusphinx/files/Acoustic%20and%20Language%20Models/

Of all the ones I tried only 1 did I get to lazyLoad completely. That was this: cmusphinx-en-us-ptm-5.2

Even its counterpart, cmusphinx-en-us-5.2 did not load giving the error: ERROR: "ms_mgau.c", line 116: Dimension of stream 0 does not match: 36 != 39

Other models gave different errors. But at least I could test cmusphinx-en-us-ptm-5.2. Unfortunately, when I tried speaking the various cities I got only a 38% accuracy rate. Even rm1_200 at least gave me a 62% rate. (If you don't include all the false positives when it recognizes background noise)

All the files in cmusphinx-en-us-ptm-5.2 amount to about 17.6MB. rm1_200 is 1MB. So it seems like I should get better accuracy with the former.

My thoughts are as follows:

1. Do I simply need to find a better model that actually works with lazyLoad? If so, where do I find it? I just want to be able to speak 30 or so English words through a PC computer microphone. Eventually I could study up on creating my own model. But right now I know a sufficient model must exist to get a simple proof of concept working.
2. Maybe the settings for cmusphinx-en-us-ptm-5.2 can be tweaked to give much better results. I've seen settings in the console like -agcthresh 2.0, -alpha 0.97, -fwdflatbeam 1e-64. I have no idea what these mean or if there is a way to set them in pocketsphinx.js.
3. I know it has been said many times that pocketsphinx.js should give the exact same performance as pocketsphinx compiled on other platforms. Is there a compiled program out there that could be used to test this? I am not a C/C++ programmer (which is what drew me to the .js version in the first place)

Thanks again.

[syl22-00]

I just tried your https://rawgit.com/Thread7/pocketsphinx.js.external/master/webapp/lazy.html on my desktop pc and it seemed to work well.

I you see very bad performance, I could recommend to keep the actual recorded audio data sent to the recognizer and see if there is an issue here. For instance it could be a buffer becoming full while recording.

[dhdaines]

It could be that you have run into a problem with Emscripten, where mmap() does not work for reading files - while it looks like it should work in the code, it seems that it just allocates zero-filled memory. Try adding ["-mmap", "no"] to the configuration for the recognizer. I'm still trying to track down the problem in Emscripten... will file a bug there at some point hopefully.