feat: add dynamic model switching

[charles-zablit]

This PR adds a feature to dynamically switch between models, without having to restart the service.

A client can send:

{
  "config": {
    "model": "model_path"
 }
}

and it will change to the correct model.

[nshmyrev]

Great, thank you!

[Zak-SA]

@charles-zablit is dynamic model switching for CPU only? if so can you add for the ASR GPU server too? Thanks!

[charles-zablit]

@charles-zablit is dynamic model switching for CPU only? if so can you add for the ASR GPU server too? Thanks!

It does not, but it's possible. What you need to do, is add a base_path parameter to this function: https://github.com/alphacep/vosk-api/blob/1053cfa0f80039d2956de7e05a05c0b8db90c3c0/src/batch_model.cc#L23

Then, replace every model/ with this new parameter, so for instance:

po.ReadConfigFile("model/conf/model.conf"); becomes po.ReadConfigFile(base_path + "conf/model.conf");

Then, in server_asr.py, copy my changes, and do: model = BatchModel(jobj['model']) where jobj['model'] is the base_path.

[nshmyrev]

not that simple, there is ivector path in ivector.conf which is a file, you have to modify it on the fly or change gpu decoder code to read ivector opts instead of a file

[charles-zablit]

Can't you have a different ivector.conf per model? Specifying the base path would point to a different model, and different ivector.conf` each time?

[Zak-SA]

@charles-zablit is dynamic model switching for CPU only? if so can you add for the ASR GPU server too? Thanks!

It does not, but it's possible. What you need to do, is add a base_path parameter to this function: https://github.com/alphacep/vosk-api/blob/1053cfa0f80039d2956de7e05a05c0b8db90c3c0/src/batch_model.cc#L23

Then, replace every model/ with this new parameter, so for instance:

po.ReadConfigFile("model/conf/model.conf"); becomes po.ReadConfigFile(base_path + "conf/model.conf");

Then, in server_asr.py, copy my changes, and do: model = BatchModel(jobj['model']) where jobj['model'] is the base_path.

That's interesting and I thought this could be a good start to make it work for gpu, even though @nshmyrev mentioned that it's more complicated but I think we need to be able to pass the model path as a config. we can also set the default model path to the current embedded path "model" if no path given. @nshmyrev as you mentioned that ivector path is in ivector.conf which will have the path for each model. each model has its own ivector.conf that refer to the correct files path for that model. I am sure you know better, and I might be wrong but it worth looking into it. Thanks for the help

[Zak-SA]

I think in addition to changing the function in: https://github.com/alphacep/vosk-api/blob/1053cfa0f80039d2956de7e05a05c0b8db90c3c0/src/batch_model.cc#L23

po.ReadConfigFile("model/conf/model.conf"); to po.ReadConfigFile(base_path + "conf/model.conf");

we need to change the lines 32-37 :

string nnet3rxfilename = "model/am/final.mdl"; string hclg_fstrxfilename = "model/graph/HCLG.fst"; string word_symsrxfilename = "model/graph/words.txt"; string winforxfilename = "model/graph/phones/word_boundary.int"; string std_fstrxfilename = "model/rescore/G.fst"; string carparxfilename = "model/rescore/G.carpa";

Also, lines 76 and 77 :

batched_decoder_config.feature_opts.mfcc_config = "model/conf/mfcc.conf"; batched_decoder_config.feature_opts.ivector_extraction_config = "model/conf/ivector.conf";

by replacing model/ with model path.

I think this way we could get dynamic model switching capability by pointing to the correct model files unless other code changes needed elsewhere

Thanks!

[nshmyrev]

Hold on with this, I'll implement tomorrow. It needs Kaldi change.

[Zak-SA]

Hold on with this, I'll implement tomorrow. It needs Kaldi change.

Any updates? Thanks!

[nshmyrev]

A bit more complicated, sorry, work in progress

[Zak-SA]

A bit more complicated, sorry, work in progress

It's good to know that work is in progress, thanks for spending the time and effort to help with this issue.