Dynamic model switch for ASR server GPU

[Zak-SA]

Hello, Is there a way to switch models without the need to restart the service for ASR server GPU? same as in https://github.com/alphacep/vosk-server/pull/206#issue-1425745792

Thanks!

[nshmyrev]

You'd better switch IP address of the server

[Zak-SA]

You'd better switch IP address of the server

The issue when using the GPU docker alphacep/kaldi-en-gpu, you can only use single model unlike when using the cpu version "kaldi-en" where you can pass "config": { "model": "model_path" restarting the gpu docker every time you need to use different model is not desired.

[nshmyrev]

You can simply start 2 servers, there is no need to restart. It is not very different from having 2 models under same process.

[Zak-SA]

You can simply start 2 servers, there is no need to restart. It is not very different from having 2 models under same process.

That's a good point but each server uses about 13GB of GPU memory and for both to work at the same time is about 26GB. it would be great to have a way to switch the model without the need to stop the docker and start another one or even having two dockers working at the same time and consume the resources

[Zak-SA]

Also, if you have ten models that means you need ten docker images while if you can switch then you need one image only

[nshmyrev]

I doubt it is easy, might require significant code change, but lets take a look.

[Zak-SA]

I doubt it is easy, might require significant code change, but lets take a look.

Thanks! it would be great to have this feature and capability. I think changes needs to be made on batch_model from Vosk api because it looks like it takes the model path as string. (line 28) Thanks again!

[nshmyrev]

Should be better now, needs testing though

[Zak-SA]

Should be better now, needs testing though

Great! I'll test it and let you know if there any issues. Thanks

[Zak-SA]

Should be better now, needs testing though Hello @nshmyrev Shouldn't we modify asr_server_gpu.py to accept model_path as config to pass it to Batchmodel similar to asr_server.py in lines (48 through 50) when passing model_path to Model?