Custom models

[NDSTHEBEST]

Hi, your app is verrrrrry good. Unfortunately, one feature is missing: custom models.

It would be amazing if the user would be able to import his own custom models (whisper and others). Maybe even set specific ones for specific languages.

Also, when I install the app on my family phones, it is far down in performance compare to mine. Maybe custom models would solve that. Buy honestly I feel like the app needs to have a bunch of performance mods, with custom models.

This comes from a tech guy, not a dev, so I would to hear what do you think.

[niedev]

Hi, thank you!

I have not implemented the possibility to use custom models because the inference of different models happens in different ways, many things are standardized but not all, for example:

• For tokenization (the process of translating input words into numbers) each model not only has a different dictionary but also translates some special symbols differently than the dictionary.
• Regarding the kv-cache (which is not mandatory but increases performance a lot for a model) you need to know some precise values ​​of the model (unique for each model).
• The decoder of each model must always have as first values ​​some particular characters (used to specify some options, for example, the input and output language for the translator, the input language (or languages) for the speech recognition, etc.), and they vary a lot from model to model (not only the initial characters but also their number).
• In walkie talkie mode the app recognizes the audio in both languages ​​at the same time, and each model has different configurations to do in order to do it.

I could have the user specify some of this information in the settings, but they are hard to find (I often had to read source code to find them), also it would be quite complex to implement, and it would not solve the Walkie Talkie mode issue.

The last reason is that I optimized the models themselves by separating some parts and eliminating others, and these optimizations would not be possible for custom models.

However, in the future, I will definitely implement the option to choose other models optimized by me (but not custom ones).

[NDSTHEBEST]

I now tried it again on my family member phone, just now with version 2.0.0. While last time on 2.1.1, it was super slow and did not work. Now it's working on that version with just a little hiccup. Weird

[niedev]

Yeah, the 2 versions use the same models. It's probably because the old GUI is a little lighter, and since that phone CPU is at the limit due to running the models, even that slight change makes a lot of difference. What is the phone model?