Diacritics restoration in Text-to-text (Arabic)

[Kentoseth]

I installed the Whisper medium Arabic model (STT) to test out the punctuation restoration but it isn't clear how to use it.

The button in settings says:

This option only works with models that do not natively support punctuation

and if enabling it, it says that I need to install the Punctuation model.

But in the Whisper model info, it says:

Additional capabilities: punctuation

Is it native in the model or what needs to be done?

[mkiol]

Thanks for the question.

Indeed, "Punctuation restoration" function is definitely not well explained in the app - sorry.

"Punctuation restoration" is an extra text processing after STT. This processing uses additional ML model to guess the right latin punctuation marks (,.?!:) in the text. The model needs to be downloaded via model browser. Right now, it supports only 16 languages (no Arabic support). "Punctuation restoration" is only needed (and only enabled) for DeepSpeech, Vosk and few April models. Whisper and Faster Whisper natively support punctuation therefore this feature is not used for these models. If you are using model from Whisper family, "Punctuation restoration" option (enabled or disables) has not impact on processing.

[mkiol]

Regarding "diacritics restoration". It is implemented but not available for text-to-text conversion. When you mark "Restore diacritics before speech synthesis" (enabled by default) and start TTS, input text is processed to restore diacritical marks in the background and output is sent to TTS engine. Right now, you can't see text after diacritics restoration.

[Kentoseth]

Whisper and Faster Whisper natively support punctuation therefore this feature is not used for these models. If you are using model from Whisper family, "Punctuation restoration" option (enabled or disables) has not impact on processing.

My understanding of punctuation is possibly wrong here then.

Arabic can come in 2 formats:

فَرَائِضُ الْوُضُوءِ وَفَرْضُهُ: غَسْلُ الْوَجْهِ، وَغَسْلُ الْيَدَيْنِ مَعَ الْمِرْفَقَيْنِ ، وَمَسْحُ رُبُعِ الرَّأْسِ، وَغَسْلُ الرِّجْلَيْنِ مَعَ الْكَعْبَيْنِ. - diacritics

فرائض الوضوء وفرضه: غسل الوجه، وغسل اليدين مع المرفقين ، ومسح ربع الرأس، وغسل الرجلين مع الكعبين. - no diacritics

Diacritics could be considered punctuation (or vowelized). I assumed that the Whisper models would restore the diacritics as a by-product of punctuation, but the output was like the second example (no diacritics) and hence why I opened this ticket.

[Kentoseth]

Regarding "diacritics restoration". It is implemented but not available for text-to-text conversion. When you mark "Restore diacritics before speech synthesis" (enabled by default) and start TTS, input text is processed to restore diacritical marks in the background and output is sent to TTS engine. Right now, you can't see text after diacritics restoration.

This would be a nice feature to implement. Considering it is already restoring the diacritics for TTS, generating the output for the user could help them see if the model is not hallucinating when reading the text.

[mkiol]

Adding to TO-DO list for the next release.

[mkiol]

Sorry it took so long.

New version 4.5.0 brings following feature:

• Text to Text repair options. With these options you can directly fix diacritical marks and punctuation in the text.

The "repair options" are not enabled by default. You have to turn-on "Show Repair text options" in the settings.

When enabled, you can use this:

You need also download additional model:

30695bb7410f173f92751c71579dfd180f22d10f

[Kentoseth]

Thank you so much for implementing this feature and no need to apologize. I am very grateful that you even considered implementing this feature, which would be of limited usage to most other users.