PranjalChitale
commented
1 month ago The inference pipeline is designed to be broad-spectrum, handling texts from a wide array of domains. However, it is not foolproof.
This regex-placeholder method is applied post-hoc as we found it effective in most cases through empirical testing.
Note that the models weren't specifically trained to retain these placeholders and you can go ahead and fine-tune the models to do so.
Sentence splitting is performed using the best open-source libraries available.
The regex pattern was developed by analyzing encountered cases and covers most general-purpose use cases.
If you have any recommendations for other libraries or improved regex patterns, please let us know.
Additionally, you can choose to bypass the inference pipeline when sentence splitting is not necessary (if you are confident about the sequence length).
Below are the results when using the Fairseq model without the inference pipeline.
मौजूदा 1,87,500 रुपये की देनदारी से 20% की कमी।
कंपनी का राजस्व 2019-20 में 12,34,567 रुपये से बढ़कर 2020-21 में 56,78,910 रुपये हो गया।
जब सरकार में सबसे अधिक मूल वेतन केवल 30,000 रुपये प्रति माह था।
Sab8605
I have setup the models and I am using the En to Indic model for translation by following the Readme file. Observed the some issues with numbers.
Issues with Numerical Handling in English-to-Indic Translations:
Missing Numbers in Translation: The model fails to correctly output numerical values in translations. For instance, in the sentence "A reduction of 20% from the existing liability of Rs. 1,87,500," the Hindi translation is: "रुपये की मौजूदा देनदारी से 20 प्रतिशत की कमी।" The numerical value is missing in the output. Upon debugging, it was found that the model itself does not return the token, resulting in a translated output: ['▁रुपये ▁की ▁मौजूदा ▁देन दारी ▁से ▁20 ▁प्रतिशत ▁की ▁कमी ▁।'].
Incorrect Sentence Splitting with Numerical Values: The model is splitting sentences incorrectly around numerical values when there are spaces in "Rs." For example, in the sentence "The company reported a revenue increase from Rs. 12,34,567 in 2019-20 to Rs. 56,78,910 in 2020-21," preprocessing results in: ['The company reported a revenue increase from Rs.', '12,34,567 in 2019-20 to Rs.', '56,78,910 in 2020-21.']. This results in the Hindi output: 'कंपनी ने रुपये से राजस्व वृद्धि की सूचना दी। 12,34,567 से 2019-20 में रु। 2020-21 में 56,78,910।' This incorrect splitting affects the translation quality.
Regex Pattern Limitations: The regex pattern defined for handling numbers does not correctly process certain number formats. For example, in the sentence "The company reported a revenue increase from 12,34,567.74 in 2019-20 to Rs. 56,78,910.74 in 2020-21," preprocessing yields: ['The company reported a revenue increase from 12,34 , in to Rs.', '56,78 , in .']. The resulting translation is: 'कंपनी ने राजस्व में वृद्धि दर्ज की जो 12,34,567.74 से 2019-20 में रु। 56, 78, 2020-21 में 910.74.'
Extra Spaces in Numerical Values: The model is generating extra spaces in numerical values. For instance, in the sentence "when the highest basic pay in the government was only Rs. 30,000 per month," the translation is: "जब सरकार में सबसे अधिक मूल वेतन केवल रु। 30, 000 प्रति माह।" The inclusion of an extra space in the number "30, 000" affects the translation quality.
Thank you.