Change Romanizer/Indicizer implementation

[anoopkunchukuttan]

The current romanizer/indicizer implementation is based on Alan Little's code. This worked for Devanagari alone and some retrofitting had been done to make it work for other languages. Now doing an implementation from scratch for ITRANS.

[anoopkunchukuttan]

Comparison between old and new romanizer

• (do nothing) Independent vowels are now correctly handled compared to the old implementation
  • e.g. (hi,old,new) --> गई gii gaii
• Nukta just copied in old implementation, now skipped
  • बड़ी baDa़ii baDaii
  • this happens because of the diacritic following the nukta, this case needs to be handled correctly
• (done) Non-hindi script characters are replaced by '-'
  • Bikaner Bikaner -------
  • Ideally we want to retain non-script characters
• script character with no mapping
  • currently replaced with same 'hi' character,because hindi was used to create the initial map
  • how best solution is to not keep it in the map at all, then missing one would just be copied over.
• (done) System can generate exceptions if starting with halant or dependent vowel. While this is invalid input, the system shoudl be robust to such input
• (done) Speed comparison: New code is about 5-6 times slower
  • Old: 1.26 s ± 89.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
  • New: 4.9 s ± 480 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
  • New (optimized): 622 ms ± 21.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
    • 4 times faster than old.
    • Optimization obtained by replacing lookup in vectors properties to directly working with offsets

[anoopkunchukuttan]

Comparing old and new indicizers

• nukta handling
• Running time: new is slightly slower
  • old: 2.87 s ± 115 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
  • new: 3.49 s ± 148 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
  • new with duplicate replacements: 4.17 s ± 329 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
  • new with duplicate replacements (and a check): 3.64 s ± 151 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

[anoopkunchukuttan]

Completed and merged