Curate Wiki Translate Corpus

[soumendrak]

• [ ] Do word to word and phrase to phrase separate files
• [ ] Divide multiple sentences into smaller chunks, may be sentence-wise ?
• [ ] Convert the Jupyter notebook code into pure Python code ?
• [ ] Lemmatization, Stemming, Inflection and derivation handling

[akpanda]

May i propose to have each of these computational linguistics algorithms ...

Lemmatization, Stemming, Inflection and derivation handling

...be split as sub project of their own 🤔 to be HTTPS "hostable" as a micro-service on their own , open to the developers with whatever tech they are comfortable in ; integrations when needed can be done will be at HTTPS layer .

For each of these algorithms, a contract can be generated like the following indicative one

• LemmatizationService URL : [base context] /sua/lemmas Method : POST input : {'sentence' : ''ଏକ ନାଗରିକର ସ୍ବପ୍ନ ", 'lan':'OR'} output : {'lemmas':["ଲ1","ଲ2","ଲ3"]}

• A team of 4 (Tech lead , SMEs for Odia , English languages , Computer science ) can form a small agile team to get these services individually

• Integration will be a separate project in itself , as an when the need arise

• When the model need to check lemmas in some phase it can call the HTTP service .

• if the network calls become an overhead we can bring the executing module behind the HTTP layer closer to the machine into a process of its own

• for further optimization when needed the same model can run in the same process as the caller

• more optimization beyond this will not be required , it will be only academic , if someone wants to do it, they can pursue , with the solutions in the field of computer science theory and applications as guides