Consider making own translator engine

sappho192 commented 3 years ago

장기적으로는 파판14용 자체 번역 엔진을 제작할 계획이 있습니다. 특정 고유명사나 은어들의 경우 어떤 엔진을 써도 이상하게 나오기 때문에 이런 데이터들을 따로 준비해서 자체 엔진을 만들면 여러가지 문제점이 해결될 수 있습니다. 다만 기반 데이터셋이나 추가 데이터 수집방법 등 구체적인 방향은 정해져 있지 않은 상태로, 오는 1월 이후에나 구체적인 계획 수립 및 개발이 진행될 것으로 보입니다.

I'm going to make my own translator engine only for FF14. Since some proper noun and slang used in FF14 will always be mistranslated with generic translators(DeepL, Papago, etc), so various issues can be resolved if specialized translator is developed. But this is only rough-excerpt step, so detailed designs and plans will be discussed after upcoming January.

sappho192 commented 3 years ago

Check this : https://aihub.or.kr/aihubdata/data/view.do?currMenu=115&topMenu=100&aihubDataSe=realm&dataSetSn=127 https://aihub.or.kr/aihubdata/data/view.do?currMenu=115&topMenu=100&aihubDataSe=realm&dataSetSn=546

sappho192 commented 1 year ago

I think I can relatively easily make my own translator specialized in FF14 with huggingface-transformers.
https://github.com/huggingface/transformers/tree/main/examples/pytorch/translation

Example of micro-tuning the translator: https://github.com/akpe12/JP-KR-ocr-translator-for-travel/blob/main/Training/training.ipynb

sappho192 commented 1 year ago

After the micro-tuning stuff, https://opennmt.net/CTranslate2/ can be the executer for my trained model to utilize within the IronworksTranslator.

sappho192 commented 1 year ago

The [sappho192/ffxiv-ja-ko-translator] project has been started based on this issue.

sappho192 / IronworksTranslator

Consider making own translator engine #45