plan

[rom1504]

• use https://github.com/UKPLab/EasyNMT
• partition by source/target language the data
• model one model per partition
• use clip retrieval inference model as template

in practice, steps:

• [x] build simple POC in example.py
• [ ] build real end to end poc taking as input parquet and outputing parquet
• [ ] benchmark 10k and add table with numbers
• [ ] build partitionner
• [ ] build the distributed version
  • [ ] build reader
  • [ ] build writer
  • [ ] build mapper
  • [ ] build distributor
  • [ ] build runner
  • [ ] build main
• [ ] benchmark for 1M
• [ ] check with m2m100 and opus mt
• [ ] benchmark for 10M
• [ ] do the big 100M run

[rom1504]

since the data is small and parquet, consider just doing the pure pyspark way

[rom1504]

https://towardsdatascience.com/high-performance-inferencing-with-large-transformer-models-on-spark-beb82e71ecc9

[rom1504]

https://docs.databricks.com/_static/notebooks/deep-learning/pytorch-images.html