Guide users to new vocabulary

[StephanAkkerman]

Idea: guide the users in learning new words for their selected language. They either have the option to input words themselves or be guided by this list (or another if there's something better)

Tasks

• Find lists that have many frequent words list (preferably corresponding with our supported languages)
  • https://github.com/frekwencja/most-common-words-multilingual
  • https://github.com/hermitdave/FrequencyWords (any issues here are also found in the original source)
  • https://opus.nlpl.eu/OpenSubtitles/corpus/version/OpenSubtitles (this is where FrequencyWords is based on, 62 langs)
  • Could download it from https://object.pouta.csc.fi/OPUS-OpenSubtitles/v2018/freq/en.freq.gz
  • https://huggingface.co/datasets/Helsinki-NLP/open_subtitles
  • https://en.wiktionary.org/wiki/Wiktionary:Frequency_lists/

Plan:

• Scrape https://github.com/hermitdave/FrequencyWords/tree/master/content/2018
• Make it into huggingface dataset
• Write down all supported languages
• Check https://en.wiktionary.org/wiki/Wiktionary:Frequency_lists for missing languages

Workflow:

• Users selects language they want to learn
• If they want to use this frequency word list, then download it from the source (or compile a dataset ourselves)
• Follow the order of the words in that repo (most frequent to least frequent)
• Translate foreign word to English
• Generate mnemonic

Option (instead of translating):

• Find word in scraped Wiktionary data (https://kaikki.org/dictionary/rawdata.html / https://github.com/tatuylonen/wiktextract / https://github.com/abuccts/wikt2pron/tree/master)
• Match the IPA and word meaning
• Use this for the #18

https://github.com/hermitdave/FrequencyWords/blob/master/content/2018/id/id_50k.txt

Supports many languages and probably better for real life scenarios

[StephanAkkerman]

Maybe translate the meaning using DeepL: https://github.com/DeepLcom/deepl-python

[StephanAkkerman]

en.wiktionary should include enough info about other languages