Other languages

[jacob-hein]

How adaptable is ktrain for say SQuAD like QA with a BERT model pretrained in another language besides English?

[amaiya]

Here are three options to do non-English QA in ktrain:

• Try a model trained on non-English QA dataset (e.g., for French: https://huggingface.co/etalab-ia/camembert-base-squadFR-fquad-piaf). You can supply the model name as an argument to the SimpleQA constructor.
• You can also fine-tune a new model on a non-English QA dataset (e.g., if no BERT Squad model for your language is uploaded to the Hugging Face model hub yet). The Hugging Face GitHub repo shows examples for how to fine-tune a new QA model.
• Use ktrain.text.translation module to translate corpus from your language to English and submit the questions in English.

I've only tried the third option.

[jacob-hein]

I used your module to create an index of N news Danish articles and applied my fine-tuned Danish BERT QA model with these.

And ktrain works surprisingly well for just N=500.

However, I'm finding that the call text.SimpleQA.index_from_folder(folder_path=ARTICLES_DIR, index_dir=INDEXDIR) takes quite a some time to run (~8 minutes) for just 500 news articles, each of them saved in a .txt file at the folder_path.

Ideally, I would want to create an index of roughly N=100,000 articles, however I fear index_from_folder() is somewhat of a bottleneck for this objective. These articles take up about 200 mb when stored in a .json format.

Is this an realistic objective for the purpose of this module?

If yes, how can I optimize the index creation with this many articles?

[amaiya]

It should not take that long for just 500 news articles.

1. Did you follow the tutorial? How long does the call to index_from_folder take when you run the tutorial exactly as shown?
2. What is the exact command you're using for index_from_folder? Are you using the same parameters as in the tutorial?