Not all available `ELRC` datasets are downloaded from OPUS

[ZenBel]

Hi all,

I have refreshed opus_index.tsv using the following commands as specified in opus_index.py:

$ curl "https://opus.nlpl.eu/opusapi/?preprocessing=moses" > opus_all.json 
$ cat opus_all.json |  jq -r  '.corpora[] | [.corpus, .version, .source, .target] | @tsv'  | sort  > opus_all.tsv 

and then mv opus_all.tsv opus_index.tsv. The new opus_index.tsv contains additional ELRC datasets, like ELRC-3083-wikipedia_health v1 ar en

However, when I run mtdata list -l eng-ara, this new dataset is not present. Any hint as to why this is the case?

Thanks in advance,

Z

P.S. The dataset is actually pulled by elrc_share.py but, still, adding it to the opus_index.tsv should allow also opus_index.py to pull it.

[thammegowda]

please pass --reindex flag https://github.com/thammegowda/mtdata/blob/72df36d1eb3a8295db273892c29b11cd11ef9b4b/mtdata/main.py#L129

Usually, the reindex happens automatically when we update version number, since we didnt change version number, we have to force reindex.

[ZenBel]

Oh, I missed that!

Thanks for pointing it out. mtdata -ri list -l eng-ara fixes the problem.