Inconsistent IDs - Githubissues

thammegowda commented 2 years ago

Hi,

Thanks for your efforts in creating/curating these datasets! These are priceless and greatly advance NLP for Indian languages.

I tried adding them into mtdata https://github.com/thammegowda/mtdata/issues/81 Since the README says your datasets are still growing, I am wondering whats the best long-term strategy is for keeping in sync.

For now, I can grep -i -o 'http[^ ]*zip' README.md, but the immediate concern is about consistency in determining name, version, and languages of datasets from URL.

The way current files are named (which act as ID for corpus) is a bit inconsistent. For example, consider these:

1) https://anuvaad-parallel-corpus.s3-us-west-2.amazonaws.com/oneindia_20210320_en_ml.zip
2) https://anuvaad-parallel-corpus.s3-us-west-2.amazonaws.com/pibarchives_2014_2016_en_ml.zip
3) https://anuvaad-parallel-corpus.s3-us-west-2.amazonaws.com/wikipedia-en-ml-20210201.zip

item (1), we can easily split by _ and get (name, version, lang1, lang2), so this is great. we can see oneindia is the name, 20210320 is the version, and en_ml are langs.
item (2), seems okay we can call 2014_2016 as version, though it would have been nice to have 2014to2016v1 as version. so splitting by _ would give exactly (name, version, lang1, lang2) as in item 1.
item (3) seems abnormal as it doesn't fit (name, version, lang1, lang2). There are more datasets matching item (1) than item (3) pattern, so I am inclined to call this abnormal.

Could you please consider having a consistent format in dataset IDs? It'd greatly help the automated downloaders such as mtdata.
^{Otherwise, do you really want your users to manually download 196 zip files via browser, and extract and merge them? :)}

Thanks.

P.S https://github.com/thammegowda/mtdata#dataset-id

thammegowda commented 2 years ago

Also, note that there is a bit of inconsistency inside zipping of item (3) as well.

$ unzip oneindia_20210320_en_ml.zip
Archive:  oneindia_20210320_en_ml.zip
   creating: en-ml/
  inflating: en-ml/oneindia_train.ml
  inflating: en-ml/oneindia_train.en
$ unzip pibarchives_2014_2016_en_ml.zip
Archive:  pibarchives_2014_2016_en_ml.zip
  inflating: en-ml/.DS_Store
  inflating: __MACOSX/en-ml/._.DS_Store
  inflating: en-ml/pib_arch_train.en
  inflating: en-ml/pib_arch_train.ml

$ unzip wikipedia-en-ml-20210201.zip
Archive:  wikipedia-en-ml-20210201.zip
  inflating: en-ml/ml.txt
  inflating: en-ml/en.txt

en-ml/wikipedia_train.{en,ml} could have made scripts/automation tools simple to write.

thammegowda commented 2 years ago

I added these datasets to v0.3.2

pip install -I mtdata==0.3.2

See mappings

``` mtdata list | grep Anuvaad | sed 's;https://anuvaad-parallel-corpus.s3-us-west-2.amazonaws.com;;' 2021-12-05 13:35:24 __init__.get_instance:48 INFO:: Loading index from cache /Users/tg/.mtdata/mtdata.index.0.3.2.pkl 2021-12-05 13:35:25 main.list_data:19 INFO:: Found 182787 Anuvaad-ik_2021-v1-eng-hin /ik-2021-v1-en-hi.zip en-hi/*.en,en-hi/*.hi Anuvaad-internal_judicial_2021-v1-eng-hin /internal-judicial-2021-v1-en-hi.zip en-hi/*.en,en-hi/*.hi Anuvaad-legal_terms_2021-v1-eng-hin /legal-terms-2021-v1-en-hi.zip en-hi/*.en,en-hi/*.hi Anuvaad-nouns_dict_2021-v1-eng-hin /nouns-dict-2021-v1-en-hi.zip en-hi/*.en,en-hi/*.hi Anuvaad-pib_2017-2020-eng-hin /pib_2017_2020_en_hi.zip en-hi/*.en,en-hi/*.hi Anuvaad-pibarchives_2009-2016-eng-hin /pibarchives_2009_2016_en_hi.zip en-hi/*.en,en-hi/*.hi Anuvaad-newsonair_2019-2020-eng-hin /newsonair_2019_2020_en_hi.zip en-hi/*.en,en-hi/*.hi Anuvaad-wikipedia-20210201-eng-hin /wikipedia-en-hi-20210201.zip en-hi/en.txt,en-hi/hi.txt Anuvaad-drivespark-20210303-eng-hin /drivespark-en-hi-20210303.zip en-hi/*.en,en-hi/*.hi Anuvaad-dd_national-20210320-eng-hin /dd-national-en-hi-20210320.zip en-hi/*.en,en-hi/*.hi Anuvaad-dd_sports-20210320-eng-hin /dd-sports-en-hi-20210320.zip en-hi/*.en,en-hi/*.hi Anuvaad-nativeplanet-20210315-eng-hin /nativeplanet-en-hi-20210315.zip en-hi/*.en,en-hi/*.hi Anuvaad-catchnews-20210320-eng-hin /catchnews-en-hi-20210320.zip en-hi/*.en,en-hi/*.hi Anuvaad-dwnews_2008-2020-eng-hin /dwnews_2008_2020_en_hi.zip en-hi/*.en,en-hi/*.hi Anuvaad-oneindia-20210320-eng-hin /oneindia_20210320_en_hi.zip en-hi/*.en,en-hi/*.hi Anuvaad-mk-20210320-eng-hin /mk_20210320_en_hi.zip en-hi/*.en,en-hi/*.hi Anuvaad-goodreturns-20210320-eng-hin /goodreturns_20210320_en_hi.zip en-hi/*.en,en-hi/*.hi Anuvaad-ie_sports-20210320-eng-hin /ie_sports_20210320_en_hi.zip en-hi/*.en,en-hi/*.hi Anuvaad-ie_tech-20210320-eng-hin /ie_tech_20210320_en_hi.zip en-hi/*.en,en-hi/*.hi Anuvaad-ie_news-20210320-eng-hin /ie_news_20210320_en_hi.zip en-hi/*.en,en-hi/*.hi Anuvaad-ie_lifestyle-20210320-eng-hin /ie_lifestyle_20210320_en_hi.zip en-hi/*.en,en-hi/*.hi Anuvaad-ie_general-20210320-eng-hin /ie_general_20210320_en_hi.zip en-hi/*.en,en-hi/*.hi Anuvaad-ie_entertainment-20210320-eng-hin /ie_entertainment_20210320_en_hi.zip en-hi/*.en,en-hi/*.hi Anuvaad-ie_education-20210320-eng-hin /ie_education_20210320_en_hi.zip en-hi/*.en,en-hi/*.hi Anuvaad-ie_business-20210320-eng-hin /ie_business_20210320_en_hi.zip en-hi/*.en,en-hi/*.hi Anuvaad-toi-20210320-eng-hin /toi_20210320_en_hi.zip en-hi/*.en,en-hi/*.hi Anuvaad-fin_express-20210320-eng-hin /fin_express_20210320_en_hi.zip en-hi/*.en,en-hi/*.hi Anuvaad-thewire-20210320-eng-hin /thewire_20210320_en_hi.zip en-hi/*.en,en-hi/*.hi Anuvaad-tribune-20210320-eng-hin /tribune_20210320_en_hi.zip en-hi/*.en,en-hi/*.hi Anuvaad-zeebiz-20210320-eng-hin /zeebiz_20210320_en_hi.zip en-hi/*.en,en-hi/*.hi Anuvaad-pa_govt-20210320-eng-hin /pa_govt_20210320_en_hi.zip en-hi/*.en,en-hi/*.hi Anuvaad-betterindia-20210320-eng-hin /betterindia_20210320_en_hi.zip en-hi/*.en,en-hi/*.hi Anuvaad-jagran_news-20210320-eng-hin /jagran-news_20210320_en_hi.zip en-hi/*.en,en-hi/*.hi Anuvaad-jagran_tech-20210320-eng-hin /jagran-tech_20210320_en_hi.zip en-hi/*.en,en-hi/*.hi Anuvaad-jagran_education-20210320-eng-hin /jagran-education_20210320_en_hi.zip en-hi/*.en,en-hi/*.hi Anuvaad-jagran_entertainment-20210320-eng-hin /jagran-entertainment_20210320_en_hi.zip en-hi/*.en,en-hi/*.hi Anuvaad-jagran_business-20210320-eng-hin /jagran-business_20210320_en_hi.zip en-hi/*.en,en-hi/*.hi Anuvaad-jagran_sports-20210320-eng-hin /jagran-sports_20210320_en_hi.zip en-hi/*.en,en-hi/*.hi Anuvaad-jagran_lifestyle-20210320-eng-hin /jagran-lifestyle_20210320_en_hi.zip en-hi/*.en,en-hi/*.hi Anuvaad-asianetnews-20210320-eng-hin /asianetnews_20210320_en_hi.zip en-hi/*.en,en-hi/*.hi Anuvaad-business_standard-20210320-eng-hin /business_standard_20210320_en_hi.zip en-hi/*.en,en-hi/*.hi Anuvaad-pranabmukherjee-20210320-eng-hin /pranabmukherjee_20210320_en_hi.zip en-hi/*.en,en-hi/*.hi Anuvaad-lokmat_entertainment-20210501-eng-hin /lokmat-entertainment_20210501_en_hi.zip en-hi/*.en,en-hi/*.hi Anuvaad-lokmat_news-20210501-eng-hin /lokmat-news_20210501_en_hi.zip en-hi/*.en,en-hi/*.hi Anuvaad-lokmat_lifestyle-20210501-eng-hin /lokmat-lifestyle_20210501_en_hi.zip en-hi/*.en,en-hi/*.hi Anuvaad-lokmat_sports-20210501-eng-hin /lokmat-sports_20210501_en_hi.zip en-hi/*.en,en-hi/*.hi Anuvaad-lokmat_tech-20210501-eng-hin /lokmat-tech_20210501_en_hi.zip en-hi/*.en,en-hi/*.hi Anuvaad-lokmat_financial-20210501-eng-hin /lokmat-financial_20210501_en_hi.zip en-hi/*.en,en-hi/*.hi Anuvaad-lokmat_healthcare-20210501-eng-hin /lokmat-healthcare_20210501_en_hi.zip en-hi/*.en,en-hi/*.hi Anuvaad-zee-30042021-eng-hin /zee_30042021_en_hi.zip en-hi/*.en,en-hi/*.hi Anuvaad-ik_2021-v1-eng-ben /ik-2021-v1-en-bn.zip en-bn/*.en,en-bn/*.bn Anuvaad-internal_judicial_2021-v1-eng-ben /internal-judicial-2021-v1-en-bn.zip en-bn/*.en,en-bn/*.bn Anuvaad-pib_2017-2020-eng-ben /pib_2017_2020_en_bn.zip en-bn/*.en,en-bn/*.bn Anuvaad-pibarchives_2015-2016-eng-ben /pibarchives_2015_2016_en_bn.zip en-bn/*.en,en-bn/*.bn Anuvaad-prothomalo_2014-2020-eng-ben /prothomalo_2014_2020_en_bn.zip en-bn/*.en,en-bn/*.bn Anuvaad-dwnews_2004-2020-eng-ben /dwnews_2004_2020_en_bn.zip en-bn/*.en,en-bn/*.bn Anuvaad-oneindia-20210320-eng-ben /oneindia_20210320_en_bn.zip en-bn/*.en,en-bn/*.bn Anuvaad-mk-20210320-eng-ben /mk_20210320_en_bn.zip en-bn/*.en,en-bn/*.bn Anuvaad-ie_sports-20210320-eng-ben /ie_sports_20210320_en_bn.zip en-bn/*.en,en-bn/*.bn Anuvaad-ie_tech-20210320-eng-ben /ie_tech_20210320_en_bn.zip en-bn/*.en,en-bn/*.bn Anuvaad-ie_news-20210320-eng-ben /ie_news_20210320_en_bn.zip en-bn/*.en,en-bn/*.bn Anuvaad-ie_general-20210320-eng-ben /ie_general_20210320_en_bn.zip en-bn/*.en,en-bn/*.bn Anuvaad-ie_entertainment-20210320-eng-ben /ie_entertainment_20210320_en_bn.zip en-bn/*.en,en-bn/*.bn Anuvaad-ie_education-20210320-eng-ben /ie_education_20210320_en_bn.zip en-bn/*.en,en-bn/*.bn Anuvaad-ie_business-20210320-eng-ben /ie_business_20210320_en_bn.zip en-bn/*.en,en-bn/*.bn Anuvaad-ie_lifestyle-20210320-eng-ben /ie_lifestyle_20210320_en_bn.zip en-bn/*.en,en-bn/*.bn Anuvaad-asianetnews-20210320-eng-ben /asianetnews_20210320_en_bn.zip en-bn/*.en,en-bn/*.bn Anuvaad-sentinel-20210320-eng-ben /sentinel_20210320_en_bn.zip en-bn/*.en,en-bn/*.bn Anuvaad-toi-20210320-eng-ben /toi_20210320_en_bn.zip en-bn/*.en,en-bn/*.bn Anuvaad-zee-30042021-eng-ben /zee_30042021_en_bn.zip en-bn/*.en,en-bn/*.bn Anuvaad-anuvaad_general-corpus-eng-ben /anuvaad_general_corpus_en_bn.zip en-bn/*.en,en-bn/*.bn Anuvaad-ik_2021-v1-eng-tam /ik-2021-v1-en-ta.zip en-ta/*.en,en-ta/*.ta Anuvaad-internal_judicial_2021-v1-eng-tam /internal-judicial-2021-v1-en-ta.zip en-ta/*.en,en-ta/*.ta Anuvaad-legal_terms_2021-v1-eng-tam /legal-terms-2021-v1-en-ta.zip en-ta/*.en,en-ta/*.ta Anuvaad-nouns_dict_2021-v1-eng-tam /nouns-dict-2021-v1-en-ta.zip en-ta/*.en,en-ta/*.ta Anuvaad-pib_2017-2020-eng-tam /pib_2017_2020_en_ta.zip en-ta/*.en,en-ta/*.ta Anuvaad-pibarchives_2014-2016-eng-tam /pibarchives_2014_2016_en_ta.zip en-ta/*.en,en-ta/*.ta Anuvaad-wikipedia-20210201-eng-tam /wikipedia-en-ta-20210201.zip en-ta/en.txt,en-ta/ta.txt Anuvaad-nativeplanet-20210315-eng-tam /nativeplanet-en-ta-20210315.zip en-ta/*.en,en-ta/*.ta Anuvaad-oneindia-20210320-eng-tam /oneindia_20210320_en_ta.zip en-ta/*.en,en-ta/*.ta Anuvaad-mk-20210320-eng-tam /mk_20210320_en_ta.zip en-ta/*.en,en-ta/*.ta Anuvaad-drivespark-20210303-eng-tam /drivespark-en-ta-20210303.zip en-ta/*.en,en-ta/*.ta Anuvaad-goodreturns-20210320-eng-tam /goodreturns_20210320_en_ta.zip en-ta/*.en,en-ta/*.ta Anuvaad-ie_sports-20210320-eng-tam /ie_sports_20210320_en_ta.zip en-ta/*.en,en-ta/*.ta Anuvaad-ie_tech-20210320-eng-tam /ie_tech_20210320_en_ta.zip en-ta/*.en,en-ta/*.ta Anuvaad-ie_news-20210320-eng-tam /ie_news_20210320_en_ta.zip en-ta/*.en,en-ta/*.ta Anuvaad-ie_lifestyle-20210320-eng-tam /ie_lifestyle_20210320_en_ta.zip en-ta/*.en,en-ta/*.ta Anuvaad-ie_entertainment-20210320-eng-tam /ie_entertainment_20210320_en_ta.zip en-ta/*.en,en-ta/*.ta Anuvaad-ie_education-20210320-eng-tam /ie_education_20210320_en_ta.zip en-ta/*.en,en-ta/*.ta Anuvaad-ie_business-20210320-eng-tam /ie_business_20210320_en_ta.zip en-ta/*.en,en-ta/*.ta Anuvaad-asianetnews-20210320-eng-tam /asianetnews_20210320_en_ta.zip en-ta/*.en,en-ta/*.ta Anuvaad-toi-20210320-eng-tam /toi_20210320_en_ta.zip en-ta/*.en,en-ta/*.ta Anuvaad-zee-30042021-eng-tam /zee_30042021_en_ta.zip en-ta/*.en,en-ta/*.ta Anuvaad-ik_2021-v1-eng-mal /ik-2021-v1-en-ml.zip en-ml/*.en,en-ml/*.ml Anuvaad-internal_judicial_2021-v1-eng-mal /internal-judicial-2021-v1-en-ml.zip en-ml/*.en,en-ml/*.ml Anuvaad-legal_terms_2021-v1-eng-mal /legal-terms-2021-v1-en-ml.zip en-ml/*.en,en-ml/*.ml Anuvaad-nouns_dict_2021-v1-eng-mal /nouns-dict-2021-v1-en-ml.zip en-ml/*.en,en-ml/*.ml Anuvaad-pib_2017-2020-eng-mal /pib_2017_2020_en_ml.zip en-ml/*.en,en-ml/*.ml Anuvaad-pibarchives_2014-2016-eng-mal /pibarchives_2014_2016_en_ml.zip en-ml/*.en,en-ml/*.ml Anuvaad-wikipedia-20210201-eng-mal /wikipedia-en-ml-20210201.zip en-ml/en.txt,en-ml/ml.txt Anuvaad-oneindia-20210320-eng-mal /oneindia_20210320_en_ml.zip en-ml/*.en,en-ml/*.ml Anuvaad-mk-20210320-eng-mal /mk_20210320_en_ml.zip en-ml/*.en,en-ml/*.ml Anuvaad-drivespark-20210303-eng-mal /drivespark-en-ml-20210303.zip en-ml/*.en,en-ml/*.ml Anuvaad-nativeplanet-20210315-eng-mal /nativeplanet-en-ml-20210315.zip en-ml/*.en,en-ml/*.ml Anuvaad-goodreturns-20210320-eng-mal /goodreturns_20210320_en_ml.zip en-ml/*.en,en-ml/*.ml Anuvaad-ie_sports-20210320-eng-mal /ie_sports_20210320_en_ml.zip en-ml/*.en,en-ml/*.ml Anuvaad-ie_tech-20210320-eng-mal /ie_tech_20210320_en_ml.zip en-ml/*.en,en-ml/*.ml Anuvaad-ie_news-20210320-eng-mal /ie_news_20210320_en_ml.zip en-ml/*.en,en-ml/*.ml Anuvaad-ie_general-20210320-eng-mal /ie_general_20210320_en_ml.zip en-ml/*.en,en-ml/*.ml Anuvaad-ie_entertainment-20210320-eng-mal /ie_entertainment_20210320_en_ml.zip en-ml/*.en,en-ml/*.ml Anuvaad-ie_business-20210320-eng-mal /ie_business_20210320_en_ml.zip en-ml/*.en,en-ml/*.ml Anuvaad-marketfeed-20210320-eng-mal /marketfeed_20210320_en_ml.zip en-ml/*.en,en-ml/*.ml Anuvaad-asianetnews-20210320-eng-mal /asianetnews_20210320_en_ml.zip en-ml/*.en,en-ml/*.ml Anuvaad-toi-20210320-eng-mal /toi_20210320_en_ml.zip en-ml/*.en,en-ml/*.ml Anuvaad-zee-30042021-eng-mal /zee_30042021_en_ml.zip en-ml/*.en,en-ml/*.ml Anuvaad-ik_2021-v1-eng-tel /ik-2021-v1-en-te.zip en-te/*.en,en-te/*.te Anuvaad-internal_judicial_2021-v1-eng-tel /internal-judicial-2021-v1-en-te.zip en-te/*.en,en-te/*.te Anuvaad-legal_terms_2021-v1-eng-tel /legal-terms-2021-v1-en-te.zip en-te/*.en,en-te/*.te Anuvaad-nouns_dict_2021-v1-eng-tel /nouns-dict-2021-v1-en-te.zip en-te/*.en,en-te/*.te Anuvaad-pib_2017-2020-eng-tel /pib_2017_2020_en_te.zip en-te/*.en,en-te/*.te Anuvaad-pibarchives_2015-2016-eng-tel /pibarchives_2015_2016_en_te.zip en-te/*.en,en-te/*.te Anuvaad-wikipedia-20210201-eng-tel /wikipedia-en-te-20210201.zip en-te/en.txt,en-te/te.txt Anuvaad-oneindia-20210320-eng-tel /oneindia_20210320_en_te.zip en-te/*.en,en-te/*.te Anuvaad-mk-20210320-eng-tel /mk_20210320_en_te.zip en-te/*.en,en-te/*.te Anuvaad-drivespark-20210303-eng-tel /drivespark-en-te-20210303.zip en-te/*.en,en-te/*.te Anuvaad-nativeplanet-20210315-eng-tel /nativeplanet-en-te-20210315.zip en-te/*.en,en-te/*.te Anuvaad-goodreturns-20210320-eng-tel /goodreturns_20210320_en_te.zip en-te/*.en,en-te/*.te Anuvaad-sakshi-20210320-eng-tel /sakshi_20210320_en_te.zip en-te/*.en,en-te/*.te Anuvaad-asianetnews-20210320-eng-tel /asianetnews_20210320_en_te.zip en-te/*.en,en-te/*.te Anuvaad-toi-20210320-eng-tel /toi_20210320_en_te.zip en-te/*.en,en-te/*.te Anuvaad-zee-30042021-eng-tel /zee_30042021_en_te.zip en-te/*.en,en-te/*.te Anuvaad-ik_2021-v1-eng-kan /ik-2021-v1-en-kn.zip en-kn/*.en,en-kn/*.kn Anuvaad-legal_terms_2021-v1-eng-kan /legal-terms-2021-v1-en-kn.zip en-kn/*.en,en-kn/*.kn Anuvaad-nouns_dict_2021-v1-eng-kan /nouns-dict-2021-v1-en-kn.zip en-kn/*.en,en-kn/*.kn Anuvaad-pib_2017-2020-eng-kan /pib_2017_2020_en_kn.zip en-kn/*.en,en-kn/*.kn Anuvaad-pibarchives_2015-2016-eng-kan /pibarchives_2015_2016_en_kn.zip en-kn/*.en,en-kn/*.kn Anuvaad-oneindia-20210320-eng-kan /oneindia_20210320_en_kn.zip en-kn/*.en,en-kn/*.kn Anuvaad-mk-20210320-eng-kan /mk_20210320_en_kn.zip en-kn/*.en,en-kn/*.kn Anuvaad-drivespark-20210303-eng-kan /drivespark-en-kn-20210303.zip en-kn/*.en,en-kn/*.kn Anuvaad-nativeplanet-20210315-eng-kan /nativeplanet-en-kn-20210315.zip en-kn/*.en,en-kn/*.kn Anuvaad-goodreturns-20210320-eng-kan /goodreturns_20210320_en_kn.zip en-kn/*.en,en-kn/*.kn Anuvaad-asianetnews-20210320-eng-kan /asianetnews_20210320_en_kn.zip en-kn/*.en,en-kn/*.kn Anuvaad-toi-20210320-eng-kan /toi_20210320_en_kn.zip en-kn/*.en,en-kn/*.kn Anuvaad-zee-30042021-eng-kan /zee_30042021_en_kn.zip en-kn/*.en,en-kn/*.kn Anuvaad-ik_2021-v1-eng-mar /ik-2021-v1-en-mr.zip en-mr/*.en,en-mr/*.mr Anuvaad-legal_terms_2021-v1-eng-mar /legal-terms-2021-v1-en-mr.zip en-mr/*.en,en-mr/*.mr Anuvaad-nouns_dict_2021-v1-eng-mar /nouns-dict-2021-v1-en-mr.zip en-mr/*.en,en-mr/*.mr Anuvaad-pib_2017-2020-eng-mar /pib_2017_2020_en_mr.zip en-mr/*.en,en-mr/*.mr Anuvaad-pibarchives_2015-2016-eng-mar /pibarchives_2015_2016_en_mr.zip en-mr/*.en,en-mr/*.mr Anuvaad-ie_sports-20210320-eng-mar /ie_sports_20210320_en_mr.zip en-mr/*.en,en-mr/*.mr Anuvaad-ie_news-20210320-eng-mar /ie_news_20210320_en_mr.zip en-mr/*.en,en-mr/*.mr Anuvaad-ie_lifestyle-20210320-eng-mar /ie_lifestyle_20210320_en_mr.zip en-mr/*.en,en-mr/*.mr Anuvaad-ie_entertainment-20210320-eng-mar /ie_entertainment_20210320_en_mr.zip en-mr/*.en,en-mr/*.mr Anuvaad-ie_business-20210320-eng-mar /ie_business_20210320_en_mr.zip en-mr/*.en,en-mr/*.mr Anuvaad-toi-20210320-eng-mar /toi_20210320_en_mr.zip en-mr/*.en,en-mr/*.mr Anuvaad-thewire-20210320-eng-mar /thewire_20210320_en_mr.zip en-mr/*.en,en-mr/*.mr Anuvaad-lokmat_entertainment-20210501-eng-mar /lokmat-entertainment_20210501_en_mr.zip en-mr/*.en,en-mr/*.mr Anuvaad-lokmat_news-20210501-eng-mar /lokmat-news_20210501_en_mr.zip en-mr/*.en,en-mr/*.mr Anuvaad-lokmat_lifestyle-20210501-eng-mar /lokmat-lifestyle_20210501_en_mr.zip en-mr/*.en,en-mr/*.mr Anuvaad-lokmat_sports-20210501-eng-mar /lokmat-sports_20210501_en_mr.zip en-mr/*.en,en-mr/*.mr Anuvaad-lokmat_tech-20210501-eng-mar /lokmat-tech_20210501_en_mr.zip en-mr/*.en,en-mr/*.mr Anuvaad-lokmat_financial-20210501-eng-mar /lokmat-financial_20210501_en_mr.zip en-mr/*.en,en-mr/*.mr Anuvaad-lokmat_healthcare-20210501-eng-mar /lokmat-healthcare_20210501_en_mr.zip en-mr/*.en,en-mr/*.mr Anuvaad-zee-30042021-eng-mar /zee_30042021_en_mr.zip en-mr/*.en,en-mr/*.mr Anuvaad-ik_2021-v1-eng-pan /ik-2021-v1-en-pa.zip en-pa/*.en,en-pa/*.pa Anuvaad-internal_judicial_2021-v1-eng-pan /internal-judicial-2021-v1-en-pa.zip en-pa/*.en,en-pa/*.pa Anuvaad-legal_terms_2021-v1-eng-pan /legal-terms-2021-v1-en-pa.zip en-pa/*.en,en-pa/*.pa Anuvaad-nouns_dict_2021-v1-eng-pan /nouns-dict-2021-v1-en-pa.zip en-pa/*.en,en-pa/*.pa Anuvaad-pib_2017-2020-eng-pan /pib_2017_2020_en_pa.zip en-pa/*.en,en-pa/*.pa Anuvaad-pibarchives_2015-2016-eng-pan /pibarchives_2015_2016_en_pa.zip en-pa/*.en,en-pa/*.pa Anuvaad-tribune-20210320-eng-pan /tribune_20210320_en_pa.zip en-pa/*.en,en-pa/*.pa Anuvaad-pa_govt-20210320-eng-pan /pa_govt_20210320_en_pa.zip en-pa/*.en,en-pa/*.pa Anuvaad-jagran_news-20210320-eng-pan /jagran-news_20210320_en_pa.zip en-pa/*.en,en-pa/*.pa Anuvaad-jagran_tech-20210320-eng-pan /jagran-tech_20210320_en_pa.zip en-pa/*.en,en-pa/*.pa Anuvaad-jagran_education-20210320-eng-pan /jagran-education_20210320_en_pa.zip en-pa/*.en,en-pa/*.pa Anuvaad-jagran_entertainment-20210320-eng-pan /jagran-entertainment_20210320_en_pa.zip en-pa/*.en,en-pa/*.pa Anuvaad-jagran_business-20210320-eng-pan /jagran-business_20210320_en_pa.zip en-pa/*.en,en-pa/*.pa Anuvaad-jagran_sports-20210320-eng-pan /jagran-sports_20210320_en_pa.zip en-pa/*.en,en-pa/*.pa Anuvaad-jagran_lifestyle-20210320-eng-pan /jagran-lifestyle_20210320_en_pa.zip en-pa/*.en,en-pa/*.pa Anuvaad-ik_2021-v1-eng-guj /ik-2021-v1-en-gu.zip en-gu/*.en,en-gu/*.gu Anuvaad-legal_terms_2021-v1-eng-guj /legal-terms-2021-v1-en-gu.zip en-gu/*.en,en-gu/*.gu Anuvaad-nouns_dict_2021-v1-eng-guj /nouns-dict-2021-v1-en-gu.zip en-gu/*.en,en-gu/*.gu Anuvaad-pib_2017-2020-eng-guj /pib_2017_2020_en_gu.zip en-gu/*.en,en-gu/*.gu Anuvaad-pibarchives_2015-2016-eng-guj /pibarchives_2015_2016_en_gu.zip en-gu/*.en,en-gu/*.gu Anuvaad-oneindia-20210320-eng-guj /oneindia_20210320_en_gu.zip en-gu/*.en,en-gu/*.gu Anuvaad-toi-20210320-eng-guj /toi_20210320_en_gu.zip en-gu/*.en,en-gu/*.gu Anuvaad-gu_govt-20210320-eng-guj /gu-govt_20210320_en_gu.zip en-gu/*.en,en-gu/*.gu Anuvaad-zee-30042021-eng-guj /zee_30042021_en_gu.zip en-gu/*.en,en-gu/*.gu Anuvaad-pib_2017-2020-eng-asm /pib_2017_2020_en_as.zip en-as/*.en,en-as/*.as Anuvaad-pibarchives_2015-2016-eng-asm /pibarchives_2015_2016_en_as.zip en-as/*.en,en-as/*.as Anuvaad-sentinel-20210320-eng-asm /sentinel_20210320_en_as.zip en-as/*.en,en-as/*.as Anuvaad-pib_2017-2020-eng-urd /pib_2017_2020_en_ur.zip en-ur/*.en,en-ur/*.ur Anuvaad-pibarchives_2015-2016-eng-urd /pibarchives_2015_2016_en_ur.zip en-ur/*.en,en-ur/*.ur Anuvaad-thewire-20210320-eng-urd /thewire_20210320_en_ur.zip en-ur/*.en,en-ur/*.ur Anuvaad-pib_2017-2020-eng-ori /pib_2017_2020_en_or.zip en-or/*.en,en-or/*.or Anuvaad-pibarchives_2015-2016-eng-ori /pibarchives_2015_2016_en_or.zip en-or/*.en,en-or/*.or ```

P.S. https://github.com/thammegowda/mtdata/blob/master/mtdata/index/anuvaad.py

project-anuvaad / anuvaad-parallel-corpus

Inconsistent IDs #1