project-anuvaad / anuvaad-parallel-corpus

Creative Commons Attribution 4.0 International
13 stars 2 forks source link

Anuvaad Parallel Corpus

This repository contains parallel language corpus links for popular Indian languages developed as part of the Anuvaad project.

Please reach out to nlp-nmt@tarento.com for any clarification/interpretation/usage of the linked datasets.

Shield: CC BY 4.0

This work is licensed under a Creative Commons Attribution 4.0 International License.

CC BY 4.0

Status

The current status of the parallel corpus built :

(Going forward, Anuvaad data contribution will be done via ULCA platform. Please refer https://bhashini.gov.in/ulca for further details)

Language Pair Parallel Corpus Count
English-Hindi 5,627,366
English-Bengali 1,633,224
English-Tamil 1,522,854
English-Malayalam 1,505,760
English-Telugu 1,660,974
English-Kannada 1,352,017
English-Marathi 2,537,853
English-Punjabi 2,154,298
English-Gujarati 1,656,616
English-Assamese 47,174
English-Urdu 162,048
English-Odia 92,309

Goal

The goal is to build high quality parallel corpus for the Indian languages across various domains (General, Legal, Education, Healthcare, Automobile, News etc). This can be eventually used to train the ML models based on the use cases.

Read more about Anuvaad @ http://anuvaad.org/

The code for building the below mentioned datasets are available under https://github.com/project-anuvaad/anuvaad-corpus-tools

Links

English-Hindi

Domain Source Year En-Hi pairs count Corpus Download Link
Legal Multiple Sources 2010-2020 947,148 Indian Judiciary
Legal HC/SUVAS NA 81,884 HC/SUVAS
Legal Legal Terminologies NA 3,980 Legal Terms
Legal Nouns Dictionary NA 72,192 Nouns Dict
News PIB 2017-2020 401,859 PIB (2017-2020)
News PIB Archives 2009-2016 288,573 PIB Archives (2009-2016)
News NewsOnAir 2019-2021(Jan) 110,748 NewsOnAir (2019-2021)
General Wikipedia Until 2021-02 193,340 Wikipedia (20210201)
Automobile DriveSpark Until 2021-03 39,545 DriveSpark (20210303)
General DD News National Until 2021-03 33,263 DD News-National (20210320)
Sports DD News Sports Until 2021-03 3,666 DD News-Sports (20210320)
Tourism Native Planet Until 2021-03 31,933 Native Planet (20210320)
News CatchNews Until 2021-03 35,525 CatchNews (20210320)
News DW News 2008-2020 68,875 DW News (2008-2020)
News OneIndia Until 2021-03 143,139 OneIndia (20210320)
Sports MyKhel Until 2021-03 15,711 MyKhel (20210320)
Financial GoodReturns Until 2021-03 23,587 GoodReturns (20210320)
Sports Indian Express Until 2021-03 74,265 IE-Sports (20210320)
Technology Indian Express Until 2021-03 57,625 IE-Technology (20210320)
News Indian Express Until 2021-03 170,917 IE-News (20210320)
Lifestyle Indian Express Until 2021-03 1,812 IE-Lifestyle (20210320)
General Indian Express Until 2021-03 547 IE-General (20210320)
Entertainment Indian Express Until 2021-03 43,202 IE-Entertainment (20210320)
Education Indian Express Until 2021-03 18,898 IE-Education (20210320)
Financial Indian Express Until 2021-03 16,212 IE-Financial (20210320)
News Times of India Until 2021-03 653,356 TOI-News (20210320)
Financial Financial Express Until 2021-03 13,345 Financial Express (20210320)
News The Wire Until 2021-03 98,137 The Wire (20210320)
News Tribune Until 2021-03 391,004 Tribune (20210320)
Financial ZeeBiz Until 2021-03 4,338 ZeeBiz (20210320)
Government Press Release Punjab Govt As of 2021-03 12,106 Punjab Govt (20210320)
General The Better India Until 2021-03 6,217 The Better India (20210320)
News Jagran Until 2021-03 75,002 Jagran-News (20210320)
Technology Jagran Until 2021-03 63,555 Jagran-Technology (20210320)
Education Jagran Until 2021-03 26,770 Jagran-Education (20210320)
Entertainment Jagran Until 2021-03 50,328 Jagran-Entertainment (20210320)
Financial Jagran Until 2021-03 26,497 Jagran-Financial (20210320)
Sports Jagran Until 2021-03 34,190 Jagran-Sports (20210320)
Lifestyle Jagran Until 2021-03 6,253 Jagran-Lifestyle (20210320)
News AsianetNews Until 2021-03 15,087 AsianetNews (20210320)
Financial Business Standard Until 2021-03 268,603 Business Standard (20210320)
Government Press Release Pranabmukherjee As of 2021-03 43,259 Pranabmukherjee (20210320)
Entertainment Lokmat Until 2021-05 16,873 Lokmat-Entertainment (20210501)
News Lokmat Until 2021-05 68,793 Lokmat-News (20210501)
Lifestyle Lokmat Until 2021-05 1,223 Lokmat-Lifestyle (20210501)
Sports Lokmat Until 2021-05 36,279 Lokmat-Sports (20210501)
Technology Lokmat Until 2021-05 1,946 Lokmat-Technology (20210501)
Financial Lokmat Until 2021-05 6,751 Lokmat-Financial (20210501)
Healthcare Lokmat Until 2021-05 2,202 Lokmat-Healthcare (20210501)
News Zee News Until 2021-04 870,065 Zee-News (2011-2021)

English-Bengali

Domain Source Year En-Bn pairs count Corpus Download Link
Legal Multiple Sources 2010-2020 625,456 Indian Judiciary
Legal HC/SUVAS NA 18,353 HC/SUVAS
News PIB 2017-2020 74,433 PIB (2017-2020)
News PIB Archives 2015-2016 27,383 PIB Archives (2015-2016)
News Prothomalo 2014-2020 283,947 Prothomalo (2014-2020)
News DW News 2004-2020 23,054 DW News (2004-2020)
News OneIndia Until 2021-03 34,453 OneIndia (20210320)
Sports MyKhel Until 2021-03 23,971 MyKhel (20210320)
Sports kolkata24x7 Until 2021-03 29,318 kolkata24x7-Sports (20210320)
Technology kolkata24x7 Until 2021-03 17,083 kolkata24x7-Technology (20210320)
News kolkata24x7 Until 2021-03 49,633 kolkata24x7-News (20210320)
General kolkata24x7 Until 2021-03 3,215 kolkata24x7-General (20210320)
Entertainment kolkata24x7 Until 2021-03 19,134 kolkata24x7-Entertainment (20210320)
Education kolkata24x7 Until 2021-03 2,035 kolkata24x7-Education (20210320)
Financial kolkata24x7 Until 2021-03 2,640 kolkata24x7-Financial (20210320)
Lifestyle kolkata24x7 Until 2021-03 1,517 kolkata24x7-Lifestyle (20210320)
News AsianetNews Until 2021-03 18,423 AsianetNews (20210320)
News The Sentinel Until 2021-03 6,800 The Sentinel (20210320)
News Times of India Until 2021-03 71,969 TOI-News (20210320)
News Zee News Until 2021-04 76,834 Zee-News (2011-2021)
General Multiple sources* NA 223,574 General Corpus
* - General corpus from Wikipedia, Custom NER, Wikinews, PMIndia etc.

English-Tamil

Domain Source Year En-Ta pairs count Corpus Download Link
Legal Multiple Sources 2010-2020 816,729 Indian Judiciary
Legal HC/SUVAS NA 10,851 HC/SUVAS
Legal Legal Terminologies NA 3,595 Legal Terms
General Nouns Dictionary NA 62,920 Nouns Dict
News PIB 2017-2020 104,836 PIB (2017-2020)
News PIB Archives 2014-2016 23,281 PIB Archives (2014-2016)
General Wikipedia Until 2021-02 74,628 Wikipedia (20210201)
Tourism Native Planet Until 2021-03 24,730 Native Planet (20210320)
News OneIndia Until 2021-03 61,937 OneIndia (20210320)
Sports MyKhel Until 2021-03 35,390 MyKhel (20210320)
Automobile DriveSpark Until 2021-03 65,601 DriveSpark (20210303)
Financial GoodReturns Until 2021-03 22,961 GoodReturns-Financial (20210320)
Sports Indian Express Until 2021-03 9,994 IE-Sports (20210320)
Technology Indian Express Until 2021-03 18,651 IE-Technology (20210320)
News Indian Express Until 2021-03 26,235 IE-News (20210320)
Lifestyle Indian Express Until 2021-03 4,356 IE-Lifestyle (20210320)
Entertainment Indian Express Until 2021-03 8,557 IE-Entertainment (20210320)
Education Indian Express Until 2021-03 3,277 IE-Education 20210320)
Financial Indian Express Until 2021-03 832 IE-Financial (20210320)
News AsianetNews Until 2021-03 24,553 AsianetNews (20210320)
News Times of India Until 2021-03 39,835 TOI-News (20210320)
News Zee News Until 2021-04 79,105 Zee-News (2011-2021)

English-Malayalam

Domain Source Year En-Ml pairs count Corpus Download Link
Legal Multiple Sources 2010-2020 1,029,757 Indian Judiciary
Legal HC/SUVAS NA 8,222 HC/SUVAS
Legal Legal Terminologies NA 3,377 Legal Terms
General Nouns Dictionary NA 66,450 Nouns Dict
News PIB 2017-2020 27,538 PIB (2017-2020)
News PIB Archives 2014-2016 13,113 PIB Archives (2014-2016)
General Wikipedia Until 2021-02 44,370 Wikipedia (20210201)
News OneIndia Until 2021-03 48,891 OneIndia (20210320)
Sports MyKhel Until 2021-03 26,526 MyKhel (20210320)
Automobile DriveSpark Until 2021-03 49,663 DriveSpark (20210303)
Tourism Native Planet Until 2021-03 27,024 Native Planet (20210320)
Financial GoodReturns Until 2021-03 16,325 GoodReturns(20210320)
Sports Indian Express Until 2021-03 13,305 IE-Sports (20210320)
Technology Indian Express Until 2021-03 15,318 IE-Technology (20210320)
News Indian Express Until 2021-03 36,337 IE-News (20210320)
General Indian Express Until 2021-03 5,972 IE-General (20210320)
Entertainment Indian Express Until 2021-03 12,430 IE-Entertainment (20210320)
Financial Indian Express Until 2021-03 345 IE-Financial (20210320)
Financial MarketFeed Until 2021-03 5,431 MarketFeed (20210320)
News AsianetNews Until 2021-03 8,932 AsianetNews (20210320)
News Times of India Until 2021-03 27,499 TOI-News (20210320)
News Zee News Until 2021-04 18,935 Zee-News (2011-2021)

English-Telugu

Domain Source Year En-Te pairs count Corpus Download Link
Legal Multiple Sources 2010-2020 1,010,463 Indian Judiciary
Legal HC/SUVAS NA 5,712 HC/SUVAS
Legal Legal Terminologies NA 3,595 Legal Terms
General Nouns Dictionary NA 63,802 Nouns Dict
News PIB 2017-2020 65,842 PIB (2017-2020)
News PIB Archives 2014-2016 15,734 PIB Archives (2015-2016)
General Wikipedia Until 2021-02 82,257 Wikipedia (20210201)
News OneIndia Until 2021-03 71,125 OneIndia (20210320)
Sports MyKhel Until 2021-03 20,912 MyKhel (20210320)
Automobile DriveSpark Until 2021-03 67,688 DriveSpark (20210303)
Tourism Native Planet Until 2021-03 40,733 Native Planet (20210320)
Financial GoodReturns Until 2021-03 17,159 GoodReturns (20210320)
News Sakshi Until 2021-03 113,458 Sakshi (20210320)
News AsianetNews Until 2021-03 10,054 AsianetNews (20210320)
News Times of India Until 2021-03 49,233 TOI-News (20210320)
News Zee News Until 2021-04 23,207 Zee-News (2011-2021)

English-Kannada

Domain Source Year En-Kn pairs count Corpus Download Link
Legal Multiple Sources 2010-2020 859,455 Indian Judiciary
Legal Legal Terminologies NA 3,356 Legal Terms
General Nouns Dictionary NA 54,126 Nouns Dict
News PIB 2017-2020 50,731 PIB (2017-2020)
News PIB Archives 2015-2016 20,910 PIB Archives (2015-2016)
News OneIndia Until 2021-03 49,622 OneIndia (20210320)
Sports MyKhel Until 2021-03 30,416 MyKhel (20210320)
Automobile DriveSpark Until 2021-03 57,151 DriveSpark (20210303)
Tourism Native Planet Until 2021-03 31,654 Native Planet (20210320)
Financial GoodReturns Until 2021-03 15,302 GoodReturns (20210320)
News AsianetNews Until 2021-03 37,204 AsianetNews (20210320)
News Times of India Until 2021-03 76,763 TOI-News (20210320)
News Zee News Until 2021-04 65,327 Zee-News (2011-2021)

English-Marathi

Domain Source Year En-Mr pairs count Corpus Download Link
Legal Multiple Sources 2010-2020 1,205,571 Indian Judiciary
Legal Legal Terminologies NA 1,052 Legal Terms
General Nouns Dictionary NA 57,307 Nouns Dict
News PIB 2017-2020 155,647 PIB (2017-2020)
News PIB Archives 2015-2016 28,622 PIB Archives (2015-2016)
Sports Indian Express Until 2021-03 51,857 IE-Sports (20210320)
News Indian Express Until 2021-03 110,823 IE-News (20210320)
Lifestyle Indian Express Until 2021-03 2,346 IE-Lifestyle (20210320)
Entertainment Indian Express Until 2021-03 49,821 IE-Entertainment (20210320)
Financial Indian Express Until 2021-03 6,869 IE-Financial (20210320)
News Times of India Until 2021-03 581,693 TOI-News (2008-2020)
News The Wire Until 2021-03 25,084 The Wire (20210320)
Entertainment Lokmat Until 2021-05 8,738 Lokmat-Entertainment (20210501)
News Lokmat Until 2021-05 77,811 Lokmat-News (20210501)
Lifestyle Lokmat Until 2021-05 2,686 Lokmat-Lifestyle (20210501)
Sports Lokmat Until 2021-05 33,025 Lokmat-Sports (20210501)
Technology Lokmat Until 2021-05 2,943 Lokmat-Technology (20210501)
Financial Lokmat Until 2021-05 5,509 Lokmat-Financial (20210501)
Healthcare Lokmat Until 2021-05 4,488 Lokmat-Healthcare (20210501)
News Zee News Until 2021-04 125,961 Zee-News (2011-2021)

English-Punjabi

Domain Source Year En-Pa pairs count Corpus Download Link
Legal Multiple Sources 2010-2020 1,261,948 Indian Judiciary
Legal HC/SUVAS NA 10,978 HC/SUVAS
Legal Legal Terminologies NA 3,578 Legal Terms
General Nouns Dictionary NA 53,665 Nouns Dict
News PIB 2017-2020 204,575 PIB (2017-2020)
News PIB Archives 2015-2016 30,923 PIB Archives (2015-2016)
News Tribune Until 2021-03 485,333 Tribune (20210320)
Government Press Release Punjab Govt As of 2021-03 32,589 Punjab Govt (20210320)
News Jagran Until 2021-03 34,162 Jagran-News (20210320)
Technology Jagran Until 2021-03 6,524 Jagran-Technology (20210320)
Education Jagran Until 2021-03 3,155 Jagran-Education (20210320)
Entertainment Jagran Until 2021-03 7,565 Jagran-Entertainment (20210320)
Financial Jagran Until 2021-03 7,741 Jagran-Financial (20210320)
Sports Jagran Until 2021-03 9,034 Jagran-Sports (20210320)
Lifestyle Jagran Until 2021-03 2,528 Jagran-Lifestyle (20210320)

English-Gujarati

Domain Source Year En-Gu pairs count Corpus Download Link
Legal Multiple Sources 2010-2020 1,227,233 Indian Judiciary
Legal Legal Terminologies NA 3,575 Legal Terms
General Nouns Dictionary NA 55,037 Nouns Dict
News PIB 2017-2020 104,591 PIB (2017-2020)
News PIB Archives 2015-2016 29,443 PIB Archives (2015-2016)
News OneIndia Until 2021-03 48,325 OneIndia (20210320)
News Times of India Until 2021-03 85,777 TOI-News (2008-2020)
Government Press Release Gujarat Govt As of 2021-03 5,260 Gujarat Govt (20210320)
News Zee News Until 2021-04 97,375 Zee-News (2011-2021)

English-Assamese

Domain Source Year En-As pairs count Corpus Download Link
News PIB 2017-2020 38,100 PIB (2017-2020)
News PIB Archives 2015-2016 6,188 PIB Archives (2015-2016)
News The Sentinel Until 2021-03 2,886 The Sentinel (20210320)

English-Urdu

Domain Source Year En-Ur pairs count Corpus Download Link
News PIB 2017-2020 141,246 PIB (2017-2020)
News PIB Archives 2015-2016 9,939 PIB Archives (2015-2016)
News The Wire Until 2021-03 10,863 The Wire (20210320)

English-Odia

Domain Source Year En-Or pairs count Corpus Download Link
News PIB 2017-2020 73,020 PIB (2017-2020)
News PIB Archives 2015-2016 19,289 PIB Archives (2015-2016)