ParsiNLU

ParsiNLU is a comprehensive suit of high-level NLP tasks for Persian language. This suit contains 6 different key NLP tasks --- Reading Comprehension, Multiple-Choice Question-Answering, Textual Entailment, Sentiment Analysis, Query Paraphrasing and Machine Translation.

These challenges are collected by expert annotators native in Persian language and from various sources. If you'd like to see additional details on the dataset and how we collected, please check out our publication.

Getting the data

You can find the data under the data/ directory.

Alternatively, you can also access the data through the HuggingFace🤗 library. Here is an example:

from datasets import load_dataset

dataset = load_dataset("persiannlp/parsinlu_entailment")
for x in dataset['train']
  print(x)

Using the finetuned models using the HuggingFace🤗 library

Our models are deployed on HuggingFace's model hub. You can our list of models in this page.
Each model readme contains descriptions on how to use it.

Examples and Baselines

First, make sure you have the data your data/ directory.

Set up your environment. You can use conda or virtualenv to create a Python environment. We have tested the code with Python3.7, but it should work on any version >= 3.5. Make sure to install all the necessary dependencies:

> python install -r requirements.txt

If you're using a GPU, make sure that you set the appropriate environmental variable. For example:

export CUDA_VISIBLE_DEVICES=YOUR_GPU_ID # for example, YOUR_GPU_ID can be 0 for using your first GPU; or 0,1 if you're using two GPUs

See the relevant section on how to train models for each task:

• Textual entailment
• Query Paraphrasing
• Reading Comprehension
• Multiple-choice QA
• Machine Translation
• Sentiment Analaysis

Textual Entailment

Textual Entailment is the task of deciding whether a whether two given questions are paraphrases of each other or not.

Here are several examples:

To reproduce our baselines (except mT5 models; see this), try train_and_evaluate_entailment_baselines.sh script.

You can also try our existing models. Visit this page to see an example.

Query Paraphrasing

QQP is the task of detecting whether two given questions are paraphrases of each other or not.

Here are several examples:

To reproduce our numbers with all our baselines, try train_and_evaluate_qqp_baselines.sh script.

You can also try our existing models. Visit this page to see an example.

Reading Comprehension

In this task, the goal is to generate a response to question and its accompanying context paragraph. Here are several examples:

To reproduce our baselines (except mT5 models; see this, try train_and_evaluate_reading_comprehension_baselines.sh script.

You can also try our existing models. Visit this page to see an example.

Multiple-Choice QA

Here the task is to pick a correct answer among 3-5 given candidate answers. Here are several examples:

To reproduce our baselines, try train_and_evaluate_multiple_choice_baselines.sh script.

You can also try our existing models. Visit this page to see an example.

Machine Translation

Machine Translation of Persian/English is one of the few tasks that has received more work in the past few years. Unfortunately, most of the evaluation done for this task is often limited to few domains/datasets.
Here we have compiled a collection of high-quality resources for Persian machine translation. Specifically, here is our collection of evaluation sets:

• Quaran: BVased on the existing translations of Quran.
• Bible: Based on the existing translations of Bible.
• Mizan: Parallel corpora constructed from human translations of literary masterpieces.
• Global Voices:
• Quora queries: the translation instance extracted from our query paraphrasing task.