Using NLI dataset for as Development dataset

[zarmeen92]

Hi, I am fine tuning training_nli_bert.py script using bert-multilingual model for generating sentence embeddings in Urdu language. I have only NLI dataset available for training and evaluation. My question is that can we use NLI dev-dataset in place of STS dataet for evaluation purpose? Are there any cons of using NLI dataset on the quality of sentence embeddings? What changes I need to made in the following code

logging.info("Read STSbenchmark dev dataset")
dev_data = SentencesDataset(examples=sts_reader.get_examples('sts-dev.csv'), model=model)
dev_dataloader = DataLoader(dev_data, shuffle=False, batch_size=batch_size)
evaluator = EmbeddingSimilarityEvaluator(dev_dataloader)

@nreimers please help

[nreimers]

Hi @zarmeen92 You can use an NLI dataset for devset. Note, in the current stable version, the LabelAccuracyEvaluator contains a bug. See issue #26 and #27 how to fix it (its fixed in the v2.0.4 branch.

But another problem arises: That training on NLI generates good sentence embeddings is a bit of magic, which worked for unknown reason quite well for English. But for other languages, I think there is not guarantee that it produces good sentence embeddings.

[zarmeen92]

@nreimers I have corrected the lines as suggested in #27 but still I am getting this error on evaluator=LabelAccuracyEvaluator(dataloader=dev_dataloader,softmax_model=train_loss)

init() got an unexpected keyword argument 'softmax_model'

[nreimers]

It appears that the corrected code is not used in your app. I recommend to install the package with pip install -e .

[zarmeen92]

Thanks @nreimers the issue is resolved