When debugging the evalution_model.py in multi_gpu case, there occurs a debugger port collision between parent and child processes. In order to avoid this: nested the evaluation_model.py within name == 'main'.
The expected behaviour of sentence_transformer_encode_multi_process_worker() function is to encode the corpus, evaluate the score against all the queries and store the metric. This was not properly handled by this function. Hence made use of the default SentenceTransformer._encode_multi_process_worker().
Appended instruction in sentences list in encode_corpus_parallel(). This change is made with reference taken from encode_corpus() function.
USE_BEIR_DEVELOPMENT is removed from evaluation/MTEB/mteb/abstasks/BeIRTask.py. This boolean just skips cqadupstack dataset download. I do not find any valid reason to do so. Hence removed.
Added explicit requirement pyarrow==8.0.0. Pip auto dependency resolver installs the latest version of pyarrow and that does not play well with the evaluate package.
Added necessary changes in train.py in order to feed in validation dataset.
Issues fixed:
sentence_transformer_encode_multi_process_worker()function is to encode the corpus, evaluate the score against all the queries and store the metric. This was not properly handled by this function. Hence made use of the defaultSentenceTransformer._encode_multi_process_worker().encode_corpus_parallel(). This change is made with reference taken fromencode_corpus()function.USE_BEIR_DEVELOPMENTis removed from evaluation/MTEB/mteb/abstasks/BeIRTask.py. This boolean just skipscqadupstackdataset download. I do not find any valid reason to do so. Hence removed.evaluatepackage.