Hi, I want to report an issue observed when running inference for a classification task.
Description
When running inference (either do_eval=True or do_predict=True), the results are different whether it's in distributed mode (multiple gpus ) or not (one gpu).
Use cola.sh to evaluate --init_model deberta-v3-large
Set other paramaters: --do_eval, --eval_batch_size 4, using 3 gpus CUDA_VISIBLE_DEVICES=7,5,6 or 1 gpu CUDA_VISIBLE_DEVICES=7
Check oder of instances after logit calculation is done and collected from all gpus. For example, print the first 10 instances in predicts and labels after the above merge_distributed
1 gpu case - as expected according to the input file:
Hi, I want to report an issue observed when running inference for a classification task.
Description
When running inference (either
do_eval=True
ordo_predict=True
), the results are different whether it's in distributed mode (multiple gpus ) or not (one gpu).When doing evaluation, data is prepared sequentially in batches using
SequentialSampler
,BatchSampler
, andDistributedBatchSampler
https://github.com/microsoft/DeBERTa/blob/4d7fe0bd4fb3c7d4f4005a7cafabde9800372098/DeBERTa/apps/run.py#L172 and then sent to gpus. But once logits are computed, there is a step to gather results across devices -merge_distributed
. https://github.com/microsoft/DeBERTa/blob/4d7fe0bd4fb3c7d4f4005a7cafabde9800372098/DeBERTa/apps/run.py#L228 After this step, the order of data instances is no longer similar to the one in the original input file (dev.tsv or test.tsv) in case of distributed modes with multiple gpus, resulting in a different accuracy.Steps to reproduce
cola.sh
to evaluate--init_model deberta-v3-large
--do_eval
,--eval_batch_size 4
, using 3 gpusCUDA_VISIBLE_DEVICES=7,5,6
or 1 gpuCUDA_VISIBLE_DEVICES=7
Check oder of instances after logit calculation is done and collected from all gpus. For example, print the first 10 instances in
predicts
andlabels
after the abovemerge_distributed
1 gpu case - as expected according to the input file:
3 gpu case:
Additional information
My system setup is: