Closed lixinye-nju closed 1 year ago
@microsoft-github-policy-service agree
Actually, we don't need DistributedSampler since we make different local_rank load different data.
Refer to https://github.com/microsoft/CodeXGLUE/blob/main/Code-Code/CodeCompletion-token/code/dataset.py#L145
input_ids = input_ids[local_rank*length: (local_rank+1)*length]
.
Thanks for your reply @celbree. Sorry for the misunderstanding since I run the training process with my own dataset and ignore the detail.
Previous version failed to set
train_sampler
as distributed data sampler and now it's fixed.