Open SingL3 opened 1 year ago
Thanks for reporting this! Could you please try a few follow-on things?
In torch, distributed samplers duplicate data to ensure it is even across all ranks. We've added code to correct for this, but it is possible it is buggy. I'd like to ensure you're running on the latest version with these fixes and confirm it's the same issue and not something else
Environment
To reproduce
Steps to reproduce the behavior:
icl_tasks:
... batch_size: 4 # to 32