Crashes on 2xT4 GPUs

[Philmod]

🐛 Bug Report

Catalyst fails on 2xT4 GPUs.

We install Catalyst in the Kaggle base image. This week we wanted to release a new image with upgraded packages. It doesn't look like Catalyst was upgraded, but Accelerate was (from 0.12 to 0.13.1).

How To Reproduce

Steps to reproduce the behavior: Run this unit test on a 2xT4 GPU.

Code sample

https://github.com/Kaggle/docker-python/blob/main/tests/test_catalyst.py

Screenshots

Expected behavior

The test passes on a P100 GPU.

Environment

https://gist.github.com/Philmod/0349a2cf16d76e8d20e960d750962241

Checklist

• [x] bug description
• [x] steps to reproduce
• [x] expected behavior
• [x] environment
• [x] code sample / screenshots

FAQ

Please review the FAQ before submitting an issue:

• [x] I have read the documentation and FAQ
• [x] I have reviewed the minimal examples section
• [x] I have checked the changelog for main framework updates
• [x] I have read the contribution guide

[github-actions[bot]]

Hi! Thank you for your contribution! Please re-check all issue template checklists - unfilled issues would be closed automatically. And do not forget to join our slack for collaboration.

[Philmod]

Update: pinning accelerate package to 0.12.0 fixes the problem.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.