pytorch / data

A PyTorch repo for data loading and utilities to be shared by the PyTorch domain libraries.
BSD 3-Clause "New" or "Revised" License
1.08k stars 142 forks source link

[StatefulDataLoader] macOS tests are too slow #1247

Closed andrewkho closed 2 months ago

andrewkho commented 2 months ago

🐛 Describe the bug

test_state_dict is very slow on macOS (and slows down CI), likely because of macOS default multiprocessing_context being spawn instead of fork. The StatefulDataLoader tests on macOS take ~1.5 hours, vs 10 minutes on Linux and Windows.

Example of test-runtimes on my local mac:

image

We should a) update CI to log test times, b) for macOS, drop some of the tests. Each test_mp* test runs 6x, and if we have coverage from Linux + Win then we probably don't need all of them for mac

Versions

Nightly