PyTorch multiprocessing pool causes OSError Too Many Files Open on Linux

[minafarid]

Currently, using torch.multiprocessing in InitAttFeaturizer fails on linux and not on macOS. The reported error is OSError Too Many Files Open. Similar issues have been reported on PyTorch github repo:

• https://github.com/pytorch/pytorch/issues/11201 (with a suggested solution)
• https://github.com/pytorch/pytorch/issues/11899

Documentation for sharing strategies:

• https://pytorch.org/docs/master/multiprocessing.html?highlight=sharing%20strategy#sharing-strategies

From the documentation:

file_descriptor is the default strategy (except for macOS and OS X where it’s not supported).