search

aws / amazon-s3-plugin-for-pytorch

Apache License 2.0
168 stars 21 forks source link

Thread safety: Dataloaders (with multiple workers) only supports multiprocessing_context "spawn" #18

Closed rehno-lindeque closed 2 years ago

rehno-lindeque commented 2 years ago

This may be a documentation issue:

Pytorch dataloaders only appear to work when multiprocessing_context is set to "spawn". (or when workers=0)

At least with pytorch 1.10

>>> torch.__version__
'1.10.0+cu113'

I obtain the error 

ERROR: Unexpected segmentation fault encountered in worker.

unless multiprocessing_context = "spawn" is explicitly set.

ydaiming commented 2 years ago

@rehno-lindeque

We're upstreaming the amazon-s3-plugin-for-pytorch into the torchdata package (https://github.com/pytorch/data/pull/318). We're dropping support for this plugin.

Thanks for raising this issue. However, we're not updating this repository anymore. If the issue still occurs in the new torchdata package, we'll investigate and resolve the issue.

rehno-lindeque commented 2 years ago

Thanks @ydaiming. I've already started transitioning to torchdata myself so that is pretty convenient :+1: