search

pytorch / data

A PyTorch repo for data loading and utilities to be shared by the PyTorch domain libraries.
BSD 3-Clause "New" or "Revised" License
1.12k stars 149 forks source link

early stop worker got Exception Error #1310

Open kamirdin opened 1 month ago

kamirdin commented 1 month ago

🐛 Describe the bug

dp = ...
dp = dp.sharding_filter()
rs = MultiProcessingReadingService(num_workers=4)
dataloader = DataLoader2(dp, reading_service=rs)

for _ in dataloader:
    pass

dataloader.shutdown()
Exception: Can not expect any response without submitted request
This exception is thrown by __iter__ of _IterateQueueDataPipes(datapipes=[QueueWrapper, QueueWrapper, QueueWrapper, QueueWrapper])

related to https://github.com/pytorch/data/commit/901b48312179865657b90792aa03f0775ec50971

Versions

[pip3] numpy==1.26.4 [pip3] pytorch-wpe==0.0.1 [pip3] rotary-embedding-torch==0.6.4 [pip3] torch==2.1.1 [pip3] torch-complex==0.4.4 [pip3] torchaudio==2.1.1 [pip3] torchdata==0.7.1

andrewkho commented 1 month ago

Hi @kamirdin, please see https://github.com/pytorch/data/issues/1196 as we won't be supporting dataloader2 nor datapipes in torchdata moving forward