TheMrZZ
commented
2 years ago After a lot of digging around, I managed to pin down the line causing the problem.
It's the line 142 in loops/epoch/training_epoch_loop.py :
class TrainingEpochLoop(loops.Loop[_OUTPUTS_TYPE]):
...
def on_run_start(self, data_fetcher: AbstractDataFetcher, **kwargs: Any) -> None:
# hook
self.trainer.logger_connector.on_epoch_start()
self.trainer.call_hook("on_epoch_start")
self.trainer.call_hook("on_train_epoch_start")
self.trainer.fit_loop.epoch_progress.increment_started()
self._reload_dataloader_state_dict(data_fetcher)
--> self._dataloader_iter = _update_dataloader_iter(data_fetcher, self.batch_idx + 1)Therefore, the culprit is:
def _update_dataloader_iter(data_fetcher: AbstractDataFetcher, batch_idx: int) -> Iterator:
"""Attach the dataloader."""
if not isinstance(data_fetcher, DataLoaderIterDataFetcher):
# restore iteration
dataloader_iter = enumerate(data_fetcher, batch_idx)
else:
dataloader_iter = iter(data_fetcher)
return dataloader_iterThe on_run_start hook is called from loops/base.py :
class Loop(ABC, Generic[T]):
...
def run(self, *args: Any, **kwargs: Any) -> T:
if self.skip:
return self.on_skip()
self.reset()
--> self.on_run_start(*args, **kwargs)
...And this run method is called from loops/fit_loop.py :
class FitLoop(Loop):
...
def advance(self) -> None:
"""Runs one whole epoch."""
dataloader = self.trainer.training_type_plugin.process_dataloader(self.trainer.train_dataloader)
data_fetcher = self.trainer._data_connector.get_profiled_dataloader(dataloader)
with self.trainer.profiler.profile("run_training_epoch"):
--> self.epoch_loop.run(data_fetcher)
...The problem is the data_fetcher. It's indeed related to dataloaders, as expected. I still don't know what is the root reason, but I'll try to find it.
a-kore
marcm-ml
carmocca
tchaton
isvogor-foi
amin-nejad
carapas
JonathanSum
mayiran1999
KameniAlexNea
gikok
lminer
rohitgr7
jzazo
awaelchli
is-jlehrer
Jason94
aktgpt
leventt
otakudj
I converted some Pytorch code to Lightning. The dataset is loaded lazily by the train & eval dataloaders.
However, when moving the code to Lightning, I noticed a huge slowdown. After digging around, I noticed that there was a ~10 seconds delay between each epoch. For comparison, on my vanilla Pytorch, an epoch takes ~4s.
I first thought it was a data loading problem, but during the 10s delay, no data is loaded (at least that's what my
printtell me).I think the issue is related to the number of workers, because setting
n_workers=0solves the problem (but is slower in the end, since only one worker is not enough). I know starting workers is slow, however I havepersistent_workers=Trueand this does not happen in normal Pytorch. My data loaders also havepin_memory=True(removing pin_memory does not solve the problem).Since this is company code, I cannot disclose the before/after, but I'll try to "anonymize" some code if necessary. Here is the lightning module:
Here is the result of
profiler="simple":Here is the result of
profiler="advanced": https://pastebin.com/q3C5P826.Finally, here is a video demonstrating the problem. I'm printing each piece of data loading, to prove it's not the issue. https://user-images.githubusercontent.com/30944236/140587623-ae184fa3-370a-42be-8593-200026d11ba4.mp4
Random informations:
conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorchcc @tchaton @rohitgr7 @borda @akihironitta