If we use dlp.iter inside main.py, this does not reflect the real fetch data since the data is already fetched inside DataLoader.next(). Because of this, we ended up tracing the fetching of batch data that resides in the memory instead of tracing its fetching duration from filesystem or other sources.
This PR addresses the problem where we put dlp.iter inside the dataloader, as close as possible to the dataset itself.
For dali dataloader, I haven't implemented the fix since there is no visible iteration there (DALI uses share_outputs)
Note: @hariharan-devarajan when we bump the dftracer version, we should provide the name of the dlp.iter.
I left a note to change dlp.iter to dlp.iter(..., name=self.next.__qualname__)
Hi @zhenghh04 and @hariharan-devarajan,
If we use
dlp.iterinsidemain.py, this does not reflect the real fetch data since the data is already fetched insideDataLoader.next(). Because of this, we ended up tracing the fetching of batch data that resides in the memory instead of tracing its fetching duration from filesystem or other sources.This PR addresses the problem where we put
dlp.iterinside the dataloader, as close as possible to the dataset itself.For dali dataloader, I haven't implemented the fix since there is no visible iteration there (DALI uses
share_outputs)Note: @hariharan-devarajan when we bump the dftracer version, we should provide the name of the
dlp.iter. I left a note to changedlp.itertodlp.iter(..., name=self.next.__qualname__)