Closed rajveerb closed 10 months ago
I have completed:
The pytorch profiler logs are too large even without our added augmentation logic.
I am creating a file compressor using LZMA library.
Finished #21 which will allow me to push the compressed files.
The goal is to log the preprocessing operations' span by instrumenting PyTorch/PyTorch vision's source code and convert the collected log to a format that PyTorch profiler produces for its own data.
Span for each event includes the following data:
In PyTorch profiler's case, process id and thread id are the same.
A script is needed to merge the converted logs to merge with PyTorch profiler json file.
Above will allow us to visualize the preprocessing operations performed by the DataLoader workers just like the PyTorch profiler visualization.