Better code profiling

[sarthakpati]

Is your feature request related to a problem? Please describe. Currently, all the code profiling is custom based on internal functions from psutil and/or pytorch. It would be great to have a more uniform solution.

Describe the solution you'd like I found a few options:

• Torch Profiler: Pros: extensive pytorch-specific debugging information; cons: requires changes to the training/inference apis of GaNDLF
• Scalene: Pros: straightforward start and stop for pieces of code that are to be profiled; cons: generic hardware-level information
• octoml: Pros: potential acceleration of prediction; cons: requires modification of model code, PyTorch-2.0 needed
• DLProf: Pros: CLI-based; cons: doesn't seem to give pytorch-specific information
• HolisticTraceAnalysis: Pros: tight integration with PyTorch; cons: very new library so might need to see how this develops

We should have a detailed discussion on any potential requirements and how we can evaluate these libraries.

Describe alternatives you've considered N.A.

Additional context N.A.

[github-actions[bot]]

Stale issue message

[github-actions[bot]]

Stale issue message

[github-actions[bot]]

Stale issue message

[github-actions[bot]]

Stale issue message

[github-actions[bot]]

Stale issue message

[github-actions[bot]]

Stale issue message

[github-actions[bot]]

Stale issue message