search

stas00 / ml-engineering

Machine Learning Engineering Open Book
https://stasosphere.com/machine-learning/
Creative Commons Attribution Share Alike 4.0 International
11.57k stars 703 forks source link

Performance Profiling #71

Closed jeromeku closed 3 weeks ago

jeromeku commented 1 month ago

@stas00

Wondering if you have any tips & tricks for working with performance profiling tools such as nsys? Or recommendations for systematically optimizing model architecture and single / multi-node training workflows?

stas00 commented 1 month ago

Wondering if you have any tips & tricks for working with performance profiling tools such as nsys?

I don't have experience with nsys.

Or recommendations for systematically optimizing model architecture

Neural Architecture Search (NAS) https://en.wikipedia.org/wiki/Neural_architecture_search? e.g. see https://developer.nvidia.com/blog/advancing-the-accuracy-efficiency-frontier-with-llama-3-1-nemotron-51b/ though I have no direct experience with it.

and single / multi-node training workflows?

This part is too vague for me to understand what you're asking about? Can you be more specific?

stas00 commented 3 weeks ago

Closing due to inactivity. Please feel free to re-open if needed.