search

TUM-DAML / seml

SEML: Slurm Experiment Management Library
Other
168 stars 30 forks source link

Feature Request: Profiling #75

Open siboehm opened 2 years ago

siboehm commented 2 years ago

I believe many projects (and many cluster quotas) would profit from having easy access to profiling for their experiments. In my seml projects I used this code: https://gist.github.com/siboehm/bf69a17cc9bca71c37a2fae0214a1eeb It profiles the whole experiment using py-spy. It works well even for long running experiments (many hours). The size of the profile is a few MB and the CPU overhead of the profiler is pretty low. It runs in a subprocess and doesn't require changes to the main experiment.

Would you be interested in getting a PR that integrates profiling via py-spy into seml directly? If not, should I add the Gist as an example or can we link to it somehow?

gasteigerjo commented 2 years ago

Sure, this sounds pretty useful! Utility functions for the experiment go into seml/experiment.py.