deepskies / DeepSZSim

MIT License
2 stars 0 forks source link

profiling for speed and memory usage #24

Open bnord opened 1 year ago

bnord commented 1 year ago

Notes and ideas on how to profile our code

  1. https://docs.python.org/3/library/profile.html
  2. there's also timeit
  3. nice overview: https://towardsdatascience.com/how-to-profile-your-code-in-python-e70c834fad89
  4. https://codesolid.com/how-do-i-profile-python-code/
  5. https://github.com/dfeneyrou/palanteer
  6. https://github.com/joerick/pyinstrument
  7. https://github.com/benfred/py-spy
  8. https://jiffyclub.github.io/snakeviz/
  9. https://github.com/sumerc/yappi/
  10. memory
    1. https://pypi.org/project/guppy/
    2. psutil
    3. https://pypi.org/project/memory-profiler/
    4. https://www.geeksforgeeks.org/monitoring-memory-usage-of-a-running-python-program/
evavagiakis commented 1 year ago

Do we want to add this to our initial release to-dos @bnord @samueldmcdermott ?

samueldmcdermott commented 1 year ago

I've always just used the ipython magic %timeit on various parts of my code, so I've never done anything more systematic or comprehensive. Could be interesting, or it could be a huge time suck and diversion from physics -- I'm not sure how to assess! I think it's probably worthwhile to at least have some sense of how long your code takes (eg to make sure you're not doing an unnecessary for loop when you could be doing a numpy list comprehension) and I'm happy to help with that at some point

bnord commented 1 year ago

I think that we should do something so that we understand how time and memory scale as a function of the number of objects.

I don't think this needs to be a part fo the package.