Adding memory profiling docs

MAAP-Project / maap-documentation

9 stars 12 forks source link

Adding memory profiling docs #371

Closed omshinde closed 4 months ago

omshinde commented 5 months ago

Documentation for performing memory profiling any Python script.

review-notebook-app[bot] commented 5 months ago

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

wildintellect commented 5 months ago

This is a good start. I think users might be a little confused about implementing in the ADE vs DPS.

Can you clarify how to specify where the output file from the profiling ends up? Maybe also include a note about getting the output from a DPS job.
How about how to call an existing python file? possibly doing the calls from a notebook, and where to see the results.
[ ] Nice to have: a paragraph at the top about when to implement this, generally after you already have working code and you want to know what size worker to use in DPS, or to look for ways to reduce the RAM.

omshinde commented 4 months ago

Thanks @wildintellect @jjfrench @smk0033 for the review comments.

Can you clarify how to specify where the output file from the profiling ends up? Maybe also include a note about getting the output from a DPS job. @wildintellect I've specifically mentioned that this tutorial is for memory profiling in ADE and can be used for estimating DPS worker size.

How about how to call an existing python file? possibly doing the calls from a notebook, and where to see the results. Good point. Added a logging example for saving the output as a log file. It would be good for the users to save the memory profiling output and share later.

chuckwondo commented 4 months ago

Looking good. One more suggestion is to remove all occurrences in the python files and notebook of this:

"""
... Code here
"""

Given that the block is a comment block, it's confusing to see ...Code here inside the comment. I think we can leave it out and readers will realize that there can be other code since we're showing only dummy functions.

omshinde commented 4 months ago

Thanks @chuckwondo ..Updated the notebook accordingly.

omshinde commented 4 months ago

Close https://github.com/NASA-IMPACT/active-maap-sprint/issues/861