cybertraining-dsc / reu2022

Apache License 2.0
2 stars 3 forks source link

suggestions: add debugging and profiling to papermill examples #59

Closed edublancas closed 1 year ago

edublancas commented 1 year ago

Hi,

I noticed you're using papermill on your training material. papermill does not allow debugging or profiling. I wrote a small package to enable that, it'd be a great addition to this repository, happy to open a PR if you want!

For debugging, it'll dump the traceback object if the notebook fails, allowing you to execute a command to start a debugging session, especially useful when running papermill jobs remotely! Recording here.

pip install ploomber-engine
papermill input.ipynb output.ipynb --engine debuglater

For profiling: since papermill executes notebooks in a separate process, monitoring the papermill process won't yield any useful information; with the new engine, the notebook executes in the same process. Recording here.

pip install ploomber-engine
papermill input.ipynb output.ipynb --engine profiling
laszewsk commented 1 year ago

will review at one point. debug seems not to be that useful for us. profiling may be. however we can add timer and have stopwatch, example of profiling is to limited to judge if it useful

edublancas commented 1 year ago

no problem! closing this