search

jupyter / nbconvert

Jupyter Notebook Conversion
https://nbconvert.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
1.73k stars 564 forks source link

Output notebook while executing #648

Open matthias-k opened 7 years ago

matthias-k commented 7 years ago

I'm using nbconvert to execute notebooks from the command line. Some of these notebooks take a long time and I don't get any feedback on the progress (and e.g. potential convergence problems etc) unless I save some results to files in the notebook. Is there any way to e.g. e.g. write the cell and the output to the console (e.g. as markdown) while processing the notebook? This would make things a lot easier.

Pita commented 6 years ago

Agreed!

AlJohri commented 5 years ago

is there a way to force a print statement to go to the command line? I found that if I use a ProcessPoolExecutor the print statements end up the the terminal but I was hoping for a way to replicate this without a separate process

screen shot 2018-11-29 at 10 29 59 pm
RoyalTS commented 5 years ago

related to #374?

MSeal commented 5 years ago

Also https://papermill.readthedocs.io/en/latest/ supports this as a wrapper to nbconvert for notebook-to-notebook transformation. I can look into pulling this functionality into the preprocessor when I get back to finishing https://github.com/jupyter/nbconvert/pull/905 (or shortly there-after).

laurentperrinet commented 3 years ago

Somewhat related to https://github.com/jupyter/nbconvert/issues/709 which suggests to save the notebook every T minutes.

Another option could be to add the possibility to save the notebook anytime a cell is executed.

MSeal commented 3 years ago

Generally at this point the advice is to use papermill for notebook execution when you want periodic saves and deeper reporting on execution. It's unlikely that this will be copied over to nbconvry, and probably not worth the library duplication, as papermill was made specifically to isolate complex execution concerns and execution ability in nbconvert is left for compatibility and ease of combining with template conversions in a singular call.

laurentperrinet commented 3 years ago

I did have a try at papermill and it works as expected:

papermill  --request-save-on-cell-execute --progress-bar input.ipynb output.ipynb

Thanks, Laurent