search

JuliaLogging / TerminalLoggers.jl

Logging sinks and utilites for interactive terminals
MIT License
41 stars 10 forks source link

Progress bars in Jupyter notebooks #25

Closed devmotion closed 3 years ago

devmotion commented 4 years ago

I tried to use TerminalLoggers for displaying progress bars in a Jupyter notebook (I just ran the example from the docs), but unfortunately the progress bar is not updated correctly. It seems TerminalLoggers does not clear existing progress bars, and hence I ended up with multiple lines of incomplete progress bars instead.

By using the superseded ConsoleProgressMonitor I can workaround this issue, but I'm wondering if there exists a workaround or fix for TerminalLoggers?

tkf commented 4 years ago

I think the easiest short-term solution is to make ConsoleProgressMonitor work with ProgressLogging 0.1.1 https://github.com/tkf/ConsoleProgressMonitor.jl/issues/9

In a long run, I think we'd need Jupyter Lab/Notebook -specific progress monitors, just like you get the GUI progress bar in Juno.

tkf commented 4 years ago

ref https://github.com/tkf/ConsoleProgressMonitor.jl/issues/10

devmotion commented 4 years ago

Wow, thanks for the prompt reply and the update!

c42f commented 4 years ago

We should definitely have a solution for Jupyter but I think it may not be in TerminalLoggers itself — TerminalLogger is largely assuming it's talking to a terminal with limited display capabilities (ANSI codes for color and some limited cursor control).

We should probably add a logger sink to IJulia which uses MIME"text/html" for formatting log records rather than text, fancy progress bar widgets etc etc. I'd also love for the log records to come up separately in a way which is somehow easiliy browsable in Jupyter without interfering with the "normal" output of a cell.

c42f commented 4 years ago

This would also fix the problem with IJulia capturing stderr text-based logs and Jupyter coloring them that pink error-looking color regardless of whether they're Info messages, etc.

tkf commented 4 years ago

I think it may not be in TerminalLoggers itself

I agree. I think it should be in somewhere else, like JupyterLogger.jl or something.

We should probably add a logger sink to IJulia

I'm not following Jupyter ecosystem for a while so I'm not 100% sure, don't you need to use different mechanisms for registering dynamic HTML/JavaScript components for each frontend (Jupyter Lab/Jupyter Notebook)?

Looking at https://github.com/nteract/nteract/issues/4573, maybe using whatever ipywidgets is using could be enough for supporting a wide range of frontend (including nteract).

I'd also love for the log records to come up separately in a way which is somehow easiliy browsable in Jupyter without interfering with the "normal" output of a cell.

For "normal" Jupyter usage (i.e., the kernel is local) it might be just enough to open a separate GUI window. Here is a prototype that only handles progress bars: https://github.com/tkf/ElectronProgressBars.jl. Or open a terminal in a separate window and then use TerminalLoggers.jl inside https://github.com/tkf/TmuxDisplays.jl

c42f commented 4 years ago

I think it should be in somewhere else, like JupyterLogger.jl or something.

Fine by me to start off! Does it need to be separate from IJulia in the longer term if IJulia is already supporting both JupyterLab and Jupyter Notebook? I haven't looked into the code behind the jupyter ecosystem so I don't know how responsibilities for frontend and backend are split there.

tkf commented 4 years ago

Actually, I don't personally have a plan to start implementing Jupyter-based loggers (sorry!). So, if whoever wants to do this thinks doing it in IJulia makes sense, maybe that's a good idea.

Does it need to be separate from IJulia in the longer term if IJulia is already supporting both JupyterLab and Jupyter Notebook?

I thought you'd need HTML widgets but it looks like there is now update_display_data in the wiring protocol. It sounds like this is all we need but I'm not sure if IJulia.jl supports this message type.

c42f commented 4 years ago

Actually, I don't personally have a plan to start implementing Jupyter-based loggers (sorry!).

Haha! No worries. As an occasional user of Jupyter I may end up doing this, but not sure when I'd get around to it. (If anyone wants it soon they should jump right in ;-) )

cossio commented 3 years ago

Bump. Any updates on this?

c42f commented 3 years ago

I now use Pluto rather than IJulia so if I was to work on this I'd put the effort into Pluto integration. But I don't even have time to work on that right now. In any case, it's outside the scope of this package to support Jupyter in any kind of properly integrated way so I should close this issue.

If there's any non-intrusive tweaks we can make to have TerminalLoggers and Jupyter interact less strangely then I'm happy to accept patches. But the real solution is a Jupyter-specific logging backend which integrates with the Jupyter GUI in some way.