Monthly Performance Meetings: 2021

fcollonval commented 3 years ago

Here is a record of the agenda/minutes for our monthly meeting. Everyone is welcome to join.

When: Every 4 weeks Wednesday 10AM Pacific Time
Where: jovyan Zoom

Let's avoid using this thread for discussion. If you'd like to discuss something in the minutes, open a separate issue and reference this thread.

Meeting Notes

fcollonval commented 3 years ago

04 Aug 2021 Monthly meeting

Attendees

Name	affiliation	username
Zach Sailer	Apple	@Zsailer
Frederic Collonval	QuantStack	@fcollonval
A. T. Darian	Two Sigma	@afshin
Mike		@krassowski
William Stein	SageMath/CoCalc	@williamstein
Piyush Jain	AWS	@3coins
Gonzalo	Quansight, inc	@goanpeca
Brian		@ellisonbg
Eric	Datalayer	@echarles
Marc		@mlucool

Notes

Marc
- Try to target run notebook with a thousand cells
- Causes seen: IDE aspect of JLab compared to classical notebooks at to the stack
- Classical notebook has optimization for long output
  - JupyterLab output has not the same control that the classical notebook
- Internally virtualization was coded for the widgets
- Improve validation in nbconvert to load notebook faster (fast validation and avoid double check)
- Virtualization is probably only possible if we use React to deal with output state
  - [Marc] https://github.com/jupyterlab/jupyterlab/issues/10103
Darian
- It will be good to benchmark classical notebook vs retrolab that behaves the same
- Vision: we should be able to deal with any size of notebook
  - This means that at some points streaming will be needed
  - Client & server virtualization will be needed
    - [Marc] Trouble with search on that front
- About using React: not using React as the main framework is mainly historic
Eric
- Benchmarking JupyterLab vs classical is about +10%
- When virtualizing a notebook but still having all codemirror to refresh.
- Current work:
  - Some bugs still to address on the strip outputs front.
  - Trial around the virtual notebook, the troubles is the state of the output.
  - Bug ahead probably with collapsing header and current virtualization
Mike
- As user, switching the tabs is slow
  - Especially switching to a big notebook
  - [Marc] Link to the refresh cascade
  - [Eric] Lots of relayout
- Has any one tried to put a notebook in an iframe?
  - [Darian] not tried - one reason is to be able to drag-and-drop things (iframe is blocking that)
  - [Brian] there is a performance overhead to iframe that may wash out (especially with multiple iframe)
  - [Marc] Better using the classical notebook approach
  - [William] in CoCalc is an iframe that is shown/hidden (using the classical notebook)
    - Colab is using it for every cells (security is more important than performance for them)
William
- Instead of using multiple editors, using only one editor that is active (has focus). It can help a lot.
  - it combines nicely with virtualization
  - though part is preserving selection and position of click in non-codemirror editor.
    - especially to deal with real time collaboration using CodeMirror.runMode
- There is a lot of outputs that are off-loaded except if the user request it.
- Can share some information.
- https://github.com/sagemathinc/cocalc/blob/master/src/packages/frontend/jupyter/codemirror-static.tsx
- About using React
  - There are lots of components that exists (for example from nteract)
    - Hard but doable - like 3 months
    - But high impact on the community
  - But it does not solve all problems
    - Need to deal with un/mount lifecycle
  - [Darian] on the long term, will a React component based of CoCalc be more efficient than a Lumino version?
    - Probably yes
  - [Brian] flexbox or table?
    - More probably flexbox
    - Chris did some benchmark that table is a bit more performant
Brian
- About using React
  - Complex to reimplement a fully functional React implementation (or any other framework)
  - So probably try other paths to improve the situation
- iframe Security
  - Containment should not be part of JupyterLab because it depends on the deployment scenario
  - If we use more iframes, this means we need to increase public API to allow extensions,... to access the various parts of the app.

Main pain points

Switch tabs

Switching between tabs, especially when going to a big notebook

Opening notebook

Next steps

Set up a monthly meeting on the subject --> @fcollonval
Measure
- Improve measurement; ideally on CI --> @fcollonval
Benchmark
- [Marc] gathers telemetry to get feedback from real user cases to know what to address in priority
  - Idea would be to be able to dump a file that user can update
  - [Brian] this needs to be opted in
    - idea push telemetry and find organization (like university) to publish their figures
- Classical vs retrolab
Solutions to try
- Analyze the switch tabs to find possibilities small patches
- One codemirror instance
- React based component
- iframe trial

Links

meeseeksmachine commented 3 years ago

This issue has been mentioned on Jupyter Community Forum. There might be relevant details there:

https://discourse.jupyter.org/t/performance-analysis-and-improvements/10126/14

meeseeksmachine commented 3 years ago

This issue has been mentioned on Jupyter Community Forum. There might be relevant details there:

https://discourse.jupyter.org/t/performance-analysis-and-improvements/10126/1

fcollonval commented 3 years ago

1 Sept 2021 Monthly meeting

Attendees

Name	affiliation	username
Frederic Collonval	QuantStack	@fcollonval
Gonzalo Peña-C	Quansight	@goanpeca
Eric Charles	Quansight	@echarles
Pete Blois	Google
Jason Grout	Bloomberg
William Stein	Cocalc
Kyle Kelley	nteract
Michał Krassowski	Oxford
Marc Udoff	D. E. Shaw & Co.
Andrew

Notes

Frederic:
- Adding benchmark test in JupyterLab CI #10936
- Public Gitter chat: https://gitter.im/jupyterlab/Performance
- Next steps:
  - Looking at layout call triggered when switching tabs
  - Prototyping single codemirror editor
- Discussion:
  - What about hidden panels? Should we detach them? In VS Code, only the current tab node is in the DOM tree
    - Create plenty of complexity to restore the scroll, and other state.
William: just comment that Pete Blois has a lot of experience with Monaco + Jupyter. Also, Kyle Kelly is here and he's implemented another React-based Jupyter notebook (nteract), so has experience with performance.
Kyle experience
- Noteable
- For windowing, you need to have the model seat out of the DOM.
- The notebook is not DOM and don't have codemirror editors.
- JG: Impact on opening big notebook?
  - Trade off between opening a small chunk (no finding) or all (need to wait)
Colab
- windowing using intersection observer. We save the size of cells in the notebook so we have that information for the scrolling
- Monaco editor are rendered only if in view
  - Editor are re-instantiated each time (not a big impact)
  - Text models are created for the whole document, just the views are windowed. Undo stack is stored in the text model.
  - Switch 2 years ago from codemirror to monaco (pain to load the page because of larger binary size but instantiantion was faster)
  - Monaco brought a big improvement on the UX although the UX is less customizable
- Advanced rendered output are also removed to avoid performance impact
  - Using of intersection observers
- iframe on output have a typing impact on performance on Chrome due to the layout rendering
Mike: Variable insertion in markdown vs performance: https://discourse.jupyter.org/t/inline-variable-insertion-in-markdown/10525/

fcollonval commented 2 years ago

29 Sept 2021 Monthly meeting

Attendees

Name	affiliation	github username
Frederic Collonval	QuantStack	@fcollonval
William Stein	SageMath, Inc.	@williamstein
Pete Blois	Google	@blois
Eric Charles	Datalayer	@echarles

Notes

Frederic
- Switch benchmark to relative computation to avoid hardware discrepancy #11128](https://github.com/jupyterlab/jupyterlab/pull/11128)
- Trouble with running 100 experiments without Playwright error
- Trouble to detect fake delay of 500ms
- At opening there are rooms to improve the situation by initializing better the cells
- drop of Blueprint JS
- [Marc] It may interesting to report the evolution of the tree map as for the benchmark.
- [William] Using the latest version of webpack is important and not using too many plugins.
- start experimenting using only one codemirror instance
  - Seems faster but want the benchmark up and running before going further.
  - look at having some editors in view; not just one.
    [Pete] The rendering time of the editors is fast enough for their experiments (using Monaco). Note: Monaco does not add line numbers by default.

Eric

The following are open questions/proposals TBD regarding the future of https://github.com/jupyterlab/benchmarks repository
1. Benchmarks running on CI should ideally compare 2 runs performed on the same VM and apply some more methods to ensure the difference is statistically relevant.
2. We could package and publish a npm library @jupyterlab/benchmarks that can be used in any project. This library should not depend on galata.
3. For now, the work is done in the main JupyterLab repository. some point, if we decide to move the code in the benchmarks repository, we will update the benchmarks docs, otherwise we will put the benchmarks repository to the attic.
- WIP https://github.com/jupyterlab/benchmarks/pull/72
- [Frederic] The latest experiments on JupyterLab core bring conclusion that using less tricks (aka dropping @jupyterlab/galata).
- [William] Describe better the tests run so it is easy to define the same experiment in other softwares, and people reading about the benchmarks better understand what they actually are.
- [Marc] It will be interesting to be able to download the tests notebooks to share easily with the other tools.
Update on work done at Datalayer for a React.js Notebook
- ✅ Step 1: Wrap Lumino in React.js (and connect Signals to Redux Observable store) https://blog.datalayer.io/2021/08/20/datalayer-0.0.6-a-more-reactjs-jupyter
```
Host application
-> Redux Observable Actions
-> Jupyter React
-> JupyterLab Commands
Host application
<- Redux State
<- Redux Observable Epics
<- Jupyter React
<- JupyterLab Signals
```
🚧 Step 2: Jupyter Editor: Fully React, on top of JupyterLab Outputs, Renderers and Services.
- Intersection Observer + incremental highlighting coming for performance.
- Any interest to expose such an Editor as React component in JupyterLab as an extensions "Open Notebook with ...", based on MUI. In line with: while removing Blueprint.js https://github.com/jupyterlab/jupyterlab/pull/11173, Material-UI is considered for the settings UI https://github.com/jupyterlab/jupyterlab/pull/11079
- [Pete] Experimenting with the Intersection Observer and using a placeholder containing the text content so that search on the page still work. But there are noise from the span for example. (ws: Basic Pete emphasizes that some users really, really value the unique power of the browser find, which is technically nearly impossible to fully replicate from Javascript.)

fcollonval commented 2 years ago

27 Oct 2021 Monthly meeting

Attendees

Name	affiliation	github username
Frederic Collonval	QuantStack	@fcollonval
William Stein	SageMath	@sagemath

Notes

Frederic
- Benchmark on JupyterLab core is now done in the same job to avoid hardware variation
Pete: demo new colab virtualization which is live and subtle.
- Notebook example (500 cells): Rendering time 25s -> 5s
- Use Intersectional observer
- Some blink due to the syntax highlighting
- Editors instantiated are for the one displayed
- Need to have a careful height of placeholder matching the one of the editors to avoid glitches
Iframe is very good security models but if you can live without it, it eases a lot the development code - especially when dealing with widgets.
Public API for the output iframes in Colab: https://github.com/googlecolab/colabtools/blob/main/packages/outputframe/lib/index.d.ts

fcollonval commented 2 years ago

This is friendly reminder that the next meeting will be held December 22nd.

The major achievement of November is the proof of concept of a windowed notebook: https://github.com/jupyterlab/jupyterlab/pull/11574 There is a huge potential but it requires important rework of some API as extensions (WIP).

fcollonval commented 2 years ago

22 Dec 2021 Monthly meeting

Attendees

Name	affiliation	github username
Frederic Collonval	QuantStack	@fcollonval
Eric Charles	Datalayer	@echarles
Michał Krassowski	Oxford	@krassowski
William Stein	CoCalc/SageMath	@williamstein
Paul Ivanov	Noteable	@ivanov
Mark Udoff
Gabriel Fouasnon	QuanSight

Notes

Mike
- hitTest call in Lumino menu mouse move
  https://github.com/jupyterlab/jupyterlab/issues/9757#issuecomment-997041811
  It seems that there are lots of update request that trigger widgets content that could be related
  https://github.com/jupyterlab/jupyterlab/issues/9383 Seen also on the status bar (title is hidden directly due to update triggered)
- benchmarking changes to lumino https://github.com/jupyterlab/benchmarks/issues/84
- new test cases: https://github.com/jupyterlab/benchmarks/issues/84
Frederic
- jupyterlab/benchmarks reboot for advanced benchmark tests on GitHub CI
  Jupyter Notebooks use for the CI tests are available at https://cocalc.com/wstein/support/benchmarks
- PoC Virtual notebook
William
- maybe worth benchmarking various publishing situations, e.g., github (difficult because it caches the rendered html), https://nbviewer.org/, and here's what cocalc does -- https://cocalc.com/wstein/support/benchmarks

fcollonval commented 2 years ago

The notes for 2022 will be in that issue

jupyterlab / benchmarks