Meeting Notes 2023 - Githubissues

blink1073 commented 1 year ago

Hello everyone,

Welcome to the Jupyter Server Team!

You can find us in our Gitter chat room or visit us at our weekly meetings!

We meet on Thursdays at 8:00am, Pacific Standard Time on Jupyter's Zoom Channel.You can add yourself to the weekly agenda here. Everyone is welcome!

Let's avoid using this thread for discussion. If you'd like to discuss something in the minutes, open a separate issue and reference this thread.

You can find previous year's notes here: 2020, 2021, 2022

Meeting Notes

Dec 28th, 2023 — No meeting (winter holidays).
Dec 21st, 2023
Dec 14th, 2023
Dec 7th. 2023
Nov 30th, 2023
Nov 23rd, 2023 — No meeting (Thanksgiving holiday in the US)
Nov 16th, 2023
Nov 9th, 2023
Nov 2nd, 2023
Oct 26th, 2023
Oct 19th, 2023
Oct 12th, 2023
Oct 5th, 2023
Sep 28th, 2023
Sep 21st, 2023
Sep 14th, 2023
Sep 7th, 2023
Aug 31st, 2023
Aug 24th, 2023
Aug 17th, 2023
Aug 10th, 2023
Aug 3rd, 2023
July 27th, 2023
July 20th, 2023
July 13th, 2023
July 6th, 2023
June 29th, 2023—no meeting due to low attendance.
June 22nd, 2023
June 15th, 2023
June 8th, 2023
June 1st, 2023
May 25th, 2023
May 18th, 2023
May 11th, 2023 (Cancelled due to JupyterCon)
May 4th, 2023
Apr 27th, 2023
Apr 20th, 2023
Apr 13th, 2023
Apr 6th, 2023
Mar 30th, 2023
Mar 23rd, 2023
Mar 16th, 2023
Mar 9th, 2023
Mar 2nd, 2023
Feb 23rd, 2023
Feb 16th, 2023
Feb 9th, 2023
Feb 2nd, 2023
Jan 26th, 2023
Jan 19th, 2023
Jan 12th, 2023
Jan 5th, 2023

blink1073 commented 1 year ago

January 5th, 2023

Name	affiliation	username
Vidar Fauske	JP Morgan Chase	@vidartf
Steven Silvester	MongoDB	@blink1073
David Brochart	QuantStack	@davidbrochart
Andrii Ieroshenko	AWS	Wandrii-i

Vidar
- JupyterLab 3.6 CI fails with jupyter_server 2.0 https://github.com/jupyterlab/jupyterlab/issues/13563

We also did some triaging of open "bug" issues

blink1073 commented 1 year ago

January 12th, 2023

Name	affiliation	username
Zach Sailer	Apple	Zsailer	-----------------
Vidar Fauske	JP Morgan Chase	@vidartf
Steven Silvester	MongoDB	@blink1073
David Brochart	QuantStack	@davidbrochart
Andrii Ieroshenko	AWS	@andrii-i
Steven Silvester	MongoDB	@blink1073
Afshin T. Darian	QuantStack	@afshin
Kevin Bates	IBM	@kevin-bates
Johan Mabille	QuantStack	@JohanMabille

Agenda

Darian
- Executive Council + SSC update
Steve
- New minor release today? Adding JPY_SESSION_NAME
  - Will do this later today or next Monday
- API docs PR - would appreciate a review to make sure I didn't accidentally break something
- Cleaning up typings in jupyter_events.
- Warning emitted by pyzmq 25: "zmq.eventloop.ioloop is deprecated in pyzmq 17" - https://github.com/jupyter/jupyter_client/issues/913
Vidar
- I expect this fails for synchronous contents managers, but I wasn't able to reproduce it locally. Has the jupyter_core code been updated to handle it? Do we need to bump lower version bound if so? https://github.com/jupyter-server/jupyter_server/blob/5cce2afcbeeb44581e9b29ab27fef75a12d651ca/jupyter_server/services/contents/manager.py#L82 -- https://github.com/jupyter/jupyter_core/pull/330
- 1.x patch / minor release?
  - https://github.com/jupyter-server/jupyter_server/releases/tag/v1.23.5 released today,
  - 2.1 release planned today or next Monday

kevin-bates commented 1 year ago

January 19th, 2023

Name	affiliation	username
Kevin Bates	IBM	@kevin-bates
Steven Silvester	MongoDB	@blink1073
Johan Mabille	QuantStack	@JohanMabille
David Qiu	AWS	@dlqqq
Andrii Ieroshenko	AWS	@andrii-i
Piyush Jain	AWS	@3coins
A. T. Darian	QuantStack	@afshin
Vidar T Fauske	JP Morgan Chase	@vidartf

Agenda

Steve
- Only load enabled extension packages PR merged
- Bug in preferred_dir affecting JupyterLab - https://github.com/jupyter-server/jupyter_server/issues/1181
Kevin
- Custom entries in kernelspec model (resources stanza)
  - Will recommend application customizations be placed into the free-form metadata stanza that is located in the spec stanza.
David Q
- Async procedures in initialize_settings()?
  - run_sync utility function in jupyter_core
  - serverapp.io_loop.add_callback()
  - Future directions: make initialize_settings() and initialize_handlers() async and have a running event loop beforehands
Johan
- JEP for optional features - https://github.com/jupyter/enhancement-proposals/pull/92
Vidar
- Any note on how to generically support both jupyter_server 2 and notebook 6 as server extension? E.g. what is the best way to detect if we are being imported/loaded as a server 2.0 extension?
- https://github.com/jupyterlab/jupyterlab/pull/13716

kevin-bates commented 1 year ago

January 26th, 2023

Name	affiliation	username
Johan Mabille	QuantStack	@JohanMabille
Kevin Bates	IBM	@kevin-bates
Piyush Jain	AWS	@3coins
Steven Silvester	MongoDB	@blink1073
Andrii Ieroshenko	AWS	@andrii-i
Vidar T Fauske	JP Morgan Chase	@vidartf
A. T. Darian	QuantStack	@afshin
Zach Sailer	Apple	Zsailer
Omar Jarjur	Google	@ojarjur

Agenda

Kevin:
- Trailets question: Is it possible to configure traits on multiple instances of a class with different values for the various traits on each instance?
  - Not natively as they are class based. Some form of custom loader/configurator would be necessary.
Piyush:
- Bug bounty program: https://github.com/jupyterlab/team-compass/issues/174
  - Should jupyter-server be included in this? Or, is jupyterlab testing going to cover that?
  - If yes, how should we track this?
Steve
- Going to release Jupyter Client 8 today. Should be no major user-facing changes, mostly internal cleanup around async handling.
Vidar
- If someone wants to take over this partial PR, I'd appreciate it: https://github.com/ipython/ipykernel/pull/1047
  - Johan mentioned he'd take a look (thanks Johan!)

Zsailer commented 1 year ago

February 2nd, 2023

Name	affiliation	username
Zach Sailer	Apple	Zsailer
Kevin Bates	IBM	@kevin-bates
Omar Jarjur	Google	@ojarjur
Steven Silvester	MongoDB	@blink1073
Piyush Jain	AWS	@3coins
Vidar T Fauske	JP Morgan Chase	@vidartf
Andrii Ieroshenko	AWS	@andrii-i
Johan Mabille	QuantStack	@JohanMabille
Jessica Xu	Cal Poly	@jess-x

Agenda

Kevin:
- New repo for gateway-provisioners
  - Zach will call a vote via the Server Council to determine if it will be moved.
Zach
- Security Bug Bounty Program
  - Decision by February 5th (Sunday)
  - What repositories should we have them review?
    - jupyter_server
    - enterprise_gateway
      - package_name: jupyter-enterprise-gateway
      - installation instructions
      - documentation
      - how to report issues: security@ipython.org or kbates4@gmail.com
    - jupyverse
    - jupyter_server_ydoc
    - ...?
    - Follow up with the Jupyter Standards group: jupyter-standards-council@googlegroups.com regarding inclusion of jupyter_client
  - Two things needed to proceed:
    - a representative to be the main point person to correspond with the bug bounty program.
    - a group of team members willing+able to review, verify, and respond to any reported bugs.
      - Kevin Bates, Jessica Xu, Andrii Ieroshenko
    - Piyush: Putting together a doc here
Steve
- 2.2 released with "Only load enabled extension packages"
- Adopted more linting rules
- Client 8 released - no real impact on server, had to fix a bug that affected papermill.

Zsailer commented 1 year ago

February 9th, 2023

Name	affiliation	username
Zach Sailer	Apple	Zsailer
Kevin Bates	IBM	@kevin-bates
Omar Jarjur	Google	@ojarjur
Andrii Ieroshenko	AWS	@andrii-i
Frederic Collonval	QuantStack	@fcollonval
Rahul Goyal	AWS	@rahul26goyal
Johan Mabille	QuantStack	@JohanMabille
Piyush Jain	AWS	@3coins

Agenda

Frederic:
- jupyter-server v2 / notebook v6 issue
  It looks like it brings trouble even outside of the Binder usage.
- Jupyter Server v2 on Binder: currently it requires two tricks
  - Use a start script to not use jupyter notebook as start application
  - Force upgrading Python to be Python 3.8 or above; the default image is using Python 3.7 that is no more supported since jupyter-server 2.0.0
  A gist to demonstrate a setup: https://gist.github.com/fcollonval/8a761e045dbf12f053f9c6bf1166cba8 On Gitter, Jeremy explain how to use conda to bump Python: https://gitter.im/jupyterlab/jupyterlab?at=63e3adb8f4ead81aa367dd0f
Darian: JVM kernels
Johan: new C++ kernel https://github.com/compiler-research/xeus-clang-repl
Zach: Add a post_start method to ExtensionApps
- Does anyone see an issue with adding such a method?
  - Nothing raised during the meeting.
- https://github.com/jupyter-server/jupyter_server/issues/1203

Zsailer commented 1 year ago

February 16th, 2023

Name	affiliation	username
Zach Sailer	Apple	@Zsailer
Frederic Collonval	QuantStack	@fcollonval
Kevin Bates	IBM	@kevin-bates
Omar Jarjur	Google	@ojarjur

Agenda

Frederic
- I want to point out the JupyterLab PR Upgrade to MathJax 3 - this will drop the usage of server config mathjax_config that for example is leverage by jupyter_server_mathjax. Is it a good or a bad change to drop that server configuration? Feel free to comment on the JupyterLab PR.
  
  We can move on with the PR. But as follow-up, JupyterLab team needs to release the older MathJax 2 extension independently (preferably in the jupyter-renderers). This is similar to vega management we have done.
Zach
- JS council: remember to vote on the gateway-provisioners proposal
- 2.3.0 Released this past week with IPv6 support
Jupyter Enhancement Proposals to begin reviewing:

ellisonbg commented 1 year ago

On the MathJax 2 extension. Is there a reason or signal why we need to support MathJax 2? MathJax 3 has been out for over 3 years, so it isn't clear that this is needed. Thoughts?

ellisonbg commented 1 year ago

ping @fcollonval on the MathJax 2 question.

dlqqq commented 1 year ago

^ On this note, the MathJax 2 extension will still be preserved in the JupyterLab 3.x branch, so it's not clear why it would need to be migrated to a separate repo; IMO it should only be maintained for as long as 3.x is maintained, so keeping it living in that branch seems sufficient.

krassowski commented 1 year ago

One of the points in discussion previously was that MathJax 3 does not support line breaking and we had a conversation about whether we should wait for it MathJax 4 which will support it here: https://github.com/jupyterlab/jupyterlab/issues/7218#issuecomment-1405377655.

dlqqq commented 1 year ago

Well, because my PR drops the version number from the package name, we aren't tied to using MathJax 3 in the package; the only commitment is that we're using any version of MathJax that we believe is best for users.

dlqqq commented 1 year ago

Oh, is the lack of linebreaking a regression between MJ2 and MJ3?

krassowski commented 1 year ago

is the lack of linebreaking a regression between MJ2 and MJ3?

Yes.

Zsailer commented 1 year ago

On the MathJax 2 extension. Is there a reason or signal why we need to support MathJax 2? MathJax 3 has been out for over 3 years, so it isn't clear that this is needed. Thoughts?

The main issue we discussed in the server meeting was that there are server-side traits (found in the jupyter_server_mathjax server extension) that folks might be using. Before eliminating a trait from the server (even though this has moved to a server extension), we try to have a proper deprecation warning period to give folks a heads up about a config change.

Until these traits are properly deprecated, we likely need to ensure that the MJ2 lab plugin is available as a fallback.

It doesn't really matter to me where the MJ2 lab plugin codebase lives, as long as we can keep it around long enough to properly deprecate the server-side traits.

ellisonbg commented 1 year ago

Thanks for the context: follow up question. Are the server-side traits part of Jupyter Server or JupyterLab? IOW, which release cycle do we need to follow to deprecate these?

On Fri, Feb 17, 2023 at 10:08 AM Zachary Sailer @.***> wrote:

On the MathJax 2 extension. Is there a reason or signal why we need to support MathJax 2? MathJax 3 has been out for over 3 years, so it isn't clear that this is needed. Thoughts?

The main issue we discussed in the server meeting was that there are server-side traits (found in the jupyter_server_mathjax server extension) that folks might be using. Before eliminating a trait from the server (even though this has moved to a server extension), we try to have a proper deprecation warning period to give folks a heads up about a config change.

Until these traits are properly deprecated, we likely need to ensure that the MJ2 lab plugin is available as a fallback.

It doesn't really matter to me where the MJ2 lab plugin codebase lives, as long as we can keep it around long enough to properly deprecate the server-side traits.

— Reply to this email directly, view it on GitHub https://github.com/jupyter-server/team-compass/issues/45#issuecomment-1435052654, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAGXUGNTUJNKWLFWXJYOULWX65CFANCNFSM6AAAAAATSFMAAY . You are receiving this because you commented.Message ID: @.***>

-- Brian E. Granger

Senior Principal Technologist, AWS AI/ML @.***) On Leave - Professor of Physics and Data Science, Cal Poly @ellisonbg on GitHub

Zsailer commented 1 year ago

You're welcome! Thanks for following up here on the discussion.

They live in jupyter_server_mathjax, a server extension under the Jupyter Server org. The traits are very minimal—here is the main chunk in the server extension.

ellisonbg commented 1 year ago

Thanks! So maybe add a deprecation warning to those now and remove them in the Juptyer Server 3.x release cycle?

On Fri, Feb 17, 2023 at 11:00 AM Zachary Sailer @.***> wrote:

You're welcome! Thanks for following up here on the discussion.

They live in jupyter_server_mathjax https://github.com/jupyter-server/jupyter_server_mathjax, a server extension under the Jupyter Server org. The traits are very minimal—here is the main chunk in the server extension https://github.com/jupyter-server/jupyter_server_mathjax/blob/main/jupyter_server_mathjax/app.py#L34-L47 .

— Reply to this email directly, view it on GitHub https://github.com/jupyter-server/team-compass/issues/45#issuecomment-1435105241, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAGXUDTOJFW6TCBBX5IHXLWX7DEZANCNFSM6AAAAAATSFMAAY . You are receiving this because you were mentioned.Message ID: @.***>

-- Brian E. Granger

Senior Principal Technologist, AWS AI/ML @.***) On Leave - Professor of Physics and Data Science, Cal Poly @ellisonbg on GitHub

Zsailer commented 1 year ago

February 23th, 2023

Name	affiliation	username
Zach Sailer	Apple	@Zsailer
Johan Mabille	QuantStack	@JohanMabille
Sylvain Corlay	QuantStack	@SylvainCorlay
Andrii Ieroshenko	AWS	@andrii-i
Kevin Bates	IBM	@kevin-bates
Piyush Jain	AWS	@3coins
David Brochart	QuantStack	@davidbrochart
Nick Bollweg	Georgia Tech	@nrbgt @bollwyvl

Agenda

Johan

Kernel protocol roadmap
Begin utilizing the jupyter_kernel_test to ensure kernels follow the protocol: https://github.com/jupyter/jupyter_kernel_test

Sylvain

Discussing parameterized kernels JEP.
- Lots of discussion around parameterizing the environment around the kernel (via provisioners) vs parameterizing the kernels themselves.
- Much of this discussion is on the JEP already
- Kevin, Sylvain, Johan and Zach planning to set up a brainstorming meeting.
  - Reach out if interested in joining.

Zach

Mix authentication logic in "modern" server extensions running on classic notebook server
https://github.com/jupyter-server/jupyter_server/pull/1221

Nick

random perf experiment

Zsailer commented 1 year ago

March 2nd, 2023

Name	affiliation	username
Zach Sailer	Apple	@Zsailer
Johan Mabille	QuantStack	@JohanMabille
Andrii Ieroshenko	AWS	@andrii-i
Kevin Bates	IBM	@kevin-bates
Steve Silvester	MongoDB	@blink1073
David Qiu	AWS	@dlqqq
Afshin T. Darian	QuantStack	@afshin
Piyush Jain	AWS	@3coins

Agenda

Vidar
- https://github.com/jupyter-server/jupyter_server/issues/1224
Zach
- Discussion about identity flow in services
  - Use case:
    - custom managers that "talk" to external services, e.g. contents manager talking to remote contents provider.
    - the managers act as "clients" to those external services.
    - they usually need the (Jupyter Server) authenticated user to authenticate with the external service.
  - Today, Jupyter Server does provide API for flowing the user through the services.
  - Proposal: pass the request handler, user model, or some sort of request context object through all the managers.
  - This is a Jupyter Server specific addition, i.e. not jupyter_client level.
- Kernelspec endpoint isn't async.
  - Do we have async kernelspec manager?
    - Make this line async https://github.com/jupyter-server/jupyter_server/blob/1dc3d58adf94428a3a4a4c5e0b200bc90dcb0ccd/jupyter_server/kernelspecs/handlers.py#L24
  - Make this line await ensure_async(...) https://github.com/jupyter-server/jupyter_server/blob/1dc3d58adf94428a3a4a4c5e0b200bc90dcb0ccd/jupyter_server/kernelspecs/handlers.py#L30
- Provide an async kernelspec manager in Jupyter Server.
  - Use a cache like in enterprise gateway
David Q.
- Two issues I noticed in recent release(s?) of Jupyter Server:
- SIGINT (Ctrl-C) fails: https://github.com/jupyter-server/jupyter_server/issues/668
- Port remains occupied after killing process: https://github.com/jupyter-server/jupyter_server/issues/1170
- Anything needed on my end for File ID?
- Think next step is to just move to aiosqlite; handling multiple processes and supporting NFS should not be mutually exclusive!
- Need to do a deep dive into aiosqlite; presumably transactions are submitted to an in-memory message queue that gets polled in serial

Zsailer commented 1 year ago

March 9th, 2023

Name	affiliation	username
Zach Sailer	Apple	@Zsailer
Andrii Ieroshenko	AWS	@andrii-i
Steven Silvester	MongoDB	@blink1073
Omar Jarjur	Google	@ojarjur
Johan Mabille	QuantStack	@JohanMabille
A. T. Darian	QuantStack	@afshin
Kevin Bates	IBM	@kevin-bates
David Brochart	QuantStack	@davidbrochart
Marc Udoff	D. E. Shaw	@mlucool
Jessica Xu	Cal Poly	@jess-x

Agenda

Zach
- Websocket instability
  - The kernel websocket appears to be instable for long connections.
  - Scenario 1
    1. User switches browser tabs or windows.
    2. JupyterLab's tab goes to sleep.
    3. Jupyter Server's websocket ping never receives a response
    4. Jupyter Server closes the server-side websocket handler
  - Scenario 2
    1. Increase the Jupyter Server websocket time to avoid closing the websocket handler.
    2. User switches browser tabs or windows.
    3. JupyterLab's tab goes to sleep.
    4. Browser sees an open, long-time, idle connection and terminals websocket.
  - Ideas to keep this from happening?
    - Websocket heartbeats both directions
      - How do we prevent the browser from terminating websocket connections?
    - Store offline changes to any documents server-side and forward to the front as soon as things reconnect.
      - Can we use y-docs for this?
      - How does it handle kernel status.
Marc
- Jupyter PageLoad https://github.com/jupyterlab/jupyterlab_server/issues/380
- json5 vs yaml https://github.com/jupyterlab/jupyterlab_server/pull/378
Steve
- Fixed the port selection bug that David mentioned last week

blink1073 commented 1 year ago

March 16th, 2023

Name	affiliation	username
David Qiu	AWS	@dlqqq
Steven Silvester	MongoDB	@blink1073
Vidar T Fauske	JP Morgan Chase	@vidartf
Omar Jarjur	Google	@ojarjur
Kevin Bates	IBM	@kevin-bates
A. T. Darian	QuantStack	@afshin
Carlos Herrero	QuantStack	@hbcarlos
Andrii Ieroshenko	AWS	@andrii-i
David Brochart	QuantStack	@davidbrochart

Agenda

David Q.
- Guidance on how to call Jupyter Server via an IPython magic
- Managed to get a fairly hacky solution working:
  - https://github.com/jupyterlab/jupyter-ai/pull/18/files
Steve
- Enable KernelSpecResourceHandler to be async
- Added error propagation to gateway_request function
- Cut a minor release during the meeting - https://github.com/jupyter-server/jupyter_server/releases/tag/v2.5.0
- ruff updates - they are adding a formatter that could replace black, and are considering adding some typing support, similar to how there is a TS plugin for ESLint.
Carlos
- Allow for registering new file loaders to read and write from disk different file types in jupyter_collaboration
- https://github.com/jupyterlab/jupyter_collaboration/pull/121

Zsailer commented 1 year ago

March 23rd, 2023

Name	affiliation	username
Zach Sailer	Apple	Zsailer
Steven Silvester	MongoDB	@blink1073
Kevin Bates	IBM	@kevin-bates
Johan Mabille	QuantStack	@JohanMabille
Andrii Ieroshenko	AWS	@andrii-i
Piyush Jain	AWS	@3coins
Afshin T. Darian	QuantStack	@afshin
Carlos Herrero	QuantStack	@hbcarlos
Omar Jarjur	Google	@ojarjur

Agenda

Setup Request context on every HTTP request
- Help with multi-tenancy
- Each manager could get access to current context to find current user
  - https://docs.python.org/3/library/contextvars.html#asyncio-support
https://github.com/jupyter-server/jupyter_server/pull/1241
Currently, the event system is not routing events based on the user requesting the event stream (via WebSocket, etc.), we need a way to start sending events only to the user who is expecting it
- One tool we now have is asyncio support for the standard library contextvars

kevin-bates commented 1 year ago

March 30th, 2023

Name	affiliation	username
Zach Sailer	Apple	@Zsailer
Afshin T. Darian	QuantStack	@afshin
Steven Silvester	MongoDB	@blink1073
David Brochart	QuantStack	@davidbrochart
Kevin Bates	IBM	@kevin-bates
Carlos Herrero	QuantStack	@hbcarlos
Omar Jarjur	Google	@ojarjur
Garland Zhang	Databricks	@garlandz-db
Andrii Ieroshenko	AWS	@andrii-i

Agenda

Carlos

precedence between jupyter_server_config.py and jupyter_server_config.json?

Zach

Flow authenticated user model to managers in each request
- https://github.com/jupyter-server/jupyter_server/issues/1250
- PR: Define a CURRENT_JUPYTER_HANDLER context var: https://github.com/jupyter-server/jupyter_server/pull/1251

Garland

https://github.com/ipython/ipykernel/issues/859

Carlos

Anonymous users https://github.com/jupyter-server/jupyter_server/issues/1245

Zsailer commented 1 year ago

April 6th, 2023

Name	affiliation	username
Zach Sailer	Apple	@Zsailer
Kevin Bates	IBM?	@kevin-bates
Carlos Herrero	QuantStack	hbcarlos
Andrii Ieroshenko	AWS	@andrii-i
A. T. Darian	QuantStack	@afshin
Omar Jarjur	Google	@ojarjur
Garland Zhang	Databricks	@garlandz-db
Johan Mabille	QuantStack	@JohanMabille

Agenda

Kevin

Kernel Manager event schema
- https://github.com/jupyter-server/jupyter_server/pull/1252
- A group is meeting synchronously today to iterate on this PR.
- The goal is to push these events down into the individual kernel managers

Carlos

Anonymous users stored in cookies
- https://github.com/jupyter-server/jupyter_server/issues/1245
- Question: do we move the (new) anonymous users logic into an extension?
  - Carlos is going to work on a PR as POC

Garland

Thoughts on ipykernel issue
- https://github.com/ipython/ipykernel/issues/859
- Pinged a few folks on this issue to get some reviews.
- Garland is happy to work on a solution once we choose how to solve it.
Zach
- Contributing Hour is cancelled this month in favor of using this timeslot to prepare the JupyterLab 4.0 / Notebook 7 release.

blink1073 commented 1 year ago

April 13th, 2023

Name	affiliation	username
Johan Mabille	QuantStack	@JohanMabille
Carlos Herrero	QuantStack	@hbcarlos
Steven Silvester	MongoDB	@blink1073
Eric Charles	Datalayer	@echarles
Kevin Bates	IBM(53)	@kevin-bates

Agenda

Codecov saga
- tldr; they pulled all of their PyPI versions and the package got locked down
- https://github.com/jupyterlab/maintainer-tools/pull/189
- https://github.com/jupyter-server/jupyter_server/pull/1257
- JupyterLab Server depends on codecov in its tests extra
  - https://github.com/jupyterlab/jupyterlab_server/pull/392
  - Affects notebook and JupyterLab tests

Kevin

Kernel Manager event schema
- https://github.com/jupyter-server/jupyter_server/pull/1252
- Should be merged shortly - thank you @rajmusuku and @Zsailer!

Eric

Impact of persist userid cookie when auth is disabled ? e.g. New anon users generated ad infinitum when using IdentityProvider token
- Is this documented on https://jupyter-server.readthedocs.io/en/latest/operators/security.html?

Zsailer commented 1 year ago

April 20th, 2023

Name	affiliation	username
Zach Sailer	Apple	@Zsailer
Vidar T Fauske	JP Morgan Chase	@vidartf
Kevin Bates	IBM (46)	@kevin-bates
Andrii Ieroshenko	AWS	@andrii-i
Omar Jarjur	Google	@google
Johan Mabille	QuantStack	@JohanMabille
Afshin T. Darian	QuantStack	@afshin
David Brochart	QuantStack	@davidbrochart

Agenda

Andrii: looking how to query the list of installed extensions to see if nbclassic is found?
- Get list of discovered extensions from the extension manager: https://github.com/jupyter-server/jupyter_server/blob/5c49253359865773439bdba69352e26a473b9e07/jupyter_server/extension/manager.py#L254-L263
- Here's the public API in the main serverapp:
```
serverapp.extension_manager
```
Zach
- Kernelspec schema JEP
  - Kernel Parameterization
- Open a schemas.jupyter.org JEP
Omar
- Best practice way for declaring something deprecated?
- Merge the gateway handlers into the standard handlers

Zsailer commented 1 year ago

April 27th, 2023

Name	affiliation	username
Zach Sailer	Apple	@Zsailer
Kevin Bates	IBM (39)	@kevin-bates
Frederic Collonval	QuantStack	@fcollonval
Andrii Ieroshenko	AWS	@andrii-i
Piyush Jain	AWS	@3coins
Omar Jarjur	Google	@ojarjur
Afshin T. Darian	QuantStack	@afshin
David Brochart	QuantStack	@davidbrochart

Agenda

Zach
- Open JEP for creating a subdomain under jupyter.org (schema.jupyter.org) to publish schemas:
  - https://github.com/jupyter/enhancement-proposals/pull/108
  - Once this is determined, we will want to update the $id fields in our event schemas
    - Follow up with the file_id_extension needs updates for the new URI.
  - We will also want to publish the OpenAPI spec there.
Frederic
- JupyterLab 4.0.0rc0 is building right now :wink:
- Cache enhancement using immutable - looking for additional opinions
  After discussion: the path forward is to upstream the idea by adding a traitlet immutable_paths to explicitly list assets paths that should get the immutable cache policy.
Steve
- PyPI organizations - https://pypi.org/org/jupyter/
- Trusted Publishers on PyPI (OIDC tokens)
- AWS Sponsoring PyPI security role
- There will be a second developer-in-residence

kevin-bates commented 1 year ago

May 4th, 2023

Name	affiliation	username
Zach Sailer	Apple	@Zsailer
Kevin Bates	IBM (32)	@kevin-bates
Afshin T. Darian	QuantStack	@afshin
Steve Silvester	MongoDB	@blink1073
Andrii Ieroshenko	AWS	@andrii-i
David Brochart	QuantStack	@davidbrochart

Agenda

Zach
- Jupyter Bug Bounty Program
- JupyterCon planning
  - No Zoom meeting next week.
    - Zach will remove from calendar
  - Who is going to Paris?
    - Darian and David.
  - Looking for a volunteer to represent Jupyter Server at the SSC meeting.
    - David will represent Jupyter Server at the leadership meeting.
  - Massive "thank you" to David for taking on the JupyterCon talk! :tada:
Andrii
- Detecting nbclassic dynamically: would writing extension_manager info to pageconfig so that it can be then consumed by front-end (notebook) is a good option?

page_config = self.serverapp.web_app.settings.setdefault("page_config_data", {})
# Add your custom page config here
...
self.serverapp.web_app.settings["page_config_data"] = page_config

Darian
- Revisit of environment setup bug for Apple Silicon (specifically M2 and M2 Pro): https://github.com/jupyterlab/jupyterlab/issues/13938
  - These instructions work for building JupyterLab and even building test packages jlpm build:test but for me, they cause a failure at runtime for tests jlpm test
- Currently working around it with ugly script that symlinks Homebrew-installed libraries in /usr/local/lib
Steve
- Security Updates
  - Jupyter PyPI Trusted Publishers
  - Jupyter PyPI Organization
- Jupyter Releaser—if folks have time, please review the following issues:

Zsailer commented 1 year ago

May 18th, 2023

Name	affiliation	username
Kevin Bates	IBM (18)	@kevin-bates
Zach Sailer	Apple	Zsailer
Andrii Ieroshenko	AWS	@andrii-i
Omar Jarjur	Google	@ojarjur

Agenda

JupyterCon Recap
- SSC and EC met and gave project updates
- Lots of great talks; shorter talk time slots was a good move.
- David's talk went really well! :tada: (THANK YOU, DAVID!)
Kevin
- KernelSpec Caching
  - Ready for review
  - Should we implement a watchfiles monitor now and replace watchdog-monitor, or let things roll a bit?
Andrii
- Update on NbClassic dynamic detection work: moved NbClassic detection logic to notebook/app.py https://github.com/jupyter/notebook/pull/6847
- Question about JupyterLab notifications: is there a plan to add server-side message storage, identity, DMs, etc?

Zsailer commented 1 year ago

May 25th, 2023

Name	affiliation	username
Zach Sailer	Apple	@Zsailer
Kevin Bates	IBM	@kevin-bates
Steven Silvester	MongoDB	@blink1073
Andrii Ieroshenko	AWS	@andrii-i
Sergey Kukhtichev	IBM	@skukhtichev
Afshin T. Darian	QuantStack	@afshin
Piyush Jain	AWS	@3coins
David Leen	Netflix	@dleen
Parul Gupta	Meta	@parul100495
Omar Jarjur	Google	@ojarjur
Johan Mabille	QuantStack	@JohanMabille

Sergey
- Proposal: Restore Notebook execution progress when a browser page is reload
- Zach
  - Another case that might break things—network latency can cause a missed message that is not caught by the message buffer.
- Darian
  - This is a major pain-point, so any improvement is a win.
  - In the long run, I think we should be solving this with RTC, but asking to wait for RTC to solve this problem might be unreasonable (it might still be far off)
  - Right now, though, it might be good to pursue a simple solution as a starting point.
- Parul
  - Similar approach is not to store the notebook state, but a log (file) of all messages that were missed during disconnection
  - Once reconnect happens, push all missed messages forward.
  - To solve the issue of missed messages due to latency, replay all messages since the last user execute request.
- Vidar
  - Comment: From previous discussion, it was mentioned that session ID should be random to avoid collisions? I.e. impossible to distinguish refresh vs new tab open?
  - Another possible case that breaks things. Run 100 cells (creating a queue) and refresh the page.
  - There is not enough information in the messages to resolve how many cells need to run or which cell is currently executing.
- Omar
  - Kernel Gateway case. You can have server/client missed messages. In the gateway case, you can also have missed messages between kernel and server.
- Kevin
  - By storing only the messages (Parul's experience) we can avoid a dependency on the content service. This allows things like a gateway server to also participate.
  - We should consider abstracting the message storage layer to accomodate a distributed cache approach which, coupled with kernel connection-info persistence, can enable the ability to introduce multiple servers (HA/DR).
Andrii
- Looking to add a server extension to https://github.com/jupyter/notebook/tree/main/packages/lab-extension for dynamic NbClassic detection
- Links to helpful tools and info for building server extensions:
  - https://jupyter-server.readthedocs.io/en/latest/developers/extensions.html
Steve
- LOL, tried to test against Python 3.12 beta but pip itself isn't yet compatible.
- Will try to make a minimum reproduction and report to pip

kevin-bates commented 1 year ago

June 1st, 2023

Name	affiliation	username
Matt Wiese	Cornell / Buckler Lab	@matthewwiese
Kevin Bates	IBM	@kevin-bates
A. T. Darian	QuantStack	@afshin
Zach Sailer	Apple	@Zsailer
Omar Jarjur	Google	@ojarjur
Steven Silvester	MongoDB	@blink1073
Andrii Ieroshenko	AWS	@andrii-i

Agenda

Matt Wiese - Interested in contributing to solving problem of resuming/reconnecting to longrunning cell execution (I think originally noted in jupyter/notebook#641)
- Question: How would a contributor best allocate their time to addressing this problem - assisting with Sergey's proposal, real-time collaboration (RTC), the experimental kernels API in Jupyverse, or via some other avenue?
- Question: Is a feature that satisifies this problem (however it may be implemented) expected to land in a public release this year? The reason I ask is if I ought to implement a stopgap for my team in the interim.
  - Jupyter Scheduler, a JupyterLab extension for running notebook jobs might be helpful https://github.com/jupyter-server/jupyter-scheduler

Zsailer commented 1 year ago

June 8th, 2023

Name	affiliation	username
Steven Silvester	MongoDB	@blink1073
Zach Sailer	Apple	@Zsailer
Kevin Bates		@kevin-bates
Andrii Ieroshenko	AWS	@andrii-i
Afshin T. Darian	QuantStack	@afshin
Parul Gupta	Meta	@parul100495
Johan Mabille	QuantStack	@JohanMabille
David Brochart	QuantStack	@davidbrochart
Omar Jarjur	Google	@ojarjur
Aayush Gupta	Amazon	@aayushgupta899

Agenda

Steve
- Issue opened in ipykernel that I think is actually a server issue: https://github.com/ipython/ipykernel/issues/1121. The underlying issue is that debugger messages on the control channel result in idle status messages that are interpreted by the front end as kernel state on the shell channel.
  - Zach will respond on the JupyterLab ticket and link to https://github.com/jupyter/jupyter_client/issues/838.
- Haven't had a chance to look into the Python 3.12 test weirdness
Extra discussion around fetching kernel/notebook state from the server
- Question about how to reconstruct kernel state from a client that got disconnected.
- Y-doc solution to notebook state problem:
  - List of repos:
    - shared document structures (notebook, text, etc.): https://github.com/jupyter-server/jupyter_ydoc
    - WebSocket handler: https://github.com/jupyterlab/jupyter_collaboration and https://github.com/y-crdt/ypy-websocket
    - Ypy: https://ypy.readthedocs.io/en/latest
    - Yjs: https://docs.yjs.dev/
  - https://jupyterlab-realtime-collaboration.readthedocs.io/en/latest/

Zsailer commented 1 year ago

June 15th, 2023

Name	affiliation	username
Matt Wiese	Cornell / Buckler Lab	@matthewwiese
Zach Sailer	Apple	@Zsailer
Andrii Ieroshenko	AWS	@andrii-i
Sergey Kukhtichev	IBM	@skukhtichev
Omar Jarjur	Google	@ojarjur
Steven Silvester	MongoDB	@blink1073
Carlos Herrero	QuantStack	@hbcarlos
Afshin T. Darian	QuantStack	@afshin
Johan Mabille	QuantStack	@JohanMabille
David Brochart	QuantStack	@davidbrochart

Agenda

In today's meeting, we mostly discussed: Restore Notebook Execution Progress (jupyter-server#1274)

The notes are a bit scattered, because the discussion was moving fast :sweat_smile:. Zach synthesized as much as he could in a single comment on the issue here.

Here are the notes taken during the meeting:

Omar: Kernel Message replay is orthogonal to notebook state problem.
3 approaches mentioned in previous messages
1. Provide a new API for managing the state of the notebook model on the server
2. Leverage y-docs from RTC to keep a state of the notebook model on the server
3. Create a log of kernel messages with time stamps. The kernel client can replay all message from a previous timestamp
Two layers to this the "notebook state" problem
- Kernel state
  - Execution state (busy/idle)
  - Execution count number
  - Which cell is executing (via cell-id)
  - Outputs
- Notebook Document state
  - Which cells are finished
  - Which cells are still executing
  - Where should outputs go?
New repo in Jupyter-server org for states replay
- API for storing and replaying kernel messages
- API for notebook state (as busy/idle execution state is independent of notebook cell contents)
- API for remote kernels (Gateway) for execution and retrieval of state without managing notebook state
Kernel as just another collaborator, such that if web clients are all disconnected, there will always be something to receive Ydoc updates?
Carlos mentioned that we have an issue with STDIN in y-docs. This is currently broken.
David opened an issue about encoding the "executing" state of a cell in the execution count.

Zsailer commented 1 year ago

June 22nd, 2023

Name	affiliation	username
Zach Sailer	Apple	@Zsailer
Steven Silvester	MongoDB	@blink1073
Matt Wiese	Cornell / Buckler Lab	@matthewwiese
Sergey Kukhtichev	IBM	@skukhtichev
Piyush Jain	AWS	@3coins
Parul Gupta	Meta	@parul100495
Aayush Gupta	Amazon	@aayushgupta899
Omar Jarjur	Google	@ojarjur
Andrii Ieroshenko	AWS	@andrii-i
Kevin Bates	Veritone	@kevin-bates
Andrey Velichkevich	Apple	@andreyvelich

Agenda

Broad questions about Jupyter Server and people's experiences
- Shared filesystem with remote kernels?
  - Used elastic filesystems—rate limiting that makes this difficult
  - Provide both local and remote kernels
    - Mount the volumes/folders in both the local/remote kernels.
  - Terminal access is not forwarded to remote kernels.
  - Build a custom contents manager
    - Instead of making your local files available to remote kernels, make your remote files available to Jupyter Server
  - Separate out source vs. data files
    - extend Python's import logic to handle finding remote source files
    - put data files in a remote location and write a custom contents manager to show them in JupyterLab.
  - Has anyone tried NFS?
    - In enterprise_gateway/gateway_provisioners, this is how many folks share the filesystem
    - In Kubernetes, we have templated YAMLs for setting up volume mounts.
  - The best we can hope for here is to share "patterns" of how folks are doing these things.
- Multi-tenancy in Jupyter Server?
  - Tried to provide this. Most of the issues/concerns were around privacy/security.
  - Not allow users to have access to the operating system.
[Save for end] Jupyter message replay discussion (continued)
- Original Issue
- We're planning to create a new repo under jupyter_server to coordinate effort
  - Proposed to Jupyter Server team: https://github.com/jupyter-server/team-compass/issues/52
  - Repo name:
    - jupyter_message_replay
    - kernel_playback
    - message_recovery
  - Voting happening now. We have enough to pass.
- Once this repo is in place, Zach can set up the server extension.
- Problem: Session ID coming from JupyterLab is unique for every connection
  - Proposal: Use something more deterministic when creating a session_id
    - Multiple documents/consoles connecting to the same kernel
  - Drive everything through kernel ID. One client to one kernel.

Zsailer commented 1 year ago

July 6th, 2023

Name	affiliation	username
Zach Sailer	Apple	@Zsailer
Michał (Mike) Krassowski	Quansight	@krassowski
Kevin Bates	Veritone	@kevin-bates
Matt Wiese	Cornell / Buckler Lab	@matthewwiese
Afshin T. Darian	QuantStack	@afshin
Johan Mabille	QuantStack	@JohanMabille
David Brochart	QuantStack	@davidbrochart

Agenda

Mike
- Resolve server absolute path to server or kernel-relative path
  - navigating to source code is useful for debugging and development in more complex codebase, even if the source is read-only
  - source code may be held within the ContentsManager's root_dir, or in kernels; usually a path in a traceback is accessible from a kernel, but:
    - we want to use ContentsManager to handle editing the file if possible to avoid re-implementing it for every kernel (the alternative, debugger-adapter-protocol source request can only fetch the source, but not modify the file)
    - in some case cases, like JupyterLite, a file may be known by ContentsManager but not kernel (which requires special code to access files from local storage visible to ContentsManager)
  - frontend has no way of knowing if a given file path exists in content manager's scope
  - there is a PR for JupyterLab implementing navigation from tracebacks: jupyterlab/jupyterlab#13390
Notebook server-side replay
- Vote to create repo has passed: https://github.com/jupyter-server/team-compass/issues/52
- Let's create the repo in the meeting.
- https://github.com/jupyter-server/message_replay
  - We can change the name later

Zsailer commented 1 year ago

July 13th, 2023

Name	affiliation	username
Matt Wiese	Cornell / Buckler Lab	@matthewwiese
Johan Mabille	QuantStack	@JohanMabille
Kevin Bates	Veritone	@kevin-bates
Vidar T Fauske	JP Morgan Chase	@vidartf
Zach Sailer	Apple	@Zsailer
Afshin T. Darian	QuantStack	@afshin
Michal Krassowski	Quansight	@krassowski
Omar Jarjur	Google	@ojarjur
Sylvain Corlay	QuantStack	@SylvainCorlay
Parul Gupta	Meta	@parulgupta

Agenda

Jupyter Server Server-side Notebook recovery
- Opened PR to merge extension boilerplate: https://github.com/jupyter-server/message_replay/pull/1
- Next step — work on a server-side notebook model that the client can recover.
  - We can begin to leverage prior art, i.e. Jupyter YDoc and Jupyverse.
  - We need to store some state about notebook that's not currently part of Notebook Ydoc.
    - See this issue for a list of items: https://github.com/jupyter-server/jupyter_ydoc/issues/169
  - Jupyverse has been working on a new kernels execution (REST) API. Let's start here to implement a POC.
    - Added a POST /execute endpoint to the kernels API.
    - JupyterLab could use this endpoint to execute cells (pending frontend changes in JLab).
    - The response, status, output, execution count, etc. is
      - tracked by a server-side Notebook YDoc model.
      - broadcast across Jupyter Server YDoc websocket to update Notebook documents in the client
    - If any client loses connection with the server, it can reconnect to this ydoc websocket and recover the document/kernel state.
  - This will require changes in JupyterLab (frontend) to use this new REST API.
    - Today, the above API/design isn't currently implement in JupyterLab.
    - Instead, RTC in JupyterLab still relies on the kernel websocket
      - kernel execute messages flow through the kernel websocket. Outputs/status also flow through the kernel websocket, update the notebook document in the client, forward these changes to the server to be tracked in a server-side YDoc, then broadcast back to (all) listening clients. This ensures multiple Notebooks clients are synced, but it does not solve the problem of tracking Notebook state when no frontends are connected when a response/status is returned.
    - JupyterLab should check to see if the POST /execute endpoint is present. If so, use this new approach to communicating with notebook's kernel and update the notebook document.
  - Once this is working in Jupyverse, we will go back and implement this REST API in a jupyter_server extension.
- How does the message replay / server side document state intersect with Comms?
  - Current opinion seems to be that Comms will need to be dropped within the proposed model, and any thing that depends on comms will need to use the current API.
- Summary—what we should do this week:
  1. Pull down Jupyverse, JupyterLab, jupyter_collaboration, and jupyter_ydoc and run JupyterLab in collaborative mode on top of Jupyverse.
  2. Begin working on JupyterLab frontend changes needed to talk to the new POST /execute API in Jupyverse.
Project Jupyter is in need of a host for the Jupyter Community Calls to replace Isabela. Resources: https://github.com/Quansight-Labs/jupyter-communitycalls Previous records: https://www.youtube.com/playlist?list=PLUrHeD2K9Cmkoamm4NjLmvXC4Y6E1o8SP You can find a sign-up form here: https://forms.gle/59TkLhj7niuTgLJa9
Mike:
- What is needed to port Enterprise Gateway to jupyter_client v7/jupyter-server v2?
- How should it work with Gateway Provisioners? Should it depend on it?
- Is anyone working on it or planning to work on it in the short term?
- Is this the right meeting to discuss it?
- Related issue https://github.com/jupyter-server/enterprise_gateway/issues/1208
Sylvain: python kernel
- use a promise chain instead of a queue for shell messages?
- solves the "deadlock" issue when awaiting a shell message such as a widget message.
  - check stdin.
- https://github.com/ipython/ipykernel/pull/1079

Zsailer commented 1 year ago

July 20th, 2023

Name	affiliation	username
Zach Sailer	Apple	Zsailer
Matt Wiese	Cornell / Buckler Lab	@matthewwiese
Kevin Bates	Veritone	@kevin-bates
Andrii Ieroshenko	AWS	@andrii-i
Sergey Kukhtichev	IBM	@skukhtichev
Omar Jarjur	Google	@ojarjur
Rosio Reyes	Anaconda	@RRosio

Agenda

Matt:
- Quick recap of message_replay discussion from last week
  - We discussed extending JupyterLab to support the new Jupyverse endpoints, as well as other measures
  - What should take priority for the moment?
- Action Items:
  - Investigate Notebook and Services plugins that already exist in JupyterLab repo
    - Might be able to subclass/extend these to use the experimental API that exists in Jupyverse?
  - Jupyter Collaboration repo
    - Might already have a drop-in replacement or otherwise be able to modify what they have?
  - Additional Resources:
    - Notebook execution in YDoc model issue
Kevin
- KernelProvisionerBase classes run async functions in different eventloops
- I don't think this is necessarily related to provisioners, but more related to "context" relative to kernel lifecycle
- Ideas for debugging:
  - Try to set an custom event-loop policy to debug further.
  - Check version of jupyter_client to see if nest_asyncio is a culprit.
  - Check if this is a race condition caused by the AsyncIOLoopKernelManager—which creates a loop on instantiation.
Jason
- I'd love to discuss this ipykernel PR as a follow-up to https://github.com/ipython/ipykernel/pull/1127, and more generally whether setting the zmq ports to 0 in a connection file is an officially supported way to indicate the kernel should pick the channel ports.
- Results of discussion
  - no one on the call had objections to this idea of setting ports to 0 in the connection file, then having the kernel write the actual ports to the connection file. Several people were in favor of this behavior.
  - To support this as official kernel behavior, it should be documented in the kernel spec and maybe added to the kernel test suite
  - Jason will ping Steve and the original PR author about this behavior
  - Matt helped Jason understand why the CI test was failing (thanks Matt!)

Zsailer commented 1 year ago

July 27th, 2023

Name	affiliation	username
Zach Sailer	Apple	Zsailer
Hind Montassif	QuantStack	@Hind-M
Andrii Ieroshenko	AWS	@andrii-i
Afshin T. Darian	QuantStack	@afshin
Eric Charles	Datalayer, Anaconda	@echarles
Johan Mabille	QuantStack	@JohanMabille
Matt Wiese	Cornell / Buckler Lab	@matthewwiese
David Brochart	QuantStack	@davidbrochart
Rosio Reyes	Anaconda	@RRosio
Sergey Kukhtichev	IBM	@skukhtichev
Sylvain Corlay	QuantStack	@SylvainCorlay

Agenda

Hind:
- Looking for a review of XPub JEP#65 https://github.com/jupyter-server/jupyter_server/pull/1303
Darian:
- Contributing hour time slot request from Jupyter Accessibility Subproject
- :+1:
David:
- Support for external kernels in jupyter-server and jupyter-client.
Rosio:
- PR to pin jupyter_client<8.0 in Notebook 6. More people are experiencing this issue.
  - Maybe related: https://github.com/jupyter/jupyter_client/issues/959#issuecomment-1645614689

Zsailer commented 1 year ago

August 3, 2023

Name	affiliation	username
Zach Sailer	Apple	Zsailer
Kevin Bates	Veritone	@kevin-bates
Afshin T. Darian	QuantStack	@afshin
David Brochart	QuantStack	@davidbrochart
Andrey Velichkevich	Apple	@andreyvelich
Rosio Reyes	Anaconda	@RRosio
Andrii Ieroshenko	AWS	@andrii-i

Agenda

Darian
- Asking for volunteers who are willing to put in 15 minutes to test virtual scrollbars: https://github.com/jupyterlab/jupyterlab/issues/13343#issuecomment-1662497888
Anything else needed for: https://github.com/jupyter-server/jupyter_server/pull/1305

Zsailer commented 1 year ago

August 10, 2023

Name	affiliation	username
Matt Wiese	Cornell / Buckler Lab	@matthewwiese
Zach Sailer	Apple	@Zsailer
Steven Silvester	MongoDB	@blink1073
Afshin T. Darian	QuantStack	@afshin
Andrii Ieroshenko	AWS	@andrii-i
Carlos Brandt	Constructor Uni	@chbrandt
Kevin Bates	Veritone	@kevin-bates
Rosio Reyes	Anaconda	@RRosio

Agenda

Jupyter Releaser v3 in Progress - no more long-lived tokens except npm (for now).
At end: "Message Replay" project
- Matt will likely be short on time during Autumn (class + lab projects)
  - Would like to work with other(s) on this, but somebody ought to be "lead" in terms of technical direction/management
  - Happy to help in any way I am able (code, documentation, etc)
  - jupyter-scheduler is a promising short-term solution for us (vis-a-vis long-running notebook executions) - thanks for the tip Andrii!
- Zach is back in office and will pick this up.
Pull Request Triage

Zsailer commented 1 year ago

August 17th, 2023

Name	affiliation	username
Zach Sailer	Apple	@Zsailer
Marc Udoff	D. E Shaw	@mlucool
Travis Ryan	D. E. Shaw	@travisjryan
Michal Krassowski	Quansight	@krassowski
Kevin Bates	Veritone	@kevin-bates
Andrii Ieroshenko	AWS	@andrii-i
Omar Jarjur	Google	@ojarjur
R Ely	Bloomberg	@ohrely
Rosio Reyes	Anaconda	@RRosio
Tim Guo	D. E. Shaw	@timg512372

Agenda

Pre-Proposal: Add Restart In Place API Support
- currently restart is "just preservation of kernel ID"
- relevant when multiple consoles/notebooks are attached to the same kernel
- but there is more complexity in UX actions associated (debugger, restart & re-run)
- relevance to local kernels
- proposed in-place restart is the same as current local restart
  - in the local case there is no distinction
  - in remote case there should be a choice
- but two buttons not relevant here
- why would users prefer to have a choice?
- ability to switch to a different host
- should it be just a configuration option at provisioner level?
- limiting UX options (pros & cons)
- discussion: could it be a server extension implementing hard restart (with an API endpoint) for remote scenario, while the default implementation would be specified to be in-place restart?
- we could standardize today's "restart" to mean "restart-in-place", i.e. teardown the kernel (sub)process and start again while preserving the same kernel ID.
- that means "fixing" EG/provisioners (possibly with an option); today, EG treats "restart" as a "hard restart", i.e. teardown kernel process and its environment.
- complexity/maintenance concerns
- extension implementing hard restart could come later if there is a demand from users
- this could still benefit from a JEP documenting the new default/tightening specification
Jupyter Open Studio Day in NYC Monday, August 28
- https://go.bloomberg.com/attend/invite/jupyter-open-studio-day-august2023/

Zsailer commented 1 year ago

August 24th, 2023

Name	affiliation	username
Zach Sailer	Apple	@Zsailer
Hind Montassif	QuantStack	@Hind-M
Matt Wiese	Cornell / Buckler Lab	@matthewwiese
Eric Gentry	Anaconda	@ericsnekbytes
Andrii Ieroshenko	AWS	@andrii-i
Kevin Bates	Veritone	@kevin-bates
Afshin T. Darian	QuantStack	@afshin
Johan Mabille	QuantStack	@JohanMabille
Omar Jarjur	Google	@ojarjur

Agenda

Hind: PR to add a test for XPUB JEP in JKT https://github.com/jupyter/jupyter_kernel_test/pull/134
- This PR to Jupyter Server still needs review: https://github.com/jupyter-server/jupyter_server/pull/1303
Eric G
- Documentation subproject proposal is open, comment if you're interested!
- Documentation is certainly an area that needs attention!
- The group was very enthusiastic about having a group form around this idea. :smile:
- Question: should this be a working group or a subproject?
  - subproject means a vote on SSC.
  - The process to bootstrapping this group:
    1. Gauge interest
    2. Find some folks that regularly commit to working on this
    3. Form a team-compass and organize yourselves
    4. Go to EC and ask for a formalization.
Meeting ended early due to a short agenda

Zsailer commented 1 year ago

August 31st, 2023

Name	affiliation	username
Zach Sailer	Apple	@Zsailer
Andrii Ieroshenko	AWS	@andrii-i
Matt Wiese	Cornell / Buckler Lab	@matthewwiese
Sergey Kukhtichev	IBM	@skukhtichev
Afshin T. Darian	QuantStack	@afshin

Agenda

Question from Andrii:
- When my teammate tries to run galata tests locally, tests don’t run and are stuck and initial screen that asks for token. Investigation showed that apparentely his configure_jupyter_server call is not getting propagated because changing port there also doesn't change anything
  - Advise: sounds like environments are getting cross-polinated, jupyter server is running from different environment than galata server. jupyterlab call would be a good place to ask more
- I'm working on event driven notebook execution and want to execute notebook locally based on Github PR push. I found one can modify "jupyter_server_config.json" to expose yourself to the internet. Is there a better way? What should I keep in mind?
  - Advise: allow access from all IPs, use Nginx/Caddy/Traefik, put everything into a docker file
```
{  
"ServerApp": {
"password": "YOUR ENCRYPTED PASSWORD IS HERE",
"allow_remote_access":true,
"ip":"*",
"port":80,
"allow_root":true
}
}
```

Zsailer commented 1 year ago

September 7th, 2023

Name	affiliation	username
Zach Sailer	Apple	@Zsailer
Johan Mabille	QuantStack	@JohanMabille
David Brochart	QuantStack	@davidbrochart
Andrii Ieroshenko	AWS	@andrii-i
Steven Silvester	MongoDB	@blink1073

Agenda

Zach
- Document state maintained in server (more discussion/brainstorming).
  - Why?
    - Real time collaboration
    - Capture changes that occur when frontend gets detached.
  - How it works today
    - The frontend starts a Jupyter Server "session"
    - the session returns a mapping of "document" to kernel
    - the frontend opens a kernel websocket
    - the frontend sends+receives messages from kernel websocket
    - the frontend determines how these messages change the document.
    - if the frontend gets disconnected, messages get dropped.
    - there is messages buffering triggered each time a kernel websocket loses connection, but JupyterLab requests a new websocket everytime it disconnects and the buffer is lost.
  - Brainstorming alternative approach for a server
    - Move the message buffering to Session service.
      - Jupyter Server "sessions" track the mapping between documents and kernels
      - We need the message buffer to persist beyond websocket lifetimes.
        
        Specifically, the buffer should exist for the lifetime of a document-kernel relationship, i.e. a "session" in Jupyter Server language.
        
        When 0 browser clients are connected, sessions can still exist and continue receiving messages from the kernel.

Zsailer commented 1 year ago

September 14th, 2023

Name	affiliation	username
Zach Sailer	Apple	@Zsailer
Johan Mabille	QuantStack	@JohanMabille
David Brochart	QuantStack	@davidbrochart
Andrii Ieroshenko	AWS	@andrii-i
Steven Silvester	MongoDB	@blink1073
Kevin Bates	Veritone	@kevin-bates
Afshin T. Daria	QuantStac	@afshin
Rosio Reyes	Anaconda	@RRosio

Agenda

Steve
- sp-repo-review - handy new tool from Scientific Python folks
- traitlets typing :tada:
  - Follow up issue for some limitations
  - We could consider using mypyc to have optional C optimization
  - Python 3.6, PEP 487 around set_name. If used properly, we could remove the metaclass in trailets. This would make traitlets a lot faster to instantiate. Allow us to remove half of the code in the base of traitlets.
Johan
- https://github.com/jupyter-server/jupyter_server/pull/
- Still need review.
Darian
- schema.jupyter.org: https://github.com/jupyter/schema/
- It is now live. Will mention at the next SSC meeting. :tada:
Sylvain
- Awakening the deadlock.

Zsailer commented 1 year ago

September 21st, 2023

Name	affiliation	username
Zach Sailer	Apple	@Zsailer
Matt Wiese	Cornell / Buckler Lab	@matthewwiese
Johan Mabille	QuantStack	@JohanMabille
David Brochart	QuantStack	@davidbrochart
Andrii Ieroshenko	AWS	@andrii-i
Rosio Reyes	Anaconda	@RRosio

Message Replay Discussion
- Four different types of sessions
  1. Kernel <-> Notebook (on server)
  2. Browser (e.g. a browser tab, can be open but disconnected from server)
  3. Kernel WebSocket (in JupyterLab, distinct from Browser jupyter-server/jupyter_server#2)
  4. "User" session (same token/cookie across many browsers, essentially same identity)
- Message dropping happens when WebSocket is down
  - As a result, messages must be buffered
  - Need to be kept around for Kernel-Notebook session mapping
- Message replay has additional subtlety/complexity due to these distinct kinds of sessions
  - Sessions must be tracked in order to correctly map messages to its correct destination
  - Need to manage the accumulation of these messages to avoid memory leaks (orphaned messages)
  - How/when can we confirm the message has reached its destination?
- Zach's idea for tackling this:
  - Create edge case scenarios involving how conflicts between the sessions are resolved
  - Flow chart for visualizing/understanding how the various sessions interact with and depend on one another
    - Add this to the documentation for Jupyter Server for our own sake, and to clarify for end-users
Question from Andrii
- Working on Jupyter Scheduler
- What is the best way to recreate, reproducibly a notebook's environment for a scheduled notebook using conda/mamba?
  - environment.yaml file for listing dependencies
  - (not implemented yet), use a lock file to fix the resolution tree one time, for all time.
- Suggestions for projects to look at:
  - https://github.com/elyra-ai/elyra
  - https://github.com/prefix-dev/pixi

Zsailer commented 1 year ago

September 28th, 2023

Name	affiliation	username
Zach Sailer	Apple	@Zsailer
Matt Wiese	Cornell / Buckler Lab	@matthewwiese
Andrii Ieroshenko	AWS	@andrii-i
Rosio Reyes	Anaconda	@RRosio
Steve Silvester	MongoDB	@blink1073

Agenda

Andrii:
- conda-forge question
- two conda-forge feedstocks, jupyter-ai and jupyter-ai-magics.
- it seems that both aren't being published to conda-forge.
- when we checked again during the meeting, it was published; so it appears resolved! :rocket:
Trait Typing
- Let's look into how this affects server
- PRs in traitlets:
  - https://github.com/ipython/traitlets/pull/818
  - https://github.com/ipython/traitlets/pull/868
- PRs for updates
- How do we move in the direction of stricter typing in Jupyter Server?
  - Start strict.
  - Then (temporarily) ignore things that fail.
  - Over time, work through the ignored pieces.
Discuss "Resolve server absolute path to server or kernel-relative path"
- https://github.com/jupyter-server/jupyter_server/issues/1280
- Reaching for content outside of root dir is the big question.
- There is already precedence for this...the debugger looks at files outside of the root dir.
- Maybe a general API for transforming things from the client via the server. The example here (file paths) is one use case, but having a general API that can transforming things the client wants, the server sees, but (we can't assume) the kernel knows about.

mbektas commented 1 year ago

Question from Andrii

Working on Jupyter Scheduler

What is the best way to recreate, reproducibly a notebook's environment for a scheduled notebook using conda/mamba?

@andrii-i we use conda-lock and conda-pack in JupyterLab Desktop to bundle and install the Python environment. It works pretty well across different OS. You can checkout the repo for reference or this PR more specifically.

andrii-i commented 12 months ago

Thank you @mbektas

jupyter-server / team-compass

Meeting Notes 2023 #45

Hello everyone,

Meeting Notes

January 5th, 2023

January 12th, 2023

January 19th, 2023

January 26th, 2023

February 2nd, 2023

February 9th, 2023

February 16th, 2023

February 23th, 2023

Agenda

March 2nd, 2023

Agenda

March 9th, 2023

Agenda

March 16th, 2023

Agenda

March 23rd, 2023

Agenda

March 30th, 2023

Agenda

April 6th, 2023

Agenda

April 13th, 2023

Agenda

April 20th, 2023

Agenda

April 27th, 2023

Agenda

May 4th, 2023

Agenda

May 18th, 2023

Agenda

May 25th, 2023

June 1st, 2023

Agenda

June 8th, 2023

Agenda

June 15th, 2023

Agenda

June 22nd, 2023

Agenda

July 6th, 2023

Agenda

July 13th, 2023

Agenda

July 20th, 2023

Agenda

July 27th, 2023

Agenda

August 3, 2023

Agenda

August 10, 2023

Agenda

August 17th, 2023

Agenda

August 24th, 2023

Agenda

August 31st, 2023

Agenda

September 7th, 2023

Agenda

September 14th, 2023

Agenda

September 21st, 2023

September 28th, 2023

Agenda