Meeting Notes 2024 - Githubissues

jupyter-server / team-compass

A repository for team discussion, syncing, and meeting notes.

https://jupyter-server-team-compass.readthedocs.io

BSD 3-Clause "New" or "Revised" License

13 stars 7 forks source link

Meeting Notes 2024 #57

Open Zsailer opened 6 months ago

Zsailer commented 6 months ago

Hello everyone,

Welcome to the Jupyter Server Team!

We meet on Thursdays at 8:00am, Pacific Standard Time on Jupyter's Zoom Channel.You can add yourself to the weekly agenda here. Everyone is welcome!

Let's avoid using this thread for discussion. If you'd like to discuss something in the minutes, open a separate issue and reference this thread.

You can find previous year's notes here: 2020, 2021, 2022, 2023

Meeting Notes

Zsailer commented 6 months ago

January 4th, 2024

Name	affiliation	username
Zach Sailer	Apple	Zsailer
Johan Mabille	QuantStack	@JohanMabille
Andrii Ieroshenko	AWS	@andrii-i
Ian Thomas	QuantStack	@ianthomas23
David Brochart	QuantStack	@davidbrochart

Agenda

Zach
- New year, new meeting thread!
- Membership maintenance check closes tomorrow.
  - Thank you, everyone, for a great year!
- Port kernel gateway to Jupyter Server 2.x
  - Thank you, Kevin, for doing a majority of the work! :clap:
  - Requires a major release (3.x) and maintenance of the old branch (2.x)
  - I will announce on discourse once released.
  - Ran into issues getting the websocket subprotocol working, so the Gateway client defaults to the legacy protocol.
- Would anyone else like to moderate the Jupyter Server meeting?
- Also, new year means we should run a fresh election for Jupyter Server representative.

Zsailer commented 5 months ago

January 11th, 2024

Name	affiliation	GitHub username
Zach Sailer	Apple	Zsailer
Vidar Fauske	JP Morgan Chase	vidartf
Jason Grout	Databricks	@jasongrout
Sylvain Corlay	QuantStack	@QuantStack
Ian Thomas	QuantStack	@ianthomas23
David Brochart	QuantStack	@davidbrochart
Johan Mabille	QuantStack	@JohanMabille
Andrii Ieroshenko	AWS	@andrii-i

Agenda

Zach
- schema.jupyter.org progress
  - https://github.com/jupyter/schema/pull/1
  - Requirements for this repo
    - Installs schemas on disk somewhere (python package?)
      - The current PR uses a Python package
    - Makes schemas available from schema.jupyter.org (permanent URLs)
    - Import schemas at runtime (in any language, in principle)
    - Generate types from schemas in any language, in principle
    - CLI as a Jupyter core application
  - Question: what schemas should go here?
    - event schemas from subprojects
    - specs for connection_file
    - specs for the kernelspec
    - specs for the kernel protocol
    - specs for jupyter mimetypes
    - jupyter_server OpenAPI spec
      - some discussion around this point.
      - there is a difference between the API surface and what the handlers do in response to that API. Maybe we can start by formalizing the API service.
  - Ownership of schemas
    - Which subproject owns this repo?
      - [@jasongrout] Foundations and Standards seems like the natural home
    - Is this the source of truth for schemas? For example, is nbformat the home for the nbformat schema, and it's copied here, or does the official schema live here?
      - [@jasongrout] It's nice having nbformat schema co-located with the tools that work with it, for example. Though I suppose it's not too much of a stretch to have the nbformat package depend on the schemas package.
  - Package versioning
    - Do we ship a separate package for each schema? Or do we bump the package for any schema update?
Sylvain / Ian: Subshells
- https://github.com/jupyter/enhancement-proposals/pull/91
- Ian is planning on summarizing the current state of the work over the last few years
- Two main approaches have been discussed
  - "Subshells": A single thread running a message router to different execution threads
    - Everyone's mental modeling of threading works
  - "Dependent kernels": Multiple execution threads, each with their own ZMQ channels (except control channel is shared), so it appears to the outside world as a collection of independent kernels
    - Pretend like it's a kernel and things "just work"
    - Except lifecycle is slightly more complicated because control thread is shared. For example, what does it mean to shutdown a kernel? Do we shut down the single thread, or the whole process?
- Ian showed demos of the "dependent kernels" approach
- There was discussion about how multiple threads interact with status messages. Right now status messages are sent for control message processing, which breaks the mental model for status messages.
- Are there concerns about multiple threads interacting with the same display objects? Is it possible to do it in a thread safe manner? Or should diplay objects be owned by specific threads? What is the case for multiple clients?
  - [@jasongrout] I used to think the GIL protects us there, but some things I read recently helped me realize Python multithreading can be really tricky. (1, )
    - PEP 703 maybe not always rely on GIL?
- Possibly in the future, status messages indicate busy/idle status for a single execution thread.
- What's next?
  - QuantStack has funding to work on this, and is planning on working through the Jupyter process in the next 4-6 months with implementation landing within a year
  - We need to decide the basic approach, so have a discussion and move forward
David
- YDrive: CRDT-based contents API
- Will implement something for JupyterLab for this collaborative drive. Starting today.

Zsailer commented 5 months ago

January 18th, 2024

Name	affiliation	GitHub username
Zach Sailer	Apple	Zsailer
Johan Mabille	QuantStack	@JohanMabille
Jason Grout	Databricks	@jasongrout
Mike Krassowski	Quansight	@krassowski
Vidar Fauske	JP Morgan Chase	@vidartf
Ian Thomas	QuantStack	@ianthomas23
David Brochart	QuantStack	@davidbrochart
Andrii Ieroshenko	AWS	@andrii-i
William Stein	SageMath, Inc.	@williamstein

Mike
- jupyter-lsp vulnerability - please update to 2.2.2 https://github.com/jupyter-lsp/jupyterlab-lsp/security/advisories/GHSA-4qhp-652w-c22x
  - Thanks to @bollwyvl and Jupyter Security team sheparding the process
  - Should we act on this issue in response: https://github.com/jupyter-server/jupyter_server/issues/389
- Ownership of lsp github org
  - jupyter-lsp pointed to ipython-security list, so the user did the right thing
  - ydoc/real-time collaboration has the same server/frontend split confusion
  - Perhaps there is a general correction in the future
    - Move everything into a single org, manage permissions with teams
    - Consolidate and simplify number of orgs
  - Mike enabled GitHub private security reports
  - Tactically: Is jupyter-lsp under Project Jupyter
- Directing output of out external threads/processes to specific cells https://github.com/ipython/ipykernel/issues/1199
Ian Thomas
- Update on concurrent kernels (subshells)
- https://github.com/jupyter/enhancement-proposals/pull/91#issuecomment-1895805262
- We spent ~5 minutes reading the newest update then opened for discussion.
- Could it be that different dependent kernels have different ip addresses, i.e., totally different connection files?
  - Ian was thinking the connection file for the new dependent kernel just has a different shell port
  - If you have a different ip address, dependent kernels becomes multi-processing instead of multi-threaded
- Dependent kernels require quite a bit of server changes
- If we make the feature optional, the dependent kernels may be easier since you don't have to modify the kernel protocol
- How backwards compatible is the approach for clients that support concurrent kernels, but kernels that don't?
  - subshells: the kernel just ignores the subshell field and messages are considered in order. The guarantee is that messages in a single subshell are processed in order.
  - dependent kernels: the api just hands back
- the big question i have is how this kernel metadata stored in the notebook format? the notebook document stores compute information in its metadata. will subshells be marked up in the cell metadata to describe compute?
  - For many users, they won't use concurrent execution. Mostly the system uses concurrent execution
- How are concurrent executions recorded in a notebook? Execution counter? Do different cells store which dependent kernel they use?
- How does reactive execution work with this? For example, ipyflow (see paper here)
Community survey for kernels?

Zsailer commented 5 months ago

January 25th, 2024

Name	affiliation	GitHub username
Zach Sailer	Apple	Zsailer
William Stein	CoCalc/SageMath	@williamstein
Vidar T Fauske	JP Morgan Chase	vidartf
David Brochart	QuantStack	@davidbrochart
Johan mabille	QuantStack	@JohanMabille
Ian Thomas	QuantStack	@ianthomas23
Afshin T. Darian	QuantStack	@afshin
Jason Grout	Databricks	@jasongrout

Zach
- Jupyter Server Representative nominations
Johan
- Jupyter Kernels Representative nominations (ends February 1st)
Questions:
- Is there a Jupyter project wide approach to documenting supported/deprecated Python versions (like NEP 29)?
  - IPython follows https://scientific-python.org/specs/spec-0000/
  - We tend to follow the CPython release cycle.
- If I write an extension today, can I target JupyterLab only or is there an expectation that I also support the classic notebook?
  - It depends on your audience.
  - There isn't an expectation to write extensions for both.
  - Notebook v7 makes it easier to write extensions that work in both.
[Ian] Update on concurrent execution? (requested by @jasongrout)
- Next step: comments from community about what Ian has written
- If no comments, update the JEP
- @jasongrout and @Zsailer both have usecases that would benefit from concurrent execution

Zsailer commented 4 months ago

February 1st, 2024

Name	affiliation	GitHub username
Zach Sailer	Apple	@Zsailer
Michał Krassowski	Quansight	@krassowski
William Stein	SageMath	@williamstein
Ian Thomas	QuantStack	@ianthomas23
Kevin Bates	Veritone	@kevin-bates
David Brochart	QuantStack	@davidbrochart

Agenda

Mike: resolving path/file ownership API
- https://github.com/jupyter-server/jupyter_server/pull/1331
- https://github.com/jupyter/jupyter_client/pull/1005
  - gateway would need to do more
- Nick:
  - Can we do this over comms? Wrap the Contents API with comms and sent files over comms to access kernel files.
  - this would avoid creating a new API spec for this purpose. we'd reuse contents API.
  - perhaps the new (proposed) subshell work is another way to do this
  - there would be no gateway issue in this case
- Where should we continue this conversation?
  - Open an issue in jupyter_client: https://github.com/jupyter/jupyter_client/issues/1006
Ian:
- https://github.com/jupyter/enhancement-proposals/blob/master/92-jupyter-optional-features/jupyter-optional-features.md
- Has anyone implemented any optional features yet?
  - Not aware of any.
- Registry of names of such features?
  - Need a schema for current kernelinfo, then extend it for optional features.
  - After meeting found JEP for this: https://github.com/jupyter/enhancement-proposals/pull/105
Kevin:
- Dependent kernels vs. subshells status?
- Dependent kernels seems to present an extra challenge when managing kernels.
  - Today, in Jupyter Server, we have MultiKernelManager-->KernelManager
  - This would introduce an additional layer of management, e.g. MultiKernelManager-->KernelManagerShells-->DependentKernelManager
- Ian: given concerns about dependent kernels, kernel subshells is the current favourite
- Ian will prepare some instructions for easy use of the demo code so that other maintainers can experiment

Zsailer commented 4 months ago

February 8th, 2024

Name	affiliation	GitHub username
Zach Sailer	Apple	@Zsailer
Johan Mabille	QuantStack	@JohanMabille
Ian Thomas	QuantStack	@ianthomas23
Steve Silvester	MongoDB	@blink103
David Brochart	QuantStack	@davidbrochart
Mike Krassowski	Quansight	@krassowski

Agenda

Zach
- Nomination period for Jupyter Server representative ended.
- Move to voting phase.
  - We don't have an established way to vote. Last year, it was a Google Form, but that isn't anonymous.
  - If we go that route, it shouldn't be managed by people on the ballot.
Johan
- voting period for Jupyter Kernels representative has started, the vote is public, on the team compass
- Nomination phase for JDCs has started
Mike
- any comments on https://github.com/jupyter/jupyter_client/issues/1006?
  - possibly suboptimal for the use case of linkifying tracebacks because we have to go to the kernel and get it
- preliminary itnent to work on secure by default (opt-in in the current version, possibly opt-out in a future major release) https://github.com/jupyter-server/jupyter_server/issues/389
Steve
- Asyncio story
  - Remove tornado eventloop, while keeping it available for backwards compatibility
  - Async ZMQContext is not compatible
    - https://github.com/voila-dashboards/voila/issues/1428#issuecomment-1927025620
    - https://github.com/zeromq/pyzmq/issues/1405
  - We should document this limitation in Client 8, and suggest upgrading to Client 9 for a proper fix.
  - Will continue to push ahead with https://github.com/jupyter/jupyter_client/pull/997, which needs a bit more work to pass the Jupyter Server 2.0 test suite.

Question:

What is the current recommended combination for RTC?
- Currently jupyter-collaboration 2 uses pycrdt-websocket, for pycrdt, which is where development will take place going forward as ypy is not being actively maintained at the moment.
Discussion about commenting
- we need https://github.com/jupyter/nbformat/issues/317

Zsailer commented 4 months ago

February 15th, 2024

Name	affiliation	GitHub username
Zach Sailer	Apple	@Zsailer
Ian Thomas	QuantStack	@ianthomas23
Jason Grout	Databricks	@jasongrout

Mike (cannot join)
- Add an option to have authentication enabled for all endpoints by default
Zach
- SSC Rep vote is open: https://github.com/jupyter-server/team-compass/issues/59
Ian: https://github.com/jupyter/enhancement-proposals/pull/91
- Kernel subshells seems like it is a winning proposal.
- Wording of the JEP has been updated in line with latest thinking, needs review.
- Dependent kernels might be something we revisit in the future.
Jason: Invited a bunch of students to the contributing hour. Probably at least 2 will show up???

Zsailer commented 4 months ago

February 22nd, 2024

Name	affiliation	GitHub username
Jason Grout	Databricks	@jasongrout
Sergey Kukhtichev	IBM	@skukhtichev
David Brochart	QuantStack	@davidbrochart
Johan Mabille	QuantStack	@JohanMabille
Ian Thomas	QuantStack	@ianthomas23

Agenda

Introductions
- [@jasongrout] BYU-Idaho students last week
  - Fix on Windows!
  - General observation = hard to develop on Windows
  - Nick: DOI important to provide credit for grad students, etc
- Sergey Kukhtichev
[@jasongrout] matplotlib and post_execute vs post_run_cell: a complication for subshells
- Matplotlib-inline corrupted images
- Comm messages on control channel not really expected
- Will be PRs from Jason in matplotlib and matplotlib-inline
Ian - schemas https://github.com/jupyter/schema/pull/1
- It appears we have two repos: https://github.com/jupyter-standards/schemas and https://github.com/jupyter/schema
  - NB: Johan just archived https://github.com/jupyter-standards/schemas which was a residue
- Need some CI to demonstrate its use
- Nick - start with kernelspec before kernel messages
- Johan - JEP open for kernelspec, needs a vote at next SSC meeting
- Suggest *_request and *_reply messages to be in the same schema
- Structure of directions in final public schema website of e.g. jupyter_client/whatever (2 levels, top one is the project)
- There is a standard schema for reporting validation errors that we should follow.

Zsailer commented 4 months ago

February 29th, 2024

Name	affiliation	GitHub username
Johan Mabille	QuantStack	@JohanMabille
Vidar T Fauske	JP Morgan Chase	@vidartf
Zach Sailer	Apple	@Zsailer
Steve Silvester	MongoDB	@blink1073
Ian Thomas	QuantStack	@ianthomas23

Agenda

[Steve] Async Updates across the stack
- Merged Ian's work to remove control queue from IPykernel
- Combine this with David's work to remove Tornado IOLoop
- Steve is working on grafting these two things together, which is proving to be challenging.
- Zach and Steve are working on getting a new implementation of Async Kernel Manager out to remove ZMQ Stream and Tornado IOLoop.

[Nick] Schema repo discussion
- Let's not do any RST for documentation.
- Markdown forward.
- That's aim for tooling that works both locally and on CI.
- Get rid of Makefile
- Add sphinx autobuild
- Validate URI of schemas against their location in the repo
- We should spend some time thinking about URI structure
  - Namespacing shouldn't be a question of "which project owns this schema", such as /jupyter_server/ or jupyter_client
  - Intead, let's namespace based on nouns, e.g. /server/, /kernels/, etc.

Zsailer commented 3 months ago

March 7th, 2024

Name	affiliation	GitHub username
Zach Sailer	Apple	@Zsailer
Jason Grout	Databricks	@jasongrout
David Brochart	QuantStack	@davidbrochart
Ian Thomas	QuantStack	@ianthomas23
Steve Silvester	MongoDB	@blink1073

Agenda

[Steve] Releases and security updates
- PR merged in server for enhanced security around release actions
- Using a Github App for publishing. No longer using Github Admin Token. This also solves the issue of having to regularly update the token.
- Can't use this for NPM yet.
- Private release feature in jupyter-releaser (thanks Fred)
  - Useful for security patch releases; using a two part
    1. Publish to PyPI and create a tag.
    2. (1 week later) publish changelong and Github release using a separate workflow
[@jasongrout] Linux Foundation proposal
- There is an open issue for discussion.
[@jasongrout] What is the current status of the prototype for concurrent kernel execution?
- We might experiment with this at Databricks next week in an engineer hackathon
- The current prototype is definitely demo code. You can create subshells, you can send execute requests. Potentially the status info is off. The main thread is a router instead of the main execution thread, where it needs to be. You can't delete subshells yet.
- Anyio prototype is also in progress, and could impact the subshell work. David was working on this, but it has passed over to Steve now. This would make ipykernel asyncio-first, and would simplify future developments. Advantages: support for trio and uvloop.
- related: AnyIO support in pyzmq https://github.com/zeromq/pyzmq/issues/1827

Zsailer commented 3 months ago

March 14th, 2024

Name	affiliation	GitHub username
Zach Sailer	Apple	@Zsailer
Ian Thomas	QuantStack	@ianthomas23

Agenda

[Zach]
- General announcements for Jupyter Server Team
  - Please read, review, comment (if you'd like) on the move to Linux Foundation proposalLinux Foundation proposal
  - Voting has opened for the Executive Council

Zsailer commented 3 months ago

March 21st, 2024

Name	affiliation	GitHub username
Jason Grout	Databricks	@jasongrout
David Brochart	QuantStack	@davidbrochart
Ian Thomas	QuantStack	@ianthomas23
Omar Jarjur	Google	@ojarjur
Zach Sailer	Apple	@Zsailer
Johan Mabille	QuantStack	@JohanMabille

Agenda

[Ryan, Gabriel, Jason] Lessons from exploring implementing the subshell demo in Databricks
- Overall, the demo worked great!
- Parent headers are per channel (e.g., "control", "shell"), so are inaccurate for subshells. We hacked a parent header per subshell
- Recent changes for output per thread and eliminating the control queue conflict with the subshell prototype
- We didn't finish swapping the main thread to be code execution instead of message routing, so interruption still didn't work
- We added a shell_id parameter to create_subshell_request so we could explicitly set a subshell id (and thereby consolidate subshell ids if desired)
- We changed the name of the thread to be shell-{name} to aid in debugging
- Is there a reason we are using zmq/pickling instead of a Python Queue or SimpleQueue to communicate messages between threads?
- Thread safety issues are tricky (like autocomplete requests might run 3rd party library code that assumes single-threaded)
- Read up on subinterpreters coming in Python 3.13. Perhaps spawning a subinterpreter is a different kind of subshell, or perhaps subinterpreter is well-suited to a dependent kernel approach
[Omar] Requesting feedback on https://github.com/jupyter-server/jupyter_server/issues/1360
- Proposed change https://github.com/ojarjur/jupyter_server/commit/30b45dbace700381203b5a07f3c8e0335fce0271

Reminder:

EC council election is currently going on, see your email if you are on a Jupyter council or committee
Linux Foundation proposal is currently under discussion

Zsailer commented 2 months ago

March 28th, 2024

Name	affiliation	GitHub username
Zach Sailer	Apple	@Zsailer
David Brochart	QuantStack	@davidbrochart
Piyush Jain	AWS	@3coins

Agenda

Zach
- I can't host contributing hour today.
Ryan
- debugger not working on background thread
Piyush
- This issue was reported on gitter a few weeks back. Are there any suggestions on how to fix this.
  - Discussed the issue and the next step is to open an issue with all the information. Zach mentioned that this could be a regression from server 1.0, so should try with that. There is also an env variable JUPYTER_SERVER_ROOT which is not configurable at the moment, but could be made into a configurable, which can help nudge the CWD for terminals at runtime.

Bug around inaccurate execution_state in the client

Zsailer commented 2 months ago

April 4th, 2024

Name	affiliation	GitHub username
Ian Thomas	QuantStack	@ianthomas23
R Ely	Bloomberg	@ohrely
Jason Grout	Databricks	@jasongrout
Maico Timmerman	Adyen	@MaicoTimmerman

Agenda

Ely:
- Jupyter Open Studio Day NYC Monday April 29
  - Registration link
Darian:
- Linux Foundation proposal: that Jupyter move from NumFOCUS to Linux Foundation
- Discussion here
Jason: update on async kernel execution?
- Ian may be able to start working on this a few weeks from now
- Jason may be able to find some engineering time as well to coordinate with Ian on writing an implementation in May/June.
- There are concerns about the tests that are broken with the anyio changes.
- No one is currently actively working on the any-io test failures, but Ian has worked on at least one.
Maico: I'm looking for some support on an MR on the enterprise-gateway. I was hoping to utilize the meeting to get in touch with the correct person.

Zsailer commented 2 months ago

April 11th, 2024

Name	affiliation	GitHub username
Zach Sailer	Apple	@Zsailer
Jason Grout	Databricks	@jasongrout
David Brochart	QuantStack	@davidbrochart
Afshin T. Darian	QuantStack	@afshin
Johan Mabille	QuantStack	@JohanMabille
Ian Thomas	QuantStack	@ianthomas23
Mike Krassowski	Quansight	@krassowski
Steve Silvester	MongoDB	@blink1073

Agenda

Random conversation about packaging
Zach:
- Drives in Jupyter Server (yes, let's talk about this again :)).
  - Implicit and Explicit paths issues
  - https://github.com/jupyterlab/jupyterlab/issues/16099#issuecomment-2043839359
  - Build on the current contents manager and defer to super() if no prefix is given
  - Prefix can't be a file name
  - Issues to consider while we do this:
    - Jupyter notebook can't open the dir named "checkpoints" https://github.com/jupyter-server/jupyter_server/issues/950
    - metrics/metrics.py file (and other files prefixed with "metrics") cannot be retrieved via contents API https://github.com/jupyter-server/jupyter_server/issues/573
  - What drives would Jupyter provide under URIs?
    - jupyter-kernels://
    - jupyter-contents://
    - jupyter-kernelspecs://
    - rtc://
    - jupyter-checkpoints://
- Default to authorization across the server?
  - Similar to https://github.com/jupyter-server/jupyter_server/pull/1392
- Add option to add identity to events.
@jasongrout:
- concurrent kernel execution
- any LF proposal questions?
Mike
- jupyter-server server-side execution, highlighting discussions in:
  - https://github.com/jupyter-server/jupyter_ydoc/pull/227
  - https://github.com/jupyterlab/jupyter-collaboration/pull/279

Zsailer commented 2 months ago

April 18th, 2024

Name	affiliation	GitHub username
Zach Sailer	Apple	@Zsailer
Steve Silvester	MongoDB	@blink1073
Johan Mabille	QuantStack	@JohanMabille
Ian Thomas	QuantStack	@ianthomas23
David Brochart	QuantStack	@davidbrochart
Vidar Fauske	JP Morgan Chase	@vidartf
Mike Krassowski	Quansight	@krassowski

Agenda

SSC rep vs Zach leaving for EC
- Nominations for a new Rep are open until Monday
https://github.com/jupyter-server/jupyter_server/issues/1416
- Bringing folks attention to this.
Mike: does PR https://github.com/jupyter-server/jupyter_releaser/pull/568 make sense?

Zsailer commented 2 months ago

April 25th, 2024

Name	affiliation	GitHub username
Zach Sailer	Apple	@Zsailer
David Brochart	QuantStack	@davidbrochart
Vidar Fauske	JP Morgan Chase	@vidartf
Afshin T. Darian	QuantStack	@afshin

Agenda

Zach
- SSC Rep moving into voting phase today
  - Multiple candidates
  - Rank-based voting
  - Single transferable vote strategy for counting votes.
  - Apache STeVe software we'll use to run the vote.
- Async start_extension hook: https://github.com/jupyter-server/jupyter_server/pull/1417
- I won't be able to run Contributing Hour anymore (at least for the time being). I'll remove the calendar invite.

Zsailer commented 1 month ago

May 2nd, 2024

Name	affiliation	GitHub username
Zach Sailer	Apple	@Zsailer
Steve Silvester	MongoDB	@blink1073
David Brochart	QuantStack	@davidbrochart
Luciano Resende	Apple	@lresende
Ian Thomas	QuantStack	@ianthomas23

Agenda

Vote for your SSC Representative!
Asking for reviews of start hook pull request: https://github.com/jupyter-server/jupyter_server/pull/1417

Zsailer commented 1 month ago

May 9th, 2024

Name	affiliation	GitHub username
Zach Sailer	Apple	@Zsailer
Vidar T Fauske	JP Morgan Chase	@vidartf
Mike Krassowski	Quansight	@krassowski
Steve Silvester	MongoDB	@blink1073
Ian Thomas	QuantStack	@ianthomas23
A T Darian	QuantStack	@afshin

Agenda

Zach
- Announcing new Jupyter Server SSC Representative!
  - Congratulations to Vidar Fauske for being elected as the new Jupyter Server SSC Repo
  - How do we make election results more transparent?
  - Add an SSC Drive to host election forms
    - Zach will open an issue on governance?
- Volunteers to run the Jupyter Server/Kernels weekly call?
Mike
- restoring full state (related to server side execution)
  - https://github.com/datalayer/jupyter-server-nbmodel
  - three missing things:
    - removing pending execution state indicator and replacing with execution count,
    - stdin boxes
    - timing metadata
  - discussions: https://github.com/jupyter-server/jupyter_ydoc/issues/169
    - rest API: https://github.com/jupyter-server/jupyter_server/issues/990
    - execution_state: https://github.com/jupyter-server/jupyter_ydoc/pull/197
    - pending_requests: https://github.com/jupyter-server/jupyter_ydoc/pull/227
      - execute_reply are not replied after refresh (not on the wire... zmq level?) :(
  - Mike asked Zach's opinion on if REST API proposal listed above is the "right" way to go?
    - Zach said this proposal was motivated by some issues he saw where we leak resources/websockets too easily in Jupyter Server
    - He was aiming to simplify the API by providing a REST API for kernel messages and leverage the Event APIs as the single source of structured/schematized websockets.
    - Mike pointed out that one challenge of this approach is that, because the request (via REST) and the reply (events websocket) are not going through the same API, the cache needed to track their relationship might be difficult to implement in a way that doesn't cause memory leaks.
  - Discussion about what model should live on the server?
    - Idea from Nick. Instead of a single document model; have a "workspace" CRDT model.
    - Make the ydoc changes looks something like JupyterLab commands.
    - These in-memory models are really "UI models", telling each client how to rebuild the UI exactly how the user left it.
    - Imagine CRDT in Voila; RTC on widgets in voila would be amazing.

Zsailer commented 1 month ago

May 16th, 2024

Name	affiliation	GitHub username
Zach Sailer	Apple	@Zsailer
Vidar T Fauske	JP Morgan Chase	@vidartf
Frederic Collonval	WebScIT / Datalayer	@fcollonval
Johan Mabille	QuantStack	@JohanMabille
David Brochart	QuantStack	@davidbrochart
Ian Thomas	QuantStack	@ianthomas23

Agenda

Frederic
- Kernel execution on server side
  Demo with jupyter-server
  Prerequisites:
  - https://github.com/jupyterlab/jupyter-collaboration/pull/307 --> Raise lots of questions when using the shared document model
  - https://github.com/jupyter/jupyter_client/pull/1023 --> AsyncClient is blocking without this
Zach
- Major "Thank you" to Frederic and Mike for quickly reviewing and releasing our bug fix to jupyterlab-git! :heart:
- Jupyter Server Meeting Hosts

Zsailer commented 1 month ago

May 23rd, 2024

Name	affiliation	GitHub username
Zach Sailer	Apple	@Zsailer
Johan Mabille	QuantStack	@JohanMabille
Ian Thomas	QuantStack	@ianthomas23
Piyush Jain	AWS	@3coins
Steve Silvester	MongoDB	@blink1073
Afshin T. Darian	QuantStack	@afshin
Frederic Collonval	WebScIT / Datalayer	@fcollonval

Agenda

Steve - recap of PyCon US 2024
- Highly recommended talk from Anthony Shaw Unlocking the Parallel Universe: Sub Interpreters and Free-Threading in Python 3.13
- Packaging Summit - we're working toward a Packaging working group, along with a dedicated effort at the PSF level including a UX expert for "onboarding onto Python", which may include a "blessed" workflow, documentation, new tool(s), etc.
- Pittsburgh is great - try to go next year! - Try the Primanti Sandwich
- The next location will be Long Beach, CA in 2026-27.
Ian - subshells (JEP91) real implementation in progress
- Aiming for more accessible demo (e.g. via binder) for people to play with.
Discussion around Jupyter extension long-term support
- Looked at CNCF's incubation project for guidance
- Jupyter has less people resources
- https://github.com/jupyter/docker-stacks
  - A great way to show opinion combinations of Jupyter pieces together and tests that they work together
Frederic
- Discussion to collaborate on an extension implementing a kernel state machine on the server side; final goal would be to upstream it when stable.
- Zach will open two lines of work
  - One on Jupyter Client with Async kernel manager
  - A new repo with a server extension and customer kernel manager with a kernel state machine

Zsailer commented 3 weeks ago

May 30th, 2024

Name	affiliation	GitHub username
Vidar T Fauske	JP Morgan Chase	@vidartf
Piyush Jain	AWS	@3coins
David Brochart	QuantStack	@davidbrochart
Zach Sailer	Apple	@Zsailer

Agenda

Zach
- Next generation kernels API
  - Problems I'm trying to solve:
    1. the control and shell channel states are conflated.
    2. the kernel client on the server doesn't "know" the state of the kernel. For example, if the client is disconnected when the kernel state changes, recovering the true state isn't easy.
    3. every kernel websocket opens a new set of ZMQ sockets. This presents multiple issues, e.g. 1) ZMQ sockets are resource limited 2) it's difficult to track all of the places where messages are flowing (making (3) difficult to get right) 3) today, we're leaking sockets somewhere (Zach has seen this for long running servers, but hasn't tracked the cause)
  - Proposal to fix
    - (1) include the parent channel in the iopub messages, or track these messages server-side to distinguish status messages from shell/control.
    - (2) track two types of kernel states in the server (via KernelManagers)
      - lifecycle state, (e.g. starting, started, connecting, connected, terminating, terminated)
      - execution state, (busy, idle, unknown, dead)
      - Add a REST API (?) for fetching kernel state anytime the client needs to confirm or reconnect.
      - Use the event system (?) to emit kernel state?
    - (3) in jupyter-server, we should use a bit more discipline. Instead of opening individual sockets through the kernel manager API, we should open a single kernel client server-side. This opens a fixed set of ZMQ channels. All client->server->kernel connections, e.g. kernel websockets, should use this single kernel client.
      - Also, stop nudging the kernel for state. Just ask the server using the kernel state tracking from (2).

Zsailer commented 3 weeks ago

June 6th, 2024

Name	affiliation	GitHub username
Vidar T Fauske	JP Morgan Chase	@vidartf
David Brochart	QuantStack	@davidbrochart
Steve Silvester	MongoDB	@blink1073
Zach Sailer	Apple	@Zsailer
Johan Mabille	QuantStack
Omar Jarjur	Google	@ojarjur

Agenda

Zach
- Follow up on "next-generation" kernel API
  - work in progress: https://github.com/Zsailer/nexgen-kernel-manager
  - Improvements made:
    - tracks kernel lifecycle state and execution state server-side.
    - uses a single kernel client (thus, single set of ZMQ channels) to communicate with the kernel. No need to open ZMQ sockets outside of this client.
    - uses a completely native asyncio approach to poll messages from the kernel, dropping the tornado IOLoop and ZMQStream logic.
    - simplifies the websocket connection logic
      - removes all nudging logic in the websocket handler, since the kernel manager owns this now.
      - the WS handle registers itself as a listener on the kernel client
      - the websocket can connect, even if the kernel is busy. (I think) this eliminates the necessity for "pending" kernels. Every kernel can be in a "pending" state.
  - how does this affect Omar's PR, which is trying to get an accurate execution_state on the server: https://github.com/jupyter-server/jupyter_server/pull/1429
    - We should proceed with reviewing this PR, if it doesn't increase our API service.
    - We should be able to simplify this PR but just "watching" the shell channel status messages.
    - Can we assume shell messages get queued?
      - If no, we might need to additional tracking of parent message ID
      - If yes, just tracking control channel should be enough.
Steve
- Published https://github.com/jupyter-server/jupyter_server/security/advisories/GHSA-hrw6-wg82-cm62

3coins commented 1 week ago

June 13th, 2024

Name	affiliation	GitHub username
Vidar T Fauske	JP Morgan Chase	@vidartf
Steve Silvester	MongoDB	@blink1073
Zach Sailer	Apple	@Zsailer
David Brochart	QuantStack	@davidbrochart
Johan Mabille	QuantStack
Ian Thomas	QuantStack	@ianthomas23

Agenda

Zach
- New Kernels API
  - Test it out :smile:
  - https://github.com/Zsailer/nextgen-kernels-api
Johan
- Two JSON specs JEPs merged last monday:
  - Connection file: https://github.com/jupyter/enhancement-proposals/pull/106
  - Kernel Spec: https://github.com/jupyter/enhancement-proposals/pull/105
- Demonstration of a prototype of the Parametrized Kernel Specs during the next SSC working hours. https://github.com/jupyter/enhancement-proposals/pull/87
Ian
- Subshells (JEP91) implementation PR in ipykernel. MVP.
- https://github.com/ipython/ipykernel/pull/1249
Ian
- Can we have an ipykernel release (6.29.5) with https://github.com/ipython/ipykernel/pull/1237?
- Fixes historic problem using Matplotlib's macos backend.
- Ian and Steve to work together on the release in 2 weeks time so that Ian can learn the process.

3coins commented 1 week ago

June 20th, 2024

Name	affiliation	GitHub username
Piyush Jain	AWS	@3coins
Vidar T Fauske	JP Morgan Chase	@vidartf
Zach Sailer	Apple	@Zsailer
Andrii Ieroshenko	AWS	@andrii-i
Afshin T. Darian	@quantstack	@afshin
Mike Krassowski	Quansight	@krassowski

Agenda

Demo of Parameterized Kernel Specs JEP
- Context:
  - JEP: https://github.com/jupyter/enhancement-proposals/pull/87
  - Draft PR in Jupyter Server:
- Vidar - Needs a flag to turn the save/load from notebook feature off/on, could cause shell injection attacks.
  - Enum flags e.g. would not need trust/untrusted as they they do not allow for arbritrary input. https://github.com/jupyter/enhancement-proposals/pull/87/files#r1647799450
- Zach - Separate the saving of kernel parameters in notebook to a separate kernel spec file, provide a save option.
  - Should the save kernel spec be a separate feature?
  - I don't think embedding detailed info about kernel runtime/environment in the notebook document is good broad solution. The moment that the notebook document leaves the current server, e.g. shared with a colleague, those parameters are essentially useless. You have to send the kernel + kernelspec.
  - In my experience doing something similar, it was better to save these details as a static (i.e. not parameterized) kernelspec.
Mike
- securing config
  - A config that user can't read, example: imagine a billing document that can be saved in settings, but should not be accessed by the user.
    - Possible solution: Make file readable only by the server user in OS, would require elevating privileges of Jupyter server, which might not be ideal.
    - Alternate solution: Running extensions in a separate container than the main server, proxying the calls.
    - Nick/Zach: Comprehensive solution to isolate access with capability based model.
Nick
- Demo of jupyter-speedscope extension to work with the speedscope profiler in the notebook