Jupyter Server Notes 2020

Zsailer commented 5 years ago

Hello everyone,

Welcome to the Jupyter Server Team!

You can find us in our Gitter chat room or visit us at our weekly meetings!

We meet on Thursdays at 8:00am, Pacific Standard Time on Jupyter's Zoom Channel. You can add yourself to the weekly agenda here. Everyone is welcome!

Let's avoid using this thread for discussion. If you'd like to discuss something in the minutes, open a separate issue and reference this thread.

Meeting Minutes

December 17th, 2020
December 10th, 2020
December 3rd, 2020
November 26th, 2020 - no meeting (US Holiday - Thanksgiving)
November 19th, 2020
November 12th, 2020
November 5th, 2020
October 29th, 2020
October 22nd, 2020
October 15th, 2020 - no meeting for JupyterCon 2020
October 8th, 2020 - no meeting for JupyterCon 2020
October 1st, 2020 - no minutes captured (sorry about that)
September 24th, 2020
September 17th, 2020
September 10th, 2020
September 3rd, 2020
August 27th, 2020
August 20th, 2020
August 13th, 2020
August 6th, 2020
July 30th, 2020
July 23rd, 2020
July 16th, 2020
July 9th, 2020
July 2nd, 2020
June 25th, 2020
June 18th, 2020
June 11th, 2020
June 4th, 2020
May 28th, 2020
May 21st, 2020
May 14th, 2020
May 7th, 2020
April 30th, 2020
April 23rd, 2020
April 16th, 2020
April 9th, 2020
April 2nd, 2020
March 26th, 2020
March 19th, 2020
March 12th, 2020
March 5th, 2020
February 27th, 2020
February 20th, 2020—meeting canceled.
February 13th, 2020
February 6th, 2020
January 30th, 2020
January 16th, 2020
January 9th, 2020
January 2nd, 2020
December 12th, 2019—no meeting.
December 5th, 2019
November 21st, 2019
November 7th, 2019
October 31st, 2019
October 24th, 2019
October 17th, 2019
October 10th, 2019

Zsailer commented 4 years ago

July 30th, 2020

Attendees

Name	affiliation	username
Zach	Jupyter Cal Poly	@Zsailer
Steve	AWS	@blink1073
Kevin	IBM	@kevin-bates
Eric	Datalayer	@echarles

Agenda + Minutes

The main focus this week: move JupyterLab to Jupyter Server:
- See this doc. It describes the list of steps needed to move JupyterLab to Jupyter Server.
- Eric has a running demo—try with this Dockerfile
Where does nbclassic go?
- Zach will open an issue on jupyterlab to discuss.

Zsailer commented 4 years ago

August 6th, 2020

Attendees

Name	affiliation	username
Zach	Jupyter Cal Poly	@Zsailer
Steve	AWS	@blink1073
Kevin	IBM	@kevin-bates
Eric	Datalayer	@echarles
Darian	Two Sigma	@afshin
Vidar	JPMorgan Chase	@vidartf

Agenda + Minutes

Zach - Jupyter Server merged into JupyterLab
- nbclassic 1.0rc released
- jupyter_server 1.0rc release
- jupyterlab 3.0.0a4 released (depends on jupyter server + nbclassic).
- To-do:
  - Jupyter Packaging issues.
    - jupyter/jupyter-packaging#39
    - jupyter/jupyter-packaging$40
    - Fixed in jupyter/jupyter-packaging#41
  - Update documentation for Jupyter Server v1.0
  - Review JupyterHub PR for Jupyter Server
    - jupyterhub/jupyterhub#3128

Zsailer commented 4 years ago

August 13th, 2020

Attendees

Name	affiliation	username
Zach	Jupyter Cal Poly	@Zsailer
Kevin	IBM	@kevin-bates
Darian	Two Sigma	@afshin

Agenda + Minutes

Zach - A few minor fixes
- Allow extensions to log directly: https://github.com/jupyter/jupyter_server/pull/274
- Logging issues in JupyterLab fixed: https://github.com/jupyterlab/jupyterlab_server/pull/115
- Make a Jupyter Server Extension "Sanity" check:
  - Separate repo
  - GitHub Actions testing a list of server extensions against JS.
  - Instructions for extension developers to add their extension for testing in this repo.
Kevin - Dusting off kernel providers (slowly)

Zsailer commented 4 years ago

August 20th, 2020

Attendees

Name	affiliation	username
Zach	Jupyter Cal Poly	@Zsailer
Steve	AWS	@blink1073
A. T. Darian	Two Sigma	@afshin
Eric Charles	Datalayer	@echarles

Agenda + Minutes

Zach
- Jupyter Server extension checker repository.
  - Extension authors can add their server extension to the list and the checker will verify that server "sees" their extension.
  - https://github.com/Zsailer/jpserver-extension-check
- Transferred nbclassic to JupyterLab org :tada:
Eric
- opened an issue on JupyterLab with side-effects from the jupyter_server migration
  - https://github.com/jupyterlab/jupyterlab/issues/8856
  - favicon.ico is not in root static directory
    - https://github.com/jupyter/jupyter_server/issues/279
  - Ensure Base URL configuration is working
    - https://github.com/jupyter/jupyter_server/issues/278

Zsailer commented 4 years ago

August 27th, 2020

Attendees

Name	affiliation	username
Zach	Jupyter Cal Poly	@Zsailer
Eric	Datalayer	@echarles
Kevin	IBM	@kevin-bates
Steve	AWS	@blink1073
A. T. Darian	Two Sigma	@afshin

Agenda + Minutes

Zach
- Mostly been addressing jupyter_server + jupyterlab issues
- Opened PR on jupyter/telemetry
  - Fine grained data collection: https://github.com/jupyter/telemetry/pull/49
Eric
- Docs review before 1.0.0 https://github.com/jupyter/jupyter_server/issues/288 (security page?)
- Support for requirement like rtc graphql https://github.com/jupyterlab/rtc/pull/73
- Will try for next week Kernel Provider https://github.com/jupyter/jupyter_server/pull/112
Kevin
- Dusting off Kernel Providers
  - PR to drop 3.5 support, add copyrights, etc.
  - Plan to add type hints
  - Dependent on Thomas - still in his "org"
Darian
- i18n question - Should we bring internationalization endpoints into the jupyter_server instead of requiring upstream applications to support them? As a group we think "yes". Will revisit this post 1.0

Notebook Review

Alignment Issue in notebook
- https://github.com/jupyter/notebook/issues/5687
- Need to revert this the change in: https://github.com/jupyter/notebook/pull/3173
- Then make a patch release.
Documentation on Trouble-shooting:
- https://github.com/jupyter/notebook/pull/5695

Zsailer commented 4 years ago

September 3rd, 2020

Attendees

Name	affiliation	username
Zach	Jupyter Cal Poly	@Zsailer
Steve	AWS	@blink1073
Eric	Datalayer	@echarles
Kevin	IBM	@kevin-bates
Vidar	JPMChase	@vidartf

Agenda + Minutes

[name=Zach]
- Update to work with traitlets 5.0.0 (see PR 294)
- Dropped support for Python 3.5
- Working with Steve to make JupyterLab run on both jupyter_server + notebook server
  - Reason: https://github.com/jupyterhub/repo2docker/issues/945
  - The fix: https://github.com/Zsailer/jupyterlab/blob/notebook-extension/jupyterlab/serverextension.py
  - Still need to changes in jupyterlab_server to fix.
[name=Steve]
- Integrate jupyter-archive - should we make archives optional so that not all contents managers have to implement?
  - We should use the same method type as Upload for extract-directories (PUT)
  - We should make the new traitlets optional and give a (405 Method Not Allowed if they don't exist
- Pass-through for render_template?
[name=Eric]
- Minor Docs fixes to prepare 1.0.0 https://github.com/jupyter/jupyter_server/pull/299
- WIP Example for JupyterLab with Kernel Provider on Kubernetes https://github.com/jupyter/jupyter_server/pull/300
  - With JupyterLab/Server master: KERNEL_ID must be set for discovery and lifecycle management!
  - With Notebook experiment:
```
File "/opt/conda/lib/python3.7/site-packages/notebook/services/kernels/kernelmanager.py", line 44, in __init__  self.connection_info, self.manager = kernel_finder.launch(kernel_type)

TypeError: cannot unpack non-iterable coroutine object
```
  - Ideas to run JupyterLab out-of-k8s-cluster?

kevin-bates commented 4 years ago

September 10th, 2020

Attendees

Name	affiliation	username
Zach	Jupyter Cal Poly	@Zsailer
Kevin	IBM	@kevin-bates
Eric	Datalayer	@echarles
Luciano	IBM	@lresende
Steve	AWS	@blink1073
Darian	Two Sigma	@afshin
Vidar	JP Morgan Chase	@vidartf

Agenda + Minutes

[name=Zach]
- Need to build a ChangeLog for 1.0
- Need to update the docs with changes.
- Fixes for binder and and allowing JupyterLab to run as a classic notebook extension
- Jupyter Telemetry Update
[name=Steve]
- Jupyter Archive PR feedback
[name=Eric]
- Kernel Provider
- Worked with Kevin on https://github.com/jupyter/jupyter_server/pull/300 -> Working POC for JLab on K8S with Kernel Provider
- How to move forward the Kernel Provider?
  - JEP Proposal for new Kernel Providers Mechanism
  - 1 missing vote?
  - next step to submit at the board?
  - Repos organisations? Should we have a Jupyter Server Org?
  - Needed code today is located in
    - https://github.com/takluyver/jupyter_protocol
    - https://github.com/takluyver/jupyter_kernel_mgmt
    - https://github.com/jupyter/enterprise_gateway (for remote kernels and temporary)
    - https://github.com/gateway-experiments * (for remote kernels)
  - Some duplicates between https://github.com/gateway-experiments and https://github.com/jupyter/enterprise_gateway
- Performance
- Progressing on client-side for JupyterLab
- Room for improvement at server-side
  - eg HTTP2 for Large file loading and cache at client
  - Tornado limits https://github.com/tornadoweb/tornado/issues/1438
Notebook
- Updates to an endpoint: https://github.com/jupyter/enhancement-proposals/issues/64

kevin-bates commented 4 years ago

September 17th, 2020

Attendees

Name	affiliation	username
Kevin	IBM	@kevin-bates
Eric	Datalayer	@echarles
Darian	Two Sigma	@afshin

Agenda + Minutes

[name=Kevin]
- Working on the 1.0 release Friday (9/18)
[name=Eric]
- Doc for 1.0 PR updated https://github.com/jupyter/jupyter_server/pull/299
- Telemetry status?
- Provider frameworks?
- https://github.com/jupyter-server reserved

kevin-bates commented 4 years ago

September 24th, 2020

Attendees

Name	affiliation	username
Kevin	IBM	@kevin-bates
Eric	Datalayer	@echarles
Darian	Two Sigma	@afshin
Vidar	JPMChase	@vidartf

Agenda + Minutes

[name=Kevin]
- More Kernel Provider concerns - "Notebook file" compatibility with other apps (not using KP)

kevin-bates commented 4 years ago

October 22nd, 2020

Attendees

Name	affiliation	username
Kevin Bates	IBM	@kevin-bates
Eric Charles	Datalayer	@echarles
Vidar T Fauske	JPMorgan Chase	@vidartf
Afshin Darian	Two Sigma	@afshin
Gonzalo Peña-Castellanos	Quansight	@goanpeca

Agenda + Minutes

Update the Zoom Link in the README, meeting issue, and meeting notes.
Discussed whether to proceed with weekly meetings for the time being and decided to continue holding the weekly meeting. Kevin will be present, record minutes, and update the corresponding repo issue.
What is the next step after the closing of Kernel Provider JEP?
- We discussed some alternatives, like abstracting the Popen layer so authors could plugin their own functionality to manage the kernel process in, say, Kubernetes, Hadoop YARN, etc.
- Discovery may be a harder issue to resolve. One idea brought up would be to introduce the idea of addtional KernelSpecManagers that jupyter_client would then call upon to produce the set of avaiable kernelspecs.
- We talked about the need to embrace the kernelspec fomat used today and try to extend that (as is the case today).
Gonzalo brought up the idea of moving the localization framework currently in jupyterlab_server into jupyter_server. The team agreed this sounded like a good idea. There may be some concern relative to nbclassic localization, but that shouldn't deter this progress.

kevin-bates commented 4 years ago

October 29th, 2020

Attendees

Name	affiliation	username
Kevin Bates	IBM	@kevin-bates
Zach Sailer	Apple	@Zsailer
Josh Hamet	Twitter	@jhamet93
Mariko Wakabayashi	Twitter	@mwakaba2

Agenda + Minutes

async contents managers
- add AsyncContentsManager (et all) with methods like:
```
async def open():
# loop = get curret loop
res = await loop.run_in_executor(...super().open())
```
- we'll need to play some inheritance/mixin games for derived classes like we did for async kernel management and the superclass pinning.
- minutes: Mariko and Josh would like to help contribute async support into the Contents Manager hierarchy. The initial approach will be to open a DRAFT PR with some initial work (probably based on the aforementioned kernel manager approach) and we can review/iterate from there. Thanks to Mariko and Josh for volunteering to implement this useful contribution!
jupyter-server Github organization.
- Before moving repos, we'll announce this org in JupyterLab meeting next week to confirm that there is no issues.
- Projects that could move:
pytest fixture namespace issues
- prefix to use: jp_, jpserver_, ?
- move pytest_plugins.py to its own package: pytest-jupyter-server?
  - Created https://github.com/jupyter-server/pytest-jupyter
  - This could be a home for other pytest fixtures that jupyter libraries/extensions use.

kevin-bates commented 4 years ago

November 5th, 2020

Attendees

Name	affiliation	username
Kevin Bates	IBM	@kevin-bates
Zach Sailer	Apple	@Zsailer
Gonzalo Peña-Castellanos	Quansight	@goanpeca
Eric Charles	Datalayer	@echarles

Agenda + Minutes

[Zach] Discourse post announcing Jupyter Server
- Added documentation for migrating server extensions.
  - https://jupyter-server.readthedocs.io/en/latest/developers/extensions.html#migrating-an-extension-to-use-jupyter-server
- Move jupyter_server to jupyter-server github org.
[Kevin] pytest-jupyter repo - https://github.com/jupyter/jupyter_server/issues/322
- add docs to pytest-jupyter
- release pytest-jupyter (plugins: jupyter_core, jupyter_server)
- update lab tests with jp_ fixture prefix
- remove jupyter_server/pytest-plugin.py
[Eric] We have discussed in last Jupyter RTC meeting to add Authentication and Authorization in Jupyter Server core
- It was suggested that we start sketching out ideas surrounding "identity management" and assess what kinds of overlap we might have with Hub's authentication/authorization capabilities, with the idea that perhaps those be shared and maintained separately.
- Eric brought up that Hub's current authoriziation model is essentially "admin"/"not admin" at that we'd want to introduce finer-grained support.

willingc commented 4 years ago

@kevin-bates @Zsailer what is the rationale for moving jupyter-server to its own org?

If this does happen, please announce beyond JupyterLab to all orgs including nteract.

Zsailer commented 4 years ago

Hi @willingc 😄

My apologies for not being clear in the notes above. We aren't moving the repo this week. Instead, we're drafting a discourse post to:

announce the first major release of Jupyter Server
propose to move jupyter_server to this new org.

Discourse will (hopefully) reach a broad audience (including nteract).

In short I shared this idea in the JupyterLab meeting and received the same feedback you gave—we need to announce this change more broadly so that other teams are aware of this change. We settled on posting to discourse. I'm working on that post now 😃 Does that seem reasonable to you?

what is the rationale for moving jupyter-server to its own org?

I think there are a few advantages to making jupyter-server its own org:

Team-compass: perhaps the most compelling reason is that jupyter_server (and related projects) has gained enough momentum (regular meetings, multiple active team members, active discussion on future ideas+maintenance, etc.) to create a team-compass repo for team management. Organizing the team in this manner would greatly benefit those involved.
Reduce friction for server-side sub projects: we have worked / are working on subprojects related to jupyter_server, but we don't have a "home" for these projects. Moving them into the jupyter github org isn't plausible, but leaving them under personal namespaces doesn't make sense either. Centralizing these experimental projects and having a team-compass page to track their discussion would be helpful.
Governance: the current direction of the new Jupyter governance model suggests that subprojects will be given a "vote" in ecosystem-wide decisions. I think jupyter_server is emerging as a key component in the ecosystem and might be given one of these votes (?). Before that happens, I think it would be useful to have a home for open+clear+transparent discussion in place. My goal would be that the jupyter-server org/team provide high quality documentation, messaging, and regular updates about all things happening around server.

I'd love feedback on these ideas! We're doing our best to keep the Jupyter community up-to-date on jupyter_server's roadmap/future, though this can be challenging as we all know :)

willingc commented 4 years ago

Thanks @Zsailer. Was great to see you the other morning at the RTC meeting.

Zsailer commented 4 years ago

:tada: Here's the discourse post announcing Jupyter Server and jupyter-server org :tada: https://discourse.jupyter.org/t/jupyter-server-1-0-and-the-new-jupyter-server-github-org/6657

meeseeksmachine commented 4 years ago

This issue has been mentioned on Jupyter Community Forum. There might be relevant details there:

https://discourse.jupyter.org/t/jupyter-server-1-0-and-the-new-jupyter-server-github-org/6657/1

kevin-bates commented 4 years ago

November 12th, 2020

Attendees

Name	affiliation	username
Kevin Bates	IBM	@kevin-bates
Zach Sailer	Apple	@Zsailer
Steve Silvester	Apple	@blink1073
A. T. Darian	Two Sigma	@afshin
Vidar T Fauske	JPMorgan Chase	@vidartf
Eric Charles	Datalayer	@echarles
Gonzalo Peña-C	Quansight	@goanpeca

Agenda + Minutes

[Kevin] pytest fixture update
- pytest-jupyter (plugins: jupyter_core, jupyter_server) released with docs - thanks Zach!
- TODO: update lab/elyra tests with jp_ fixture prefix
- Remove jupyter_server/pytest-plugin.py and entrypoint definition
  - Do we need an interim server release prior to removal? (Seems users could pin to 1.0.5)
Should we deprecate quit_button? Is this an overlooked remnant from notebook?
- Kevin will open an issue with options of whether to deprecate the current trait, rename to to something that has some code in the server that controls whether the server can shutdown via the client, etc. (see jupyter-server/jupyter_server#336)
Jupyter Server v1 was announced on Discourse last week.
- Planning to move the jupyter_server repo into the new jupyter-server org this week.
[Eric] Looking at Add authorization layer to server request handlers https://github.com/jupyter/jupyter_server/pull/165 and add
- Pluggable user token creation/validation https://github.com/jupyter/jupyter_server/issues/50
- Add a Session Management infrastructure for extension developers https://github.com/jupyter/jupyter_server/issues/122

Zsailer commented 4 years ago

November 19th, 2020

Attendees

Name	affiliation	username
Kevin Bates	IBM	@kevin-bates
Zach Sailer	Apple	@Zsailer
Jeremy Tuloup	QuantStack	@jtpio
A. T. Darian	Two Sigma	@afshin

Agenda + Minutes

[Kevin]
- PRs released to prefix fixtures and removed previous plugin and entrypoint (Thanks to Nick Bollweg - @bollwvyl!). Release 1.0.6 (thanks Zach) can be used by apps using the non-prefixed fixtures.
- Release 1.1 will be coming out soon. We need to get a patch into lab's test framework for prefixing the server-related fixtures prior to the release
- I will be looking at the async contents PR (https://github.com/jupyter/jupyter_server/pull/324)
  - Would be good if others could check this out as well
- Ported a couple more Notebook PRs, plenty to go still
- I will be out starting 12/2 through 1/3 - Happy New Year - good riddance 2020!
[Jeremy]
- Should nbresuse move to jupyter-server?
  - Yes. We will move to the jupyter-server org with a long-term goal of baking into the server and "closer" to kernels
[Zach]
- Added steering council members as managers of jupyter-server org.
- Move Jupyter Server to new org in this meeting, today!
- Done! :tada:

kevin-bates commented 3 years ago

December 3rd, 2020

Attendees

Name	affiliation	username
Pierre-Olivier Simonard	ioapps.io / Quansight	@pierrotsmnrd
Zach Sailer	Apple	@Zsailer
Kevin Bates	IBM	@kevin-bates
Mariko Wakabayashi		@mwakaba2

Agenda + Minutes

Async ContentsManager PR #324
- needs a bit more testing before review.
- will work on documentation for how to use
Extension discovery for Python-based extensions Issue #351
- Group was generally on board with using entrypoints as a discovery mechanism.
- Would solve our issues with dev mode + data_files install.
- Currently, data_files don't get installed in dev mode by default in setuptools. Jupyter packaging tries to address this issue. It moves data_files properly on dev install; however, it doesn't remove them when dev mode is uninstalled.
Move nbclassic to jupyter-server org? Yes. Will mention in next week's JupyterLab meeting.

Zsailer commented 3 years ago

December 10th, 2020

Attendees

Name	affiliation	username
Zach Sailer	Apple	@Zsailer
Kevin Bates	IBM	@kevin-bates

Agenda + Minutes

Backport #340 to 1.0.x branch.
- This was missed in the backport of #350.
- PR https://github.com/jupyter-server/jupyter_server/pull/359
- Release a v1.0.10 with this PR.
Move pytest_plugin back into Jupyter Server from pytest-jupyter.
- Open a PR in jupyter_core with the fixtures that are jupyter_core specific.
- We don't need to wait for the jupyter_core side. We'll include a duplicate of those fixtures in Jupyter Server temporarily, then remove later.
Plan to release 1.1 Friday.
- Merge Async Contents changes (#324) today.
- Merge pytest_plugin work today.
- Release 1.1 by tomorrow, EOD (pacific).
- Bump jupyter_server version in JupyterLab and JupyterLab Server.

Zsailer commented 3 years ago

:tada: Next Thursday is our last meeting of the year! 🎉 (After that, it's Christmas Eve 🎅 and New Year's Eve).

Wear your ugly holiday sweater is you have one! :wink:

Zsailer commented 3 years ago

December 17th, 2020

Attendees

Name	affiliation	username
Zach Sailer	Apple	@Zsailer
A. T. Darian	Two Sigma	@afshin
Jeremy Tuloup	QuantStack	@jtpio
Kevin Bates	IBM	@kevin-bates
Vidar T Fauske	JP Morgan	@vidartf

Agenda + Minutes

Deprecated the pytest-jupyter plugin.
- Jupyter Server has its own pytest_plugin now.
- Documentation on this plugin will be added to the jupyter_server docs.
- Pytest-jupyter might become a "best practices" page for pytest around Jupyter.
Released 1.1.1 patch yesterday.
- Security Advisory Published.
- Minor fix in Async Contents API: https://github.com/jupyter-server/jupyter_server/pull/363
Notebook server and jupyter server extensions in one dist? https://github.com/jupyter/nbdime/pull/551
- Server extensions should be both forwards and backwards compatible.
  - Jupyter Server's new JupyterHandler currently works with classic notebook server.
  - Notebook's IPythonHandler works with Jupyter Server.
Min's student, Kien, is working Telemetry in Jupyter Server.
- Concerns about posting to /api/endpoint.
- Posting events to the server without properly checking who "owns" the event seems like a problem.
Published jupyter-resource-usage to PyPI
- Invited a couple of people as maintainers / owners
- Next: conda-forge
- Jeremy has another resource usage extension, https://github.com/jtpio/jupyterlab-system-monitor.
  - Move this into jupyter-resource-usage.
  - Deprecate th

Zsailer commented 3 years ago

I've opened a new issue for meeting notes in 2021. Let's leave this issue open through January 2021, then close it.

jupyter-server / team-compass