search

jupyter / enhancement-proposals

Enhancement proposals for the Jupyter Ecosystem
https://jupyter.org/enhancement-proposals
BSD 3-Clause "New" or "Revised" License
115 stars 65 forks source link

Jupyter Notebook version 7 #79

Closed fperez closed 2 years ago

fperez commented 2 years ago

Summary

This PR proposes a path forward for the user experience of the jupyter notebook application to continue to be available to users, with the upcoming version 7 based on the JupyterLab TypeScript codebase and APIs. Version 7 of the notebook will use what is today RetroLab, with suitable repo restructuring as detailed in the JEP.

In #78 I created the issue, but as indicated there, I'm moving directly to making a PR, given the extensive amount of community discussion this topic has already received. This PR should be where specifics of the proposal itself and Notebook v7 plan are discussed. Any meta/process issues can be dealt with in #78, if need be.

References

For reference, these are the main issues where this topic had been previously discussed:

In addition, see the Enhancement Proposal guidelines for information about the goals of the process itself.

Discussion points

✔️ = resolved ❌ = unresolved

Voting from the @jupyter/steeringcouncil

fperez commented 2 years ago

Pinging co-authors Sylvain Corlay (@SylvainCorlay), Afshin Darian (@afshin), Sharan Foga (@sharanf), Kevin Goldsmith (@KevinGoldsmith), Brian Granger (@ellisonbg) , Jason Grout (@jasongrout), Zach Sailer (@Zsailer), Jeremy Tuloup (@jtpio) that the proposal is ready!

Folks - I reordered all names alphabetically, but kept myself first as I didn't want to pop one of you "in front" (that would be @SylvainCorlay given the current names we have, unless a new co-author comes in before Corlay) without your authorization. But I'm making zero claims of authorship here, and I'm perfectly happy moving my name into the P slot as long as everyone, and particularly --at least for now-- Sylvain, are OK with being in front :)

fperez commented 2 years ago

Note that I edited the PR title to reflect we should allow for some public input in this full version before moving on to a vote. While we've had a lot of discussion that I think warrants putting it here, the finished document only came together this morning, and the community should have a chance to mull it over and provide input so we have a solid plan moving forward with broad support.

choldgraf commented 2 years ago

Hey all - just to make it explicit, and to match the title, I've converted this to a "DRAFT" in github's UI as well

afshin commented 2 years ago

I reordered all names alphabetically, but kept myself first ...

I am happy with the way you have it right now. Thanks, Fernando!

meeseeksmachine commented 2 years ago

This pull request has been mentioned on Jupyter Community Forum. There might be relevant details there:

https://discourse.jupyter.org/t/jep-draft-open-for-the-notebook-v7-transition/11769/1

fperez commented 2 years ago

I would like to suggest @choldgraf as shepherd for this JEP - Chris has worked a lot on the JEP process itself, has enough knowledge about the notebook and our community, but is reasonably removed from the specific decisions to provide the right balance of knowledge and impartiality, I think. I consulted and he's OK with the idea.

[ I'd originally posted this in the top summary, but best to have it here as a standalone comment for the record/discussion, as we may update the description in-place ]

choldgraf commented 2 years ago

I am happy and willing to steward this JEP, as long as others are 👍 as well! If there are no objections then I'll take a closer look at the JEP language etc in the coming days.

choldgraf commented 2 years ago

I've cleaned up the top comment a little bit to create more natural sections of information. I'm also going to try an experiment and provide links to major ongoing discussions, to make it easier for people to follow where the major discussion points are.

choldgraf commented 2 years ago

~I thought it would be useful to make this JEP follow the standard template in the enhancement proposals guidelines. Here's a PR that rearranges the content here to match what is in the template: https://github.com/fperez/enhancement-proposals/pull/3~

~If folks don't think that is a helpful contribution, feel free to close the PR, but I wanted to give it a quick shot to make sure that the major questions we suggest answering in JEPs are answered.~

After some reflection, I don't think that it's a productive use of people's time to look at that PR, and instead we should focus our efforts on this JEP's language instead. Sorry for the noise y'all :-)

choldgraf commented 2 years ago

Proposal

Given that we've already had a lot of conversation in the threads linked in the top comment, and a few iterations here, I suggest that we try to move this forward sooner than later.

Unless there are objections, I suggest we do the following:

afshin commented 2 years ago

I am 👍 on your plan, Chris.

choldgraf commented 2 years ago

Hey all - I believe that all major outstanding comment threads are resolved. I suggest that the JEP authors make any final updates to the language, or resolve any questions you think need resolving, and then mark the JEP as "ready for review". This will trigger the voting phase, at which point the language should not substantially change (if it does, we will need to nullify votes and start over).

ellisonbg commented 2 years ago

I have made a suggestion with one additional users story that is focused on the extension author persona and their experience of supporting extensions in both Notebook 7 and Lab 4.

fperez commented 2 years ago

@choldgraf I made a tiny metadata update and now marked it officially as 79 in the folder name. I think it's ready to go...

Oh wait! I just saw @ellisonbg's update - I agree with it but can't seem to push it from the UI. Weird...

fperez commented 2 years ago

I guess I don't have write permissions to make the commit. Odd... I'll do it manually on the branch now then.

fperez commented 2 years ago

@ellisonbg do you have any others? Else we can move it out of draft status...

ellisonbg commented 2 years ago

@ellisonbg do you have any others? Else we can move it out of draft status...

Nope, I think that is it, I am fine with us moving it out of draft.

fperez commented 2 years ago

Perfect, I'm pushing that change now, just figuring out the git syntax for multi-authors :)

fperez commented 2 years ago

OK 🚀! @choldgraf how do I turn off the draft flag on this? I can't seem to find the UI element you'd showed me earlier, not sure if it's a repo permissions issue...

I've removed "DRAFT" from the title though - this is ready for review.

choldgraf commented 2 years ago

It should be just above the comment box:

image

fperez commented 2 years ago

Ah, thanks! I was looking for it in up above and missed it below. Duh.

fperez commented 2 years ago

Huge thanks to everyone who made it possible to get us here. I'll post now on discourse and send a quick email to the SC (which right now we need for voting) so this moves forward promptly.

choldgraf commented 2 years ago

Hey all - I believe that the Review Team for this JEP is the @jupyter/steeringcouncil - I have added a voting list to the top of this document (I copy/pasted the same list of names from #72).

Instructions

For the @jupyter/steeringcouncil - please check the box next to your name according to how you'd like to vote on this JEP. If you wish to propose minor changes (e.g. small language or typos), that is OK. If you wish to propose changes that are significant, mark your vote as "no", as we cannot change the JEP language significantly without starting the process again.

choldgraf commented 2 years ago

Process update: Try to get comments in by 2021-12-13

I wanted to provide a process update to this JEP. The JEP guidelines suggest we use a 10 day minimum for this Final Comment Period. This date would be roughly the 13th.

We don't have to make a decision on the 13th, but I suggest that since we're had a lot of conversation already, we try to move forward soon after this date, unless there are major objections to this plan.

In addition, I have run into a few confusion points for myself in shepherding this PR, and have opened an issue to document some of these so that we can do process improvements in the future (#83)

SylvainCorlay commented 2 years ago

Quick update on the front of extension support in notebook v7:

afshin commented 2 years ago

The QuantStack team just had a quote approved to port the UI of nbgrader to the JupyterLab extension system. Work on nbgrader should start in January! 🎉

That's awesome news!

cc @jhamrick

krassowski commented 2 years ago

I fully agree that having the UI for extension manager that allows to install prebuilt extensions equivalents for each of the extensions that were available from notebook contrib should be part of the plan.

Currently the Extension Manager in JupyterLab installs source extensions which leads to a lot of issues as discussed in https://github.com/jupyterlab/jupyterlab/issues/10554. The current plan is to remove the source extensions searching from the core for 4.0 and to allow extensions to provide lists of extensions that can be installed (as well as to let them handle the installation itself).

While I fully agree that we should switch to a modular approach and drop source extensions (in favour of prebuilt extensions which are easier to install), I believe that ideally we should also include a "good enough" provider of extensions that handles the most common use cases of installation in a Python virtual environment and in a conda environment.

During the recent JupyterLab meetings there was a consensus that this is a hard problem (conda vs pip vs non-Python packages, querying PyPI API to create the list of extensions, etc), but I think it is manageable.

bollwyvl commented 2 years ago

users will also have to have an NPM environment installed... snippets

Ah, but the fine print about jupyter lab build: If you use JupyterLab 2.x,

It's probably worth re-iterating:

No end user feature of Notebook 7 will require nodejs or any other non-pip installable package

I think by the time Notebook 7 lands (some time after Lab 4), all important extensions where the developers intended to supporting them at all will already offer one-step, pip installable pre-built extensions.

They want a single step or command for installing extensions.

As with nbextensions in Notebook <7, end users will be able to pip install or conda install new web UI features, requiring a browser reload to use new UI features.

There were certain ways this could be made to work well in Notebook <7 without a page reload (e.g. widgets), but we could never offer sound compatibility guarantees for everything: just slamming some JS out of a kernel into an output and expecting a new top-level menu item to show up is not a capability we can maintainably offer in Notebook 7.

Server extensions (e.g. new REST APIs) will continue to require a restart of the notebook server... but this has always been a system property of the notebook server. This could theoretically be fixed at an architectural level... but has the implicit potential, especially in a managed/Hub setting, of leaving a user with a completely wedged, unstarteable, uninspectable, remote web server, which has always prevented anyone from pushing too much harder on it.

we should also include a "good enough" provider of extensions

Yeah, it would be nice... but as there never was an in-app "Extension Store," in Notebook <7, I don't think, it's in-scope for this JEP.

gutow commented 2 years ago

It's probably worth re-iterating:

No end user feature of Notebook 7 will require nodejs or any other non-pip installable package

I think by the time Notebook 7 lands (some time after Lab 4), all important extensions where the developers intended to supporting them at all will already offer one-step, pip installable pre-built extensions.

This is good. I think that would work with my end users. Are there clear examples I can follow that do this using menus that inject code into selected cells (the key thing I do)? My testing with the examples I have found do not work as simple pip installs. I've actually had trouble getting some of them running at all. Maybe, this reflects things changing "under-the-hood" since they were last updated? Is the API still changing a lot?

There were certain ways this could be made to work well in Notebook <7 without a page reload (e.g. widgets), but we could never offer sound compatibility guarantees for everything: just slamming some JS out of a kernel into an output and expecting a new top-level menu item to show up is not a capability we can maintainably offer in Notebook 7.

I do not wish to just "slam some JS out of the kernel". However, there are many use cases where the menus available are context sensitive (in my case some examples might be: am I collecting data using a hardware A-to-D versus processing data versus creating a worksheet for students). Thus, there needs to be clearly documented ways to control what menus and buttons show via python (ideally, also any other language running in the kernel), because a specific kernel is running, and also through the settings menu. If the API is stable this should be doable, IMHO.

bollwyvl commented 2 years ago

Are there clear examples I can follow that do this using menus that inject code into selected cells (the key thing I do)?

The snippets example you linked to has most of the pieces. Similarly, the dask-labextension shows some approaches for working with the context menu.

trouble getting some of them running at all

without any more info, it's hard to say... but the venue for further discussion is likely issues on the extensions themselves, where it could prove out no, this developer does not intend to support this extension (which is of course their choice).

ways to control what menus and buttons show via python

At present, there isn't a way for (any) kernel to know some "client name" against which they are running (it could be multiple, simultaneously). Again, this could be made more lax, but brings a host of other deep issues, and is outside the scope of this JEP. But basically, kernel-emitted JS code that only works in one client (e.f. Notebook <7) probably already didn't work in any other first-party (lab, nbconvert) or third-party client (nteract, vscode, spyder, colab). Part of the maintainability issue we are addressing here is precisely this... perhaps widgets or mimetypes for these kinds of things are the way to move forward. had a comment about that earlier, but it appears to be gone.

choldgraf commented 2 years ago

Hey all - we're now outside the "final comment period" minimum window. I'll note that several @jupyter/steeringcouncil members haven't given their thoughts or a vote yet. Could somebody send another message to that group (or tell me how I can best get in touch with the group) to let them know we'd like final thoughts and comments before moving forward?

Also, after reading the conversation above, between @gutow and @bollwyvl , I don't believe that this requires changes to the JEP language itself, since we're all in agreement already that the most important extensions should be pip-installable alone by the time NTBKv7 comes out. Do you agree, or should the language be updated?

gutow commented 2 years ago

ways to control what menus and buttons show via python

At present, there isn't a way for (any) kernel to know some "client name" against which they are running (it could be multiple, simultaneously). Again, this could be made more lax, but brings a host of other deep issues, and is outside the scope of this JEP. But basically, kernel-emitted JS code that only works in one client (e.f. Notebook <7) probably already didn't work in any other first-party (lab, nbconvert) or third-party client (nteract, vscode, spyder, colab). Part of the maintainability issue we are addressing here is precisely this... perhaps widgets or mimetypes for these kinds of things are the way to move forward. had a comment about that earlier, but it appears to be gone.

I understand this to mean that I would have to have my students launch different versions of JLab depending on their use case. At that point, might I be better off writing dedicated applications using QT or wx? However, I do not really want to have to duplicate the very useful simple notebook interface. The other alternative is to have a single heavy weight version of JLab with lots of menus, buttons, etc. that they don't need most of the time. In that case, I would be producing a specialized version of JLab that would have many of the problems for my use case that this JEP is trying to find a path to avoid.

I am also confused by "there isn't a way for (any) kernel to know some "client name" against which they are running." One should be able to look at either existing DOM elements or interfaces to determine what is appropriate. If the necessary information is completely hidden or obfuscated, then I believe you have lost one of the key advantages of the classic notebook.

I have no objection to having to maintain two different javascript codes during the transition from the old notebook to a JLab based one. There just needs to be a clear way to do that. I am struggling with getting what were relatively simple things to do in the old notebook working in JLab. I believe it is primarily due to the opacity of the API and lack of good examples. When I say opacity, I mean it is often hard to tell without stepping back through hundreds of lines of code exactly what options, variables, etc to pass to the high level API calls. I think a concerted effort on documentation and examples would go a long way to improving this situation.

I am strongly in favor of finding a path forward. I am just worried that as mentioned by @choldgraf and @Carreau that too much of the capabilities available when working within the Classic Notebook will be lost for many of us to effectively make the transition. I personally would like to see more language concerning support for more of the on-the-fly flexibility of the classic notebook. I do understand that that will be less compatible with things outside of JLab and has some security issues. From my perspective the ease of producing something I can use for 5 - 10 years with less development overhead is more important.

ivanov commented 2 years ago

Could somebody send another message to that group (or tell me how I can best get in touch with the group) to let them know we'd like final thoughts and comments before moving forward?

I just sent a nudge to the group and the individuals who haven't registered their decisions here, @choldgraf

rgbkrk commented 2 years ago

Thanks for the nudge.

Zsailer commented 2 years ago

It's worth noting here that there's a live, weekly discussion happening around this JEP and the pending work in the weekly Notebook meeting. Here is a record of the notes from previous weeks: https://github.com/jupyterlab/team-compass/issues/133.

As always, these meetings are open to anyone and everyone. Feel free to join the conversation. It's always a hoot! 😄

jasongrout commented 2 years ago

Thank you very much for moving this forward!

choldgraf commented 2 years ago

Hey all - we have had a few weeks of time for comments and conversations, and we've gone through many rounds of discussion and edits (both in this PR and in the linked issues). Here's a tally of the @jupyter/steeringcouncil votes so far:

Proposal

As we have a strong majority of the steering council voting in favor of this, and have had a lot of productive conversations thus far in other issues and here, I propose the following steps:

ivanov commented 2 years ago

I was asked privately why I voted against this proposal, and my reply was deemed to contain aspects of the issues at hand that were not covered in this or some of the other threads around the proposal, so I am reposting the bulk of it here.

This isn't exhaustive but here goes. I don't agree with changing the entire project and keeping the same name, which is what notebook7 will be. I can't think of quite an equivalent thing here, but it would be like installing notepad, but getting Word that's themed to look like notepad. I guess the positive example here was Mozilla Application Suite / SeaMonkey, the re-written browser component of which is where Firefox came from, but when Firefox came out, it was a new thing. People didn't get it as an update to SeaMonkey. It would have been more responsible to make a new name for this and let people migrate to it. I also do not think JupyterLab as the code base is the thing to get behind. Additionally, I do not think there is any urgency to considering this change now - we've been in limbo on governance and decision making for the entire project for quite some time, and yet this was pushed through without wrapping up the governance refactor before prioritizing other business. I understand that it is difficult to push forward on any front with so much up in the air, but I would have preferred for us to wrap up to the new decision making process and navigated the decisions of this JEP in the new governance configuration.

Carreau commented 2 years ago

I don't agree with changing the entire project and keeping the same name, which is what notebook7 will be. I can't think of quite an equivalent thing here, but it would be like installing notepad, but getting Word that's themed to look like notepad.

While I understand this point I will point you toward GCC/EGCS. I think that if the replacement is good/close enough it is worth it to have a more cohesive community. But again this is not trying to convince you, just maybe giving you a historical example. This is also why I've strongly pushed toward not phrasing comparison WRT jupyterLab, but notebook v6.

I also do see the reason for the same name with respect to user muscle memory being to pip install notebook

blink1073 commented 2 years ago

The voting window has completed, merging as accepted, thanks everyone!

choldgraf commented 2 years ago

Just wanted to say thanks @ivanov for providing your thoughts here - I think it complements @Carreau's thoughts as well, that Notebook v7 should be thought of as "an evolution of Notebook v6" rather than "an adaptation of JupyterLab". And thanks @blink1073 for merging it in 👍

I will update the top comment with some of the major conversation points here, so that is it easier to discover in the future if people refer back to this PR thread.