[Discuss] The future of the classic notebook interface

Zsailer commented 3 years ago

The purpose of this issue is to discuss the future of the jupyter/notebook repository.

Background: The classic Jupyter notebook interface is currently maintained by a skeleton crew of developers. On the other hand, many people still use the classic interface today.

Problem: We have a widely-used piece of software with very little developer support.

Let's have an open discussion about what to do with this project moving forward.

Zsailer commented 3 years ago

I'll do my best to document some of the things discussed in our weekly Notebook meetings.

Zsailer commented 3 years ago

A major issue is that this repository still receives a steady stream of issues and PRs, even though the project is technically in maintenance mode, meaning we'll only review bug/security fixes.

Zsailer commented 3 years ago

The challenge is that most of the people currently "maintaining" this repository are not frontend developers. When issues are raises in the Javascript APIs, the team lacks the experience to review/respond to these issues. Further, the Javascript is a bit outdated, so even frontend develops don't want to engage these issues.

Zsailer commented 3 years ago

As a result, there is a heavy push from the developer community to move this project into an archived state and encourage users to switch to JupyterLab or other Jupyter frontends (e.g. nteract, retrolab, etc.)

Zsailer commented 3 years ago

An alternative proposal was to offer this project as an opportunity for another team/company/group to maintain this project moving forward. There might be some team with vested interest in making sure this project stays alive indefinitely.

There was some concern raised with this idea. Handing off the project to another (possibly not Jupyter-affiliated) team puts the Jupyter brand (?, just echoing what I've heard said here) at risk.

Zsailer commented 3 years ago

The urgency of this issue (i.e. what to do with classic notebook) comes from a growing fear/possibility that notebook will break permanently because of decaying dependencies.

Bower—the JS package manager that notebook uses—is deprecated.
Tornado—the Python web server framework notebook uses—is causing bottlenecks in performance and also struggles with supporting its large user community.

Moving the classic notebook project off of these dependencies is a massive amount of work.

Zsailer commented 3 years ago

No matter what we decide, this issue impacts a large number of users. We'll want to carefully consider how to proceed, seek out consensus amount developers and users, and provide the necessary messaging to help everyone have a clear understanding of the future.

I hope this issue helps promote this discussion.

meeseeksmachine commented 3 years ago

This issue has been mentioned on Jupyter Community Forum. There might be relevant details there:

https://discourse.jupyter.org/t/the-future-of-classic-notebook-interface/11328/1

Zsailer commented 3 years ago

Another thing I've heard in this discussion is that there are key/critical notebook extensions on which many users depend, e.g. RISE, nbgrader, pixiedust, etc.

What is the story for users who depend heavily on such extensions?

jgarte commented 3 years ago

Could a jupyter notebook interface be developed that would approach the philosophy/aesthetics of a UI similar to the SourceHut ecosystem/suite of apps?

That style of app would be more maintainable for a classic notebook in the long term with minimal dependency on fast moving js libraries. That said, I imagine this would be a ton of work to implement :)

psychemedia commented 3 years ago

Apologies in advance for a long post... This repeats a lot of things posted elsewhere, but I think I still haven't found a way to express the growing gulf I see between the experience and expectations of highly skilled developers, and people who are very naive end-users of Jupyter tools doing very, very simple tasks that appear to be little more than (and often are very little more than) the following to a seasoned developer:

for i in [1, 2, 3, 4]:
  print("hello world")

At the same time, this is not to say that we shouldn't be providing such novices with environments that are, underneath, very powerful, and scalable in terms of what you can do with them. What I try to do below is explore what notebooks made possible, who they empowered, and what we are at risk of losing.

As I've commented on this issue many times, I'll argue the position of:

an educator, using notebooks to present material to novice learners in disciplines that may only touch tangentially on computing (for example, a newly graduated K12 teacher teaching basic programming to a class of two dozen 12 year olds; a near-retirement lecturer teaching first year undergrad mechanical engineering);
educator coding experience: basic HTML, inline style/CSS, maybe some elementary (non-framework) Javascript (2 -30 lines enough to implement a basic function to do a single thing);
institutional support: essentially none; maybe some grudging hosting of a JupyterHub server.

Skill level of users: JupyterLab IDE is overkill. Classic notebook UI is at the limit of UI complexity; notebook extension toolbar buttons bewilder a significant a percentage of users and make the UI unusable.

Duty cycle for upgrading software: 5 years +. (In my institution, modules run for 5-6 years then possibly get extended for another 3+ years. They take two years to produce. The software freeze (except for minor updates once a year) is typically during the first year of production, so things then need to stay largely in place for the next 7-8 years. In general, educational materials get reused year on year on year. Computers in K12 schools may be 5 years+ old. Note that education needs are global and extend to developing nations not just Ivy League feeder High Schools and inner-city deprived ghetto schools with low student expectations and harassed teaching staff.

The original notebooks allowed people who could write basic HTML pages to write extensions; the jupyter_contrib_nbextensions package includes many extensions created very early on. Many of those packages were based around very simple basic javascript and served as a template for creating new extensions based on copying code. Looking at the same extensions now (with frameworks and promises and syntax and far-from basic Javascript), I would struggle to create my own extension based on inspection of that code. And JupyterLab extension code is an order of magnitude harder to get starter with in terms of code complexity, required knowledge of Jupyter framework/API calls, and development environment.

My skill level excludes me from updating my original classic notebook extensions. There is no internal developer support available eg for porting extensions to JupyterLab. I do not have the skill or the time to learn either js frameworks or js dev tools, let alone make sense of JupyterLab API docs and how-to example fragments that are buried in code tests. My instructional material relies on classic Jupyter notebooks with their current extensions.

My institution has large numbers of developers working in institutional IT projects and growing numbers of courses using Jupyter notebooks, but with zero central IT or developer support provided to support the development of educational end-use of Jupyter notebooks. At the moment, pretty much all "development" is carried out by academics who are not developers (why should a statistician using LaTex and R know how to develop front-end web apps?) and who are supposed to be spending their time developing subject matter material and teaching students. For them, the Jupyter notebook environment is a means to an end - a way of maximising the chance that a novice with very poor computing skills can run 3 lines of code and see the result. The notebook model means the code can be placed in instructional text and made into an interactive textbook.

In many cases, Jupyter Book + Thebe (and even better, thebe w/ a pyodide kernel) would be appropriate for supporting this sort of interactive text book use case. But it lacks the ability to persist code changes or add and persist new cells against a user account.

In other cases, the more general authoring environment that classic notebook allows users to populate their own notebook (a digital version of a kid's paper school exercise book that they write their homework in), with text, media assets, maybe some code and code outputs (with a total single lines of code count across the whole notebook in the low tens at most). A single notebook might take up a complete K12 school lesson (35 minutes) and be the equivalent of a page or two pages of A4 paper.

My fear is that this large body of legacy users, who you are unlikely to see on Github, are going to be left in limbo (why would a K12 maths teacher in a rural school who has a dozen or so notebooks used over a whole school term to run some simple ipyturtle activities, need a Github account?). I think we saw some signs of this when a recent classic notebook update clobbered inline HTML styles in markdown cells.

Classic notebooks provided folk who don't have access to webservers, and don't need the grief of installing or supporting even basic, default Python environments, with a read-write-render-execute medium for HTML, rich media (embedded images, videos, audio), and REPL based code execution and code output previewing. Teachers of 12 year olds could get instructional material they'd written in front of kids, and the kids could run the code, write their own, write their own notes, and save a copy of it.

That's what drove my adoption. Classic notebook was an on-ramp. If I came to Jupyter now, and was exposed to JupyterLab as the first experience, I'd probably walk away because I'd stand zero chance of persuading anyone else it was something we'd be able to use with very novice distance education students. (It's been hard enough selling classic notebooks.)

Another fear is that a company like Pearson will pick up the maintenance of classic notebook and then lock users in. A regret is that institutions such as my own (which contributes to Moodle core development, for example) do not appear to value the benefit of putting support into classic notebook, not in terms of improving accessibility, nor providing maintenance effort, nor helping turn educator contributed have-a-go extensions into production stable extensions that can be of general effort.

The Jupyter project is a phenomenal undertaking, but it's evolution is starting to mirror that of the web. A couple of decades a go, you could see something interesting on a web page, view source, and stand a good chance of being able to appropriate the HTML+CSS code you saw and do something interesting. Classic notebook provided a UI that supported that level of engagement. But it's not what developers who want to push things forward are interested in. We need to find a way of resourcing some developers who want to push things back to as simple an environment as possible, that still affords the power of working with other Jupyter components.

IMVHO!;-)

stefanv commented 3 years ago

Bower—the JS package manager that notebook uses—is deprecated.

Tornado—the Python web server framework notebook uses—is causing bottlenecks in performance and also struggles with supporting its large user community.

Moving the classic notebook project off of these dependencies is a massive amount of work.

Bower is a problem, but it seems like it should not be too hard to port that over to webpack or similar. Tornado, however, has had frequent releases (at least until October 2020) and can handle the load, so I'd like to understand that concern better.

gutow commented 3 years ago

I agree with @psychemedia and would like to add some more to their arguments. First, I to am not primarily a programmer/developer. I am an active researcher in the physical sciences (chemistry) and teach chemistry at the undergraduate level. As such my computer use and coding is directly applied to solving experimental and educational problems. For me that means finding the most efficient use of my time to solve the problem at hand. Thus, like @psychemedia, I am not pleased with the large additional coding overhead and abstraction required to produce extensions for JupyterLab. Additionally, for educational purposes simpler interfaces are better. JupyterLab is an attempt to build an IDE inside of a web browser, while I need an easy way to process data in an ordered fashion that keeps an accurate record of what was done. The classic notebook excels at this. In theory JupyterLab could support an interface that runs more like the classic notebook, but even if a classic interface is developed I will not be able to use it for most of what I do because of the overhead/difficulty of extending JLab.

In my opinion before we give up on the classic notebook a number of things need to happen:

There will need to be a working interface that behaves like the classic notebook (or is at least simple in the way of the Google Colabratory). Attempts so far work poorly, for example retrolab.
There need to be clear and simple paths to adding specialized menus and widgets to JLab. They should not depend on setting up a development environment that depends on the lots of different tools. I understand that there are clear reasons that developers think some of the newer javascript tools are useful because of scope isolation and things like that. But currently the documentation for how to use the tools are poor at best. As an example jQuery is old (and many would say cranky), but you can go to the jQuery documentation and easily figure out how to implement jQuery based interactive elements. As an example, I tried to figure out how to make an interactive dialog with some custom inputs and other interactive elements inside JLab. I could not figure it out. I got it working in classic Jupyter notebook by utilizing the underlying well documented jQuery. It isn't elegant, but it works and took me less time than I spent discovering I could not figure out how to do it in JLab.
There needs to be more collaboration between the coders/developers and what I will call middle users, such as myself and @psychemedia. I have lots of good applied projects that need work and could benefit from more coding expertise and time than I have.

The problem I see with JupyterLab is that the vast majority of people working on it are developers/programmers. So they are developing a tool to do what they do. Many of us need tools to do other things and the classic notebook is better at those other things.

The very best software I have used over my 40+ years was either written by the people who use it or a team consisting of the people who use the software and collaborating expert programmers.

Here's a list of some of the very best pieces of software I have used: IgorPro (commercial, by Wavemetrics, lab scientists writing tools for lab scientists); Gamess (opensource quantum chemistry); VMS (Vax operating system, still available as openVMS); Linux/Unix; Python (I hate the syntax, but its ease of use and facile sharing of modules makes it a winner), Jupyter notebook, Jmol (opensource), Plone (opensource).

JupyterLab does not make this list.

If people are interested in some of the projects I am working on that benefit from the classic notebook, see Jupyter Physical Science Lab.

willingc commented 3 years ago

I completely agree with @psychemedia. In the early IPython notebook days, I saw how the simple, intuitive classic notebook helped make coding more accessible for young coders, people who were returning to the workforce, and people who had had horrible first coding experiences (i.e. women in their 20s and 30s who were told by arrogant teachers that they couldn't program because they weren't good in math and science).

Much like the early Macs and iPod, the classic Notebook is an easy to learn, approachable, and intuitive tool (after teaching folks Shift+Enter). It's also the user interface that drove the success of JupyterHub and Binder.

While JupyterLab serves advanced users and Jeremy's notebook based on JupyterLab is a nice project, the classic notebook appeals to a large group of users who value a computational notebook that is a communication tool with the ability to do computation and make concepts easier to understand with the use of prose, visualizations, and multimedia.

While I do think the classic notebook needs a more modern JS foundation, I think there are many users and use cases where the classic notebook is preferred.

As @stefanv mentions, Tornado is still viable and moving to asyncio or another async framework is certainly possible.

psychemedia commented 3 years ago

I've been struggling to try to articulate my issues with JupyterLab for several years, so value the additional takes provided by @gutow and @willingc.

I don't want to knock the huge contributions made by the core developers but I think part of the issue is that for the not-developer community it's hard to know where to raise concerns. Github is a development environment and issues (and even more, PRs) are often not the place, or not felt to be a place, where non-developers feel comfortable in raising what we recognise as technical issues.

Part of this is because we don't necessarily have the vocabulary or technical understanding to be able to communicate our concerns in what is essentially a specialist technical community workplace.

And part of this is that our concerns, as users of medium, don't resonate with the developers who don't see why we want to do particular things in a notebook (example answer: because we are using it as an authoring and worksheet medium to print out classical Latin declensions, draw electrical circuits, work with very simple musical scores, author chemical compound lookup sheets, or render flashcards, not as a development environment or a scripting environment).

One way I think of Jupyter notebooks is a read/write/execute environment for communicating scholarly content. I don't care about the plumbing. The notebook UI surfaces a simple development environment for me where I can write HTML and simple javascript, or more commonly, generate rich outputs using third party subject-matter-specialist packages, or perhaps author some simple magics, at a level that it very close to the end user but still hidden from them.

Some very simple patterns were evident in the original Jupyter notebook extensions that were enabling of a wide range of customisations. For example, an extension that iterated over cells, tested cell type, and then applied simple styling to HTML of the cell is very powerful and usefully reusable. I still can't fathom how to do this in a JupyterLab extension (in the following I deliberately abuse all sorts of notation to make a frankenstein flavour of pseudo-code):

when notebook_loaded:
  for cell in cells:
    tags = cell.metadata.tags
    if mytag in tags and cell.type=my_desired_cell_type:
      update_css(cell, mystyle)
      update_tags(cell, add_tag=[newtag1, newtag2], remove_tag =[oldtag])

Another really powerful pattern is a toolbar button:

add_toolbar_button(myicon, myfunction)

@acts_on_selected_cell
def myfunction(cell):
  tags = cell.metadata.tags
  if tag in tags:
    update_css(cell, mystyle)
    update_tags(cell, add_tag=[newtag1, newtag2], remove_tag =[oldtag])

As soon as that gets buried in a 150 lines of boilerplate, plumbing, variously nested flavours of bracket, awaits, development environments, node, gulp, test, and poetry, I really don't know where to start.

I think another gulf is that a lot of "educational hacks" are bits of code to get a job done once and once only for a particular end user task. The code only has to work in a very limited context (known to the author), potentially only have to work once, and even then, potentially under supervision, where you can say "don't try this because it won't work" and then move on. This is not the same as production code...

gutow commented 3 years ago

As I read the comments made so far, I notice that the theme seems to be that the classic Jupyter notebook is being abandoned by the original authors because they are moving on to something they can envision as a better tool for the things they do (generate "production" code). I also think they believe JupyterLab will be easier to maintain. So the questions is how do we convince a core of developers to maintain or update the Classic Notebook in a way that is useful for the large number of users who do not need an IDE in a browser, and actually find it a hindrance to their work?

My personal opinion is that there are already other very good IDEs out there (PyCharm and the venerable Eclipse to name two). Thus, those working on JupyterLab are spending their time rewriting an existing tool (reinventing the wheel) in a new front end (web browser) rather than developing something truly new and exciting. The only part of JupyterLab I can interpret as a genuine improvement over existing options is its potential for real-time collaboration on the same piece of code (I believe PyCharm already supports something similar). I've been using computers long enough to be confident that we will cycle back to more localized computing from cloud computing again. When I started using computers most computing was sent to computing centers (the equivalent of today's "cloud" computing). Then, except for really big jobs, the computing was distributed to PCs. Recently, the "cloud" has been popular again. However, you can already see the return to more localized computing starting up with the discussion of "edge computing for IOT". I think there will be a mix, so JupyterLab is not going to win out over other IDEs that run more locally, just because it is inherently a server-client application.

A second reason pure cloud computing is doomed is that anything in the cloud is accessible by anybody who is determined enough. This means there will always be a need for localized computing resources and tools.

A third reason pure cloud computing is doomed is that there are many instances where being on a network is not practical.

So what do we need to do to get help maintaining and updating this really useful tool?

gutow commented 3 years ago

Some ideas for a path forward but migrating to JupyterLab:

Encourage work on RetroLab so that it has the simplicity Classic Notebook users want/need.
Encourage or pay for some templates for common things/extensions people want to be able to do as extensions:
1. Dialog boxes with common elements (check boxes, radio buttons, buttons, textline input, textbox input at the least).
2. Way to extract code from a particular cell.
3. Way to put code into a selected code cell or markdown cell.
4. Way to force execution of a particular cell.
5. Ability to lock cells against editing.
6. Ability to freeze cells so that they cannot be run again and their output cannot be deleted.
7. probably a few more...
Encourage or pay for development of a much simpler paradigm for accessing cells and creating simple extensions to the interface. What do people think of this list and how might we actually implement it?

jph00 commented 3 years ago

Many thanks for opening this issue @Zsailer . One important thing to note is that many data scientists (including me) choose to use classic notebook, despite having the technical proficiency to use Lab, since it meets our needs better. Classic Notebook is not just something that people use because they find Lab too complex.

Classic Notebook has had years of fine-tuning such that for people that use it every day, it's exceptionally productive. For instance the rapid ability to jump around sections with keyboard shortcuts provided by the collapsible headings extension, or the direct access to editing keyboard shortcuts from any notebook menu.

In the most recent Kaggle developer survey, more than twice as many people use classic notebook than use lab. People really like it, and for good reason!

I'm extremely grateful that the classic notebook project hasn't been archived, and that some kind folks are continuing to maintain it.

mcrutch commented 3 years ago

Trying to do extensions in lab is still very much a moving target. I haven't been able to replicate all of our previous extensions in JupyterLab yet, and some I haven't even been able to truly map because of the complexity of the new APIs for extensions and the sparseness of examples compared to classic. Some features I needed to complete the extensions didn't even exist till Lab 3. I have even been able to actually devote a small portion of my time to writing lab extensions, as opposed to any organizations that don't have the resources to do so. We had several, more casual, developers who contributed to the classic extensions. Now, I am the only one writing our Lab extensions as the casual devs either couldn't wrap their heads around how Lab extensions work, or just didn't have the time to try and figure things out from what examples are out there.

While I love the Lab interface, writing extensions for it (which we absolutely need for our environment) has been a bear and so we still lean on classic for some things and probably will for a while.

SylvainCorlay commented 3 years ago

Bower is a problem, but it seems like it should not be too hard to port that over to webpack or similar. Tornado, however, has had frequent releases (at least until October 2020) and can handle the load, so I'd like to understand that concern better.

Yeah tornado is a problem in the Jupyter stack general due to lack of maintenance - and the fact that it's async model is essentially a compatibility wrapper on top of asyncio. But this specific issue is accessory to the main discussion here.

SylvainCorlay commented 3 years ago

I support putting the classic notebook in an archived state, which would reflect the level of support and maintenance that the project is currently receiving. If we don't communicate this clearly, we will be failing our users who expect more.

The "simple mode" of lab and "retrolab" both offer a more maintained implementation of the "full-page single document" model of the classic notebook, that benefit from all the work of the JupyterLab team on all other aspects (collaborative editing, accessibility, visual regression testing, internationalisation).

matthew-brett commented 3 years ago

@SylvainCorlay - the general theme on this thread seems to be "please, please don't archive this project, I need it" - but you are saying you think it should be archived anyway.

Do you mean to say that the various arguments that people have made here, that the classic notebook continues to be a better match for their needs - are based on a misunderstanding? Or that the needs of the project developers to move people onto the newer platform must override the needs of the users? And if you're saying neither of those things, surely it would be better to consider the options for continued support the classic notebook?

matthew-brett commented 3 years ago

On Sun, Oct 24, 2021 at 8:49 PM Jonathan Gutow @.***> wrote:

As I read the comments made so far, I notice that the theme seems to be that the classic Jupyter notebook is being abandoned by the original authors because they are moving on to something they can envision as a better tool for the things they do (generate "production" code).

I realize this is somewhat off-topic - but that is a related question for the JupyterLab developers - do you, in fact, use JupyterLab for writing production code? In the past, when I've asked Jupyter developers, they have said they do not use Jupyter very much, for their own work, largely because writing code for libraries, like Jupyter, really needs a proper code editor like Vim, Emacs, PyCharm or VSCode.

SylvainCorlay commented 3 years ago

@SylvainCorlay - the general theme on this thread seems to be "please, please don't archive this project, I need it" - but you are saying you think it should be archived anyway.

Absolutely, I am an independent person stating (and arguing) my independent opinion. Hopefully it is fine!

I think we should archive the notebook repository for the reasons mentioned earlier by @Zsailer and the other reasons I listed before (security, lack of maintenance by the core team, use of deprecated technology, internationalisation, accessibility). We will fail our users if we continue to pretend that notebook is fine, and people will move away if they think that Jupyter is just that.

I am confident in the JupyterLab community to address the needs of classic notebook users. It is one of the most welcoming and supportive open-source communities out there. The public weekly call (for the general lab meetings) has dozens of participants each week, with new faces every time, showing amazing advancements and extensions. This is really one of the most active venues for engaging with the Jupyter project.

SylvainCorlay commented 3 years ago

Regarding the notebook deprecation, one thing we should probably do is to recommend jupyter-server more explicitly to those who already use JupyterLab but serve it with the classic notebooks server (or jupyter-server + nbclassic for those who use JupyterLab and want to keep offering the classic notebook UI). This could be done independently of the archiving of the notebook repository.

matthew-brett commented 3 years ago

Thanks for the reply.

I guess my next question is - what would it take to persuade the Jupyter developers that the project is in danger of failing? I'm not saying it is, I'm just asking, what would y'all look for to warn you, that that may happen?

For example, imagine for a moment that the slow uptake of JupyterLab does not accelerate, and many people continue to prefer the classic notebook, for the reasons given here. Perhaps you do archive this repository, but someone, for the reasons given here, forks the repository, and carries on maintaining it. Meanwhile, imagine Visual Studio Code finally succeds in developing a good notebook interface. I would imagine, from the outside, that the JupyterLab project would be in serious trouble at that point. I mean, I am sure development will continue, but it seems likely that you'd lose a lot of users, and surely, developers will follow. Do you agree, that would be a problem? If so, do you think that some version of those events is unlikely? And, if y'all thought there was a realistic chance that something like those events would happen, what would y'all do?

rgbkrk commented 3 years ago

I would maintain the classic notebook, if I had the time and the resources to do it. Back when JupyterLab was in its infancy there was a bit more of a concerted effort to keep a UI going that was similar. It's much of the reason why nteract started being built out in the way it was. Attempts to modernize the javascript while keeping the style and layout the same, were met with big resistance as well as a "why can't you just work on JupyterLab?". There were some deeper conversations related to funding and direction. There were also areas where it was difficult to have robust technical conversations around javascript libraries, not just because of who was funding but also a fundamental misunderstanding of the javascript ecosystem and frontend development.

A lot of us just bounced over time as a result. I'd love to have conversations again if anyone wants to do a zoom call and figure out the pulse of today.

Given where everything is at now though, I recommend just going the Jupyterlab route and using the JupyterLab code to create the classic notebook experience (without supporting extensions).

SylvainCorlay commented 3 years ago

For example, imagine for a moment that the slow uptake of JupyterLab does not accelerate, and many people continue to prefer the classic notebook, for the reasons given here. Perhaps you do archive this repository, but someone, for the reasons given here, forks the repository, and carries on maintaining it. Meanwhile, imagine Visual Studio Code finally succeds in developing a good notebook interface. I would imagine, from the outside, that the JupyterLab project would be in serious trouble at that point. I mean, I am sure development will continue, but it seems likely that you'd lose a lot of users, and surely, developers will follow. Do you agree, that would be a problem? If so, do you think that some version of those events is unlikely? And, if y'all thought there was a realistic chance that something like those events would happen, what would y'all do?

Well, I run a company that is entirely devoted to contributing to open-source, and to Jupyter specifically, so I have a lot of skin in the game (and the people on my team too). I think a lot about the project and its future, and I have read carefully the arguments of the vocal folks who have blogged against JupyterLab and criticised its developers. The development of retrolab by Jeremy at QuantStack is something that we self-funded to address some of these concerns.

I am worried about VSCode and Microsoft's deep pockets. It did not take them long to integrate our work on JupyterLite in VSCode and support pure front-end workflows... They are really after Jupyter - but my take is that JupyterLab is our best shot.

gutow commented 3 years ago

@SylvainCorlay I agree that JupyterLab is probably the correct way to go. However, until it develops into something at least as straightforward to use and extend as the classic jupyter notebook (which definitely has its own opacity issues for extension) you are in danger of leaving a large number of users out in the cold. I know people who are migrating to the Google collaboratory even though it means giving up some of the useful tools people have made for Jupyter notebook, rather than try to use JupyterLab.

RetroLab is a good start. I took another look at it this weekend and it is pretty close to the simplicity of the old notebook. That is good. The next question is how to make it easy to migrate all the tools that depended on classic notebook behavior to the RetroLab interface. I am not adverse to doing that for the code I have developed. However, I would need a clear path. Note: I still need to test RetroLab for stability and usability for the specific tasks I do, so I am not yet at the stage of saying I could successfully switch.
I have one big worry. I often find myself in the situation of needing things to run on old/low power hardware. This means that what appears to be the many layers of abstraction in the way JupyterLab works could bog these systems down. I already run into occasions where the old Jupyter front end can push the limits of the client processing resources. Is there data on JupyterLab's resource requirements and the demand it puts on the browser javascript engine relative to the classic Jupyter?

SylvainCorlay commented 3 years ago

I have one big worry. I often find myself in the situation of needing things to run on old/low power hardware. This means that what appears to be the many layers of abstraction in the way JupyterLab works could bog these systems down.

QuantStack currently has funding to work specifically on JupyterLab performance. As part of this work, Frederic Collonval developed a tool to systematically measure performances as part of the continuous integration of PRs, and to produce a report of the result, so that we don't degrade the performances as we work on new features. Performance of Lab-based application is one of our main focus for the coming months.

matthew-brett commented 3 years ago

On Mon, Oct 25, 2021 at 5:56 PM Sylvain Corlay @.***> wrote:

I am worried about VSCode and Microsoft's deep pockets. It did not take them long to integrate our work on JupyterLite in VSCode and support pure front-end workflows... They are really after Jupyter - but in my take JupyterLab is our best shot.

My question here is - how do you know? I mean, what are you shooting for, what are the various options, and how to you know this one is best? And, what would you use to tell you that you might be wrong? For example, imagine that the project had, in fact, taken a wrong direction, and adopted a technology stack that was too complex to work with. What kind of thing would warn you that that might be the case, and is there any situation that would cause you to stop and re-evaluate whether the project needed to take a different direction, rather than continue ahead along the same track?

SylvainCorlay commented 3 years ago

My question here is - how do you know?

Well, who really knows anything? Maybe the main risk of failure comes from not being able to move on from the classic notebook repository...

gutow commented 3 years ago

I have one big worry. I often find myself in the situation of needing things to run on old/low power hardware. This means that what appears to be the many layers of abstraction in the way JupyterLab works could bog these systems down.

QuantStack currently has funding to work specifically on JupyterLab performance. As part of this work, Frederic Collonval developed a tool to systematically measure performances as part of the continuous integration of PRs, and to produce a report of the result, so that we don't degrade the performances as we work on new features. Performance of Lab-based application is one of our main focus for the coming months.

Super. So what is the answer? Is JupyterLab, as I fear, a heavier weight front end than the Classic Jupyter notebook?

SylvainCorlay commented 3 years ago

Super. So what is the answer? Is JupyterLab, as I fear, a heavier weight front end than the Classic Jupyter notebook?

It depends for what. There is measurable overhead in switching between dock area tabs compared to browser tabs, that we are addressing now. I think lab is handling the slow-starting kernels better. The opt-in collaborative mode has a slight memory overhead over the normal mode, but we picked a very efficient CRDT implementation.

matthew-brett commented 3 years ago

My question here is - how do you know?

Well, who really knows anything?

I feel very sure that the combined wisdom of the Jupyter team do a lot better than that in predicting what will happen in the future, in different reasonable situations. But it does frighten me for the future of open-source notebook computing, that that does not appear to be happening.

gutow commented 3 years ago

Super. So what is the answer? Is JupyterLab, as I fear, a heavier weight front end than the Classic Jupyter notebook?

It depends for what. There is measurable overhead in switching between dock area tabs compared to browser tabs, that we are addressing now. I think lab is handling the slow-starting kernels better. The opt-in collaborative mode has a slight memory overhead over the normal mode, but we picked a very efficient CRDT implementation.

The issues I have seen may be lower level than that. I am worried about responsiveness of the interface within a single notebook page. What is the relative javascript processing times in Chromium or Firefox for simple calls like: selecting a set of cells, returning the cell content as a javascript text, pasting new content into a cell. Is there an overhead cost of the abstraction in JLab versus Classic Jupyter to these common javascript calls?

matthew-brett commented 3 years ago

In the most recent Kaggle developer survey, more than twice as many people use classic notebook than use lab. People really like it, and for good reason!

Just because I was interested, I chased up this reference (thanks to @jph00 for pointing it out).

It's Kaggle's 2021 survey

Jupyter notebook - 73.5%
Visual Studio Code - 38.8%
JupyterLab - 33.6%
PyCharm - 33.2%

SylvainCorlay commented 3 years ago

What is the relative javascript processing times in Chromium or Firefox for simple calls like: selecting a set of cells, returning the cell content as a javascript text, pasting new content into a cell. Is there an overhead cost of the abstraction in JLab versus Classic Jupyter to these common javascript calls?

I don't think anyone runs benchmarks on lab vs notebook for these things, but these are not very expensive operations. I don't think there is any intrinsic reason for them to be more expansive for lab than for notebook.

gutow commented 3 years ago

What is the relative javascript processing times in Chromium or Firefox for simple calls like: selecting a set of cells, returning the cell content as a javascript text, pasting new content into a cell. Is there an overhead cost of the abstraction in JLab versus Classic Jupyter to these common javascript calls?

I don't think anyone runs benchmarks on lab vs notebook for these things, but these are not very expensive operations. I don't think there is any intrinsic reason for them to be more expansive for lab than for notebook.

You are correct they are not very expensive. However, if JLab uses 3X the processing power that Jupyter notebook does because of multiple levels of branching logic then this will add up if you try to do things that stack these processes. On low power hardware this can become noticeable even in the classic notebook. I suspect a factor of 3X cost would be OK. If it is 10X I think there will be issues.

Based on my experience I suggest some benchmarks be run comparing low-level operations between JLab and the old interface.

SylvainCorlay commented 3 years ago

Based on my experience I suggest some benchmarks be run comparing low-level operations between JLab and the old interface.

Thanks for the suggestion. A PR would be super welcome!

Carreau commented 3 years ago

As I read the comments made so far, I notice that the theme seems to be that the classic Jupyter notebook is being abandoned by the original authors because they are moving on to something they can envision as a better tool for the things they do (generate "production" code).

I don't think that is completely true. I still prefer notebook to lab, and there is also muscle memory of typing jupyter notebook. For me there are multiple contributing factors to not work on notebook anymore:

I don't have enough time.
I'm already barely capable of maintaining other lower infrastructure in Jupyter.
burnout from early interactions when JLab was created (see below).

I also think they believe JupyterLab will be easier to maintain.

No true either. I hate when I have to patch jupyterlab. I have to touch the source rarely but everytime do it takes me hours to get it to properly work when build from source. I had to patch an extension for a client so I had to track time, and it took me 3:30 to get it to build properly. I understand that types are useful if you are regularly contributing, but when you are an occasional contributor it is still way too hard (better than it was).

So the questions is how do we convince a core of developers to maintain or update the Classic Notebook in a way that is useful for the large number of users who do not need an IDE in a browser, and actually find it a hindrance to their work?

I don't think that's a convincing problem. IMHO it's part of a governance/decision making in the project. There are things that people don't like to do, and at some point you just have to find the carrot and the sticks to get people to do what they don't like. While I'm happy with recent advance in the governance (https://github.com/jupyter/governance/pull/103) , I have the feeling the various councils have in effect no power as they rely on the community voluntary following the councils recommendations.

Classic Notebook has had years of fine-tuning such that for people that use it every day, it's exceptionally productive. For instance the rapid ability to jump around sections with keyboard shortcuts provided by the collapsible headings extension, or the direct access to editing keyboard shortcuts from any notebook menu.

I find that the experience has been worse recently (can't reopen the pager after you close it). And personally I spent an inconceivable amount of time trying to change some of the decision lab made on the UI/UX that was different than notebook. I'm now burned out and am discourage to ever work on those issue in any projects. I know of folks to now refuse to work on anything notebook related even for paid work because of similar burnout.

I guess my next question is - what would it take to persuade the Jupyter developers that the project is in danger of failing? I'm not saying it is, I'm just asking, what would y'all look for to warn you, that that may happen?

I think that for many you don't have to convince them that it's in danger. What you have to find is how to get people to do the right thing, and how to fight against things like sunk cost fallacies, or recognise that you sometime need to say no to new features. IMHO there should be a list of what is critical for the Jupyter organisation, and guaranties about maintenance, performance, etc. It's not limited to notebook i'm thinking traitlets, nbformat, nbclient... I think that if this is not done those discussions will keep happening.

goerz commented 3 years ago

As a user, I have zero attachment to the old Jupyter Notebook codebase as such and I can't imagine anyone would hesitate to move to the more modern incarnations of Jupyterlab, assuming there is feature parity for whatever workflow they're used to.

Personally, I don't care much for the default Jupyterlab interface: It's a UI inside a UI (my browser), which is clunky and causes problems: I don't want to touch the mouse while I'm working, and I'd like the UI to be about as responsive as a CLI terminal. The browser already has window/tab management, which is really all I need in terms of UI. Navigating between tabs inside Jupyterlab sort of gets in the way of navigating between browser tabs. Plus, the considerable complexity of the Jupyterlab UI makes it less responsive than I'd like (although this has gotten better, and I expect will get better even more in the future). Anyway, I don't want to dump too much on Jupyterlab... I'm sure lots of people like it, and to each their own.

The upside is that the Jupyterlab frameworks enable a feature that I probably want more than anything: The ability to have the same notebook open in multiple tabs (synchronized), and by extension, collaborative editing. The Retrolab UI is nearly perfect, with one big caveat: it doesn't have full feature parity with classic notebook extensions. There's actually only two extensions I rely on: execute_time, which pretty much works now in Jupyterlab/Retrolab, and toc2, which is still sorely missing, see https://github.com/jupyterlab/retrolab/issues/168 and https://github.com/jupyterlab/jupyterlab/issues/10452

If the TOC feature was available in retrolab, I would switch in a heartbeat, and never look back.

jph00 commented 3 years ago

I support putting the classic notebook in an archived state, which would reflect the level of support and maintenance that the project is currently receiving. If we don't communicate this clearly, we will be failing our users who expect more.

The vast majority of jupyter users use classic notebook. Archiving the project they use would literally be failing them. To clearly communicate the support and maintenance levels being provided, simply do that: communicate it. Don't do it by proxy by effectively shutting down the project they rely on.

If the project were archived, the result would not be that all those users would switch to Lab. Lab has been around a long time, users are pushed heavily towards it and away from classic notebook, but users have on the whole chosen not to switch. So users would either switch to something like vscode, or the user community would fork classic notebook to allow it to continue to be maintained. Either way, it would be very harmful to the jupyter project overall.

The "simple mode" of lab and "retrolab" both offer a more maintained implementation of the "full-page single document" model of the classic notebook, that benefit from all the work of the JupyterLab team on all other aspects (collaborative editing, accessibility, visual regression testing, internationalisation).

Unfortunately the user experience for these products is a substantial regression at this time, at least for my needs. I try using Lab, including simple mode and retrolab as long as they been around, for a couple of weeks every six months to see how it's progressing and whether I can recommend fast.ai students and users switch. My last test was 3 months ago. There's still a very long way to go.

SylvainCorlay commented 3 years ago

The vast majority of jupyter users use classic notebook.

It's hard to say, even with the polls published above. Many users talk about the "jupyter notebook" while they really mean the notebook in lab.

matthew-brett commented 3 years ago

The vast majority of jupyter users use classic notebook.

It's hard to say, even with the polls published above. Many users talk about the "jupyter notebook" while they really mean the notebook in lab.

Well - again - I don't think it is very hard to say. The Kaggle figures do specifically distinguish between the Jupyter notebook and JupyterLab. But the key question is - would you do anything different if you were convinced that most people using the notebook were in fact using the classic notebook? Or if you were convinced that it was unlikely they were going to migrate to JupyterLab in the near future. If the answer is "no" - then "it's hard to say" is about as good an answer as "go away I'm working" - and leads to the same result.

SylvainCorlay commented 3 years ago

then "it's hard to say" is about as good an answer as "go away I'm working" - and leads to the same result.

We are specifically putting work and resources (while being a small company) on the single document UI and retrolab - so we are definitely doing something to address the concerns of people who like the full-page single document UI.

Now, in all my interactions on this thread, I have only been posting positive comments about the community and what our team was doing to address the specific concerns of people worried by the sunsetting of the classic notebook.

@matthew-brett please remain civil and be considerate that there are quite a few people who are working around the clock on Jupyter - and specifically on JupyterLab, both at a technical level and to build a stronger community around it.

nicolaskruchten commented 3 years ago

Many users talk about the "jupyter notebook" while they really mean the notebook in lab.

In my experience helping folks debug things, many people will say "jupyter notebook" to mean generically any notebook, so I personally take these figures with a grain of salt!

For my own understanding and that of other readers in my position: do retrolab and/or Simple mode support extensions written against the old Classic Notebook API? (I assume no) Is there any conceivable glidepath for Classic extensions to ever live in Lab? Technically could someone build a lab extension that 'hosted' classic notebook extensions?

matthew-brett commented 3 years ago

@matthew-brett please remain civil and be considerate that there are quite a few people who are working around the clock on Jupyter - and specifically on JupyterLab, both at a technical level and to build a stronger community around it.

Please do understand - I'm not accusing you of not working hard - I am truly sure that you are. But it's true, my remarks are expressing some despair about what appeared to be your lack of interest in whether the current notebook strategy is working - or has a reasonable chance of working. If that is impolite, I'm sorry, I did not mean it to be. My only concern is the health of the community infrastructure for notebooks on which so many of us depend. As you can see, I am worried that, if the current situation continues without any conscious attention to direction and strategy, that may lead to - or even - is likely to lead to - a situation where the community no longer has much agency in the development of the notebook.

Carreau commented 3 years ago

For my own understanding and that of other readers in my position: do retrolab and/or Simple mode support extensions written against the old Classic Notebook API?

No, as you assumed.

(I assume no) Is there any conceivable glidepath for Classic extensions to ever live in Lab?

No. And even if it was It would not a reasonable request.

Technically could someone build a lab extension that 'hosted' classic notebook extensions?

For simple extensions, maybe (?) though likely not worth the effort.

To be clear, the way we did extension in classic notebook is horrendous (and I am responsible for most of that design), it was ok for the tech at the time (bower/requirejs) and the fact that none of us were JS developers. It was ok for the market share of the notebook at the time. It's mostly monkey patching, rely on globals, and has no defined boundary. It's one of the reason classic notebook is stalling, we can't change anything as everything is part of the API.

The way extension work in JupyterLab (and that everything is an extension including the core) is the proper way to do it from a technical POV if you want to scale. It could be a bit simpler by dynamically JS and monkey patching like old notebook for really basic extensions but the drawbacks would far outweigh advantages:

Users would get arbitrary impossible to debug error messages at run time, or worse no error but JLab would not work.
Businesses could not vet extensions and could not adopt JupyterLab, as it would be security risk. Unfortunately that comes with the complexity of typescript and all the build steps/linking.

With enough $ or manpower most extensions could be rewritten for Jlab and with env detection we could have a single package for both Jlab and classic. But that requires rewrite most of the time, and the code base would basically contain two extensions. Maybe some logic could be shared, but minimally.

There is no reasons that extensions written for JupyterLab to work only with JupyterLab though (and they work for retrolab for example). So if any efforts should be put in extensions it should be on converting extension to the JLab model.

Moreover the number Classic notebook extension is finite and relatively stable, so converting is a 1 time cost spear across the extension maintainers. Writing a shim layer to let classic extension work in Jlab would be an indefinite cost on the Jlab maintainers.

jph00 commented 3 years ago

@matthew-brett please remain civil and be considerate that there are quite a few people who are working around the clock on Jupyter - and specifically on JupyterLab, both at a technical level and to build a stronger community around it.

@SylvainCorlay I'd also ask that you be considerate that you are, in a rather detached and dispassionate way, effectively proposing destroying something that is key to the livelihoods of many people (including me).

jupyter / notebook

[Discuss] The future of the classic notebook interface #6210