search

jupyter / notebook

Jupyter Interactive Notebook
https://jupyter-notebook.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
11.63k stars 4.89k forks source link

[Discuss] The future of the classic notebook interface #6210

Closed Zsailer closed 2 years ago

Zsailer commented 2 years ago

The purpose of this issue is to discuss the future of the jupyter/notebook repository.

Background: The classic Jupyter notebook interface is currently maintained by a skeleton crew of developers. On the other hand, many people still use the classic interface today.

Problem: We have a widely-used piece of software with very little developer support.

Let's have an open discussion about what to do with this project moving forward.

Zsailer commented 2 years ago

Thank you, all, for your input here! I realize this is a super challenging discussion to navigate. That's okay; I know this community can handle it 😃 .

A really challenging aspect of the Jupyter community (and I recognize this isn't unique to us) is that there are many users and relatively few maintainers (part-time/full-time). I'm making up numbers a little bit here, but if I had to guess, there are likely millions of users and maybe a few dozen active maintainers? That's a pretty large disparity.

As the project's ecosystem grows, these maintainers are being spread thin over many projects; hence you'll hear some exhaustion/burnout (maybe even frustration) in some responses. The number of issues, opinions, and frustrations coming from user across various sub-projects has quickly outpaced the number of maintainers; as a consequence, many maintainers feel overwhelmed. Personally, I will admit that the time I spend maintaining the jupyter/notebook repo often feels like being a punching bag.

There are various things we could do (or could have done) to grow the list of maintainers; however, this takes a lot of bandwidth too. That's a separate discussion... :wink:

I say all this to point out, maintaining the jupyter/notebook repo is no small task. It requires a lot of maintainer time that Jupyter doesn't have.

And we have this strange problem—the project is hemorrhaging because it relies on outdated/deprecated dependencies+codebase. At the same time, updating/evolving/changing the codebase is nearly impossible since so many people use the project and extend it in non-sustainable ways. That's why projects like nteract and JupyterLab were born; it was easier to start from scratch and offer new stable frontends as alternatives than change things in-place.

At this point, (I think) the best we can do with this codebase is keep the classic notebook interface hobbling along. The problem with that is we're short staffed, unfortunately. We're doing our best to host weekly Notebook meetings to carry the current load with a small group of people, but it's a bit discouraging.

I think what others are proposing is that we continue devoting our efforts to making modern projects like retrolab, simple-mode in JLab, nteract, etc. fill the role of classic notebook. Identifying the gaps is super helpful, so I am extremely grateful for everyone sharing these gaps on this issue. ❤️

gutow commented 2 years ago

I think what others are proposing is that we continue devoting our efforts to making modern projects like retrolab, simple-mode in JLab, nteract, etc. fill the role

@Zsailer based on this I see a consensus forming around the following steps:

  1. Continue maintaining as best possible classic Jupyter until the goals in the following steps are achieved.
  2. RetroLab is a stable alternative providing near parity with the simplicity of the classic notebook.
  3. A path is devised for users who need to add simple one off interactive elements without the overhead of many times more boilerplate than would be necessary to add a DOM element in javascript. Examples of what should be possible are (the following list should be refined):
    1. A button that activates a notebook action.
    2. A menu that activates notebook actions or injects strings into a selected cell.
    3. Selection of cells.
    4. Extraction of cell contents in the form of a javascript string.
    5. Insertion of text at a selection point.
    6. Dialog boxes (not nb_widgets) that can be used to contain arbitrary DOM elements and return values from input elements.
    7. Simple javascript calls to change the metadata for a cell or the notebook.
    8. Simple javascript calls to access the metadata.
    9. I'm sure there are a few more, but I am trying to figure out what the minimum is. Maybe one or more of the above does not belong.

Aside: I also have a dream that the markdown cell would support one of the good WYSIWYG html editors. That would open things up to even more users. I think it would be one of the killer extensions for the jupyter ecosystem.

maegul commented 2 years ago

I agree with https://github.com/jupyter/notebook/issues/6210#issuecomment-951486222.

IMO, extension/plugin parity is one of the major issues with JLab uptake. I still use classic mainly for the TOC extension, which only recently has a comparable alternative in JLab (for my use cases). I imagine there are many similar perspectives/experiences.

@Carreau ’s suggestion of reimplementing the classic extensions in the modern framework, perhaps for Retrolab for instance, makes a lot of sense to me. Without wanting to be rudely critical, I’ve personally thought such would have been a sensible investment for a long time (without truly knowing the viability myself).

Having a more easily hackable interface in jupyter seems like a great idea going forward. I’ve done minor dev against JLab, as a non front-end dev, and have found it prohibitively difficult, but nonetheless regretted not having my desired and simple functionality.

In short, reimplementing the classic notebook with the lessons learned and tools built for JLab makes a lot of sense. Against VS Code, which may very well eat JLab (and for good reason), polishing the simple and elegant classic interface is probably a good way forward for longer term success and posterity.

For me, focusing on the notebook/kernel interface (see Observable for comparison), rather than the IDE wrapping that is JLab, would be a more valuable investment.

agoose77 commented 2 years ago

I am predominantly a Jupyter user, although I do contribute to the various projects from time to time.

In (my experience of) academia, Jupyter Notebook is still heavily used by students. I believe this is because the University itself often doesn't even have Lab installed in the computing clusters, or courses have not updated to include reference to it. I've shown many students JupyterLab, and in most cases they quickly prefer it to classic notebook. This is only an anecdote to support the fact that it is difficulty to gauge the reason that people use notebook (in addition to discerning who actually means notebook vs lab).

The comments in this issue remind me of the general problem with open source maintainer burn out. Jupyter has become so central to so many people's lives that it's easy to forget that many of the people working in the project are volunteering at least a portion of their time. Ultimately, it is not reasonable for the users to dictate how developers spend their time - if maintainers say that JupyterLab is more maintainable than classic notebook, then we all need to listen to that.

Clearly, however, the contributors to Jupyter are invested in supporting the classic notebook user experience with RetroLab and simple mode Lab. Perhaps we can go one step further, and poll the user base to determine the features that will most be missed if they moved to retrolab / simple mode. Evidently, there will be some Notebook users who have heavily invested in tooling such as classic extensions that will find it harder than most to upgrade to lab. I suspect for the rest (majority) of notebook users, however, that this is not the case. It might be that some users are waiting on a couple of extensions such as toc2, or nbgrader. If this were the case, there are things that the community can do to move itself towards the modern (extensible) lab ecosystem. Compiling a list of this most needed extensions could be a good start for contributors who want to help out

paugier commented 2 years ago

As a simple user (and teacher using Jupyter and "notebooks"), I just like to mention that I don't care too much about jupyter-notebook or jupyter-lab. Now that jupyter-lab is better than jupyter-notebook in many aspects (in particular since jupyter-lab 3), let's move on and use/teach jupyter lab and/or jupyter retrolab.

I also see that there are still issues with extensions. First, few popular extensions should be ported to lab/retrolab. I would mention RISE and nbgrader, but I guess there are others. Is there a kind of official page with a list of extensions that have to be ported and possibly a way for users (people and institutions) to finance the porting?

Second, it seems to me that for Lab extensions the request "simple things should be doable without strong skills in front-end dev" is reasonable. Is there a way to improve the situation with jupyter-lab?

matthew-brett commented 2 years ago

There was a fairly long discussion of the future vision of JupyterLab over on this mid-2020 thread. I think it's fair to say that didn't lead to any clear resolution (happy to be corrected if I misunderstood). There is also a long discourse thread with further discussion.

As a further data point, I am pretty sure that the largest Berkeley data science course (data8) still uses the classic Jupyter notebook interface, and I am also pretty sure that the Berkeley setup has a lot of extensions used in various places - which seems to match the experience that @psychemedia is describing. It would be interesting to hear why Data8 has not yet made the switch, and what the barriers are.

matthew-brett commented 2 years ago

@Carreau - sorry to put you on the spot here, and please feel free to ignore me - but if someone waved a magic wand, and gave you the power to decide where the Jupyter project would go next, what do you think you would do?

benbovy commented 2 years ago

Also as a simple Jupyter user, I'd like to show here some love for JupyterLab and I hope that it will continue to be actively developed and maintained as an alternative to big-$$$ company products like VSCode. To be honest, I don't really understand why people find it too complex. To me, its interface looks pretty familiar compared to many other (desktop or web) applications. 90% of the time I just use the file browser on the left and notebook tabs on the right (I also like to keep everything in a pinned tab browser). This is my personal experience, though, and I respect other users experiences which may be different.

I'd also like to express my gratitude to the developers of the Jupyter (classic) Notebook front-end. Although it's been a while since I switched to JupyterLab, I've been a happy user of the classic notebook front-end for a long time before that. Without the classic notebook interface, JupyterLab probably wouldn't exist today.

Besides its community-led governance, what makes the Jupyter ecosystem outstanding is the diversity of tools and applications (front-ends) that are integrated between each other. I really hope that this will continue in a way that makes both users (porting nbextensions, hacking the interface) and developers (switch to lab-based classic interface like RetroLab) happy, even though it's a lot of work.

Regarding the "simple things should be doable without strong skills in front-end dev", @jtpio do you think that ipylab would fit for that goal? It looks like it has a great potential:

guiwitz commented 2 years ago

This is a very interesting discussion on many levels. Just so that multiple points of views are exposed, I'll just add my experience here. Just like @paugier I'm also just a Jupyter user for scientific programming (e.g. I never created extensions) and teacher. I teach a lot of different courses on image processing, data analysis, visualisation etc. After some similar discussions about a year ago I decided (hesitantly) to entirely switch to using Jupyterlab for teaching. The two main fears I had for which I didn't switch earlier where the difficulties of installing extensions and the fear that the interface was "too complicated". The first problem has been solved by version 3 and retrospectively, the second one was completely unfounded. I work mostly with people who have little or no previous coding experience and the Jupyterlab interface hasn't been a problem at all. Some in this thread mentioned preferring e.g. Google Colab, but Jupyterlab is actually much closer to Colab's interface than classic Jupyter. My superficial impression is that e.g. the "folder browser" of the classic notebook that lives in its own tab is actually more confusing than the panel in Jupyterlab.

For my own work I now almost uniquely use Jupypterlab as well. Unlike what others feel, I find having multiple notebooks open in a single browser tab a huge advantage compared to the multiple tabs of classic notebooks that take a lot of "tab space".

Regarding adoption: I think a lot of the apparent preference for classic notebooks is stil a sort of legacy effect. For example the mybinder project that is used a lot for demos and courses only recently switched to JLab as default interface. And probably many teachers haven't switched yet, "priming" students to use the classic interface.

So overall I'm completely happy with Jupyterlab, and I think it's very important to not entirely switch to things like Colab or VSCode. I think the recent issues around .Net are in this regard are an important warning.

matthew-brett commented 2 years ago

I think the recent issues around .Net are in this regard are an important warning.

That piqued my interest - I don't follow .Net - what happened there?

matthew-brett commented 2 years ago

And here - just to avoid confusion - I suppose everyone on this thread would prefer that the community in general, and the Jupyter project in particular, continues to be the primary driver in defining notebook protocols and user experience. But, as we all know, and as VSCode shows, users will - reasonably, and in large numbers - choose other options, if those options are more compelling. This is just one more example where it isn't enough to identify what would be best for the community; it also matters what is best for the users.

I really don't think it's unreasonable to say, at this point, that a large proportion of the users of the classic notebook have not found a compelling reason to switch to JupyterLab. In some cases this is because it is -I think everyone agrees - difficult to write and port extensions. In some cases, it's just because there isn't anything that JupyterLab does, that we really need. See the discussions I linked above for more on that theme.

The immediate question is - in that circumstance, what should happen to the classic notebook? The two-and-a-half answers here are:

As you can see from my other messages, at least from the outside, and from talking to some of you who are closer to the inside, it's hard for me not to think that the third option is the one that is needed.

agoose77 commented 2 years ago

I think the recent issues around .Net are in this regard are an important warning.

That piqued my interest - I don't follow .Net - what happened there?

Microsoft removed the hot-reloading feature from its open source SDK in order to push people towards the proprietary offering from Visual Studio.

agoose77 commented 2 years ago

Attempt to force users onto JupyterLab by explicitly shutting down maintenance of the classic interface, and hope that they don't choose other options

I understand your frustration @matthew-brett, but I think we need to be careful in how things are framed. From my understanding of the situation, it's not that we're debating whether to actively stop the development of Notebook, but rather that it is already not being actively developed (due to insufficient resources, skills, and particularly poor design), i.e. "that ship has already sailed". I believe that this thread is here to discuss "what next", i.e. how can we encourage users to move to the other frontends (in particular, the Jupyter provided Lab).

In other words, we're already at

Wait patiently for people to adopt JupyterLab, while keeping the classic interface struggling along for as long as practical

So, your final point

Take a completely new look at the problem, and ask - is the lack of adoption of JupyterLab telling us something important about the direction of the Jupyter project?

I believe that this is partly what this thread is here for - what do the existing users of Jupyter Notebook need in order to switch?

My opinion (and it may be wrong) is that most people use Jupyter Notebook not because they've had a bad experience with JupyterLab or that they need a particular extension, but rather that Notebook "works fine" and is familiar. My experience in academia has been that many people learn a tool, and then stick with the same tool until they are forced to change. I suspect that if jupyter notebook were to launch RetroLab, a majority of users wouldn't really care. Jupyter Notebook does not currently warn users that it is the "old" frontend, so for those who don't really engage with the community at large, there is likely little impetus to change what they're using.

But, my opinion is just an opinion - if we really want to answer the question, I believe that we need some kind of semi-formal study. It might be as simple as adding a banner to Notebook that invites users to answer some questions (adding banners to things doesn't always go down well, so I defer to people with more experience here), or establishing some focus groups sampled from the community. Note that I firmly believe that a good number of the Jupyter Notebook user base are "silent users" that do not strongly engage with the Jupyter community. It would be nice to try and reach these people too.

Some thoughts:

benbovy commented 2 years ago

if we really want to answer the question, I believe that we need some kind of semi-formal study.

Not sure whether or not it is a good suggestion (effective but probably too aggressive): do A/B testing on platforms like Binder (e.g., either launch classic notebook or retrolab, with a clear button to let the user switch from one front-end to another if he/she wants to).

eduardobonet commented 2 years ago

Adding my 2 cents:

I really don't think it's unreasonable to say, at this point, that a large proportion of the users of the classic notebook have not found a compelling reason to switch to JupyterLab. In some cases this is because it is -I think everyone agrees - difficult to write and port extensions. In some cases, it's just because there isn't anything that JupyterLab does, that we really need. See the discussions I linked above for more on that theme.

I disagree with this. I worked as a Data Scientist insights, where the notebook was the final artifact that was used to inform business on potential decision outcomes, on a team of 150+ data scientists doing the same (so, not scripts or "production code", just notebooks). Notebooks, for us, was also just a means to an end. The only reason why we used Jupyter notebooks instead of lab was because changing to lab internally required a lot of effort and backend expertise, which we did not have. Once some of us took the burden, and found the right help, we were able to offer Lab (1.x at the time). After a month over 80% of users had switched to lab, even if the default url still pointed to the classic Jupyter, so the decision to use Lab was made at every time they accessed the UI. So it was not about Lab not offering anything that the users didn't look for, it was because users couldn't do the switch at all.

Now working as machine learning engineer, JupyterLab is vastly superior, since it offers a lot of the features that make it that much more productive (debugger, attaching console to python file, easier file navigation, etc), and the extensions are much more powerful, at the cost of requiring more skill to be implemented. Managing extensions became a lot easier, specially considering security, so I appreciate the effort done on that area.

The discussion seems to resolve around extensions, how difficult it is to port them, and how difficult it is to create new ones.

Could it be a reasonable investment from the team to help porting the major extensions that are still lagging behind. For a lot of them JupyterLab provides even better possibilities with room for custom UI (a slide view for RISE, or the entire nbgrader UI within Lab). Two core issues with this is a) Thinning out the contributor efforts even further and b) How to select the candidates for time investment. A big side benefit here is that it would push core contributors to dogfood their own API's, which tends to make the final result a lot more user friendly, in addition to strengthen bonds with the community.

echarles commented 2 years ago

@echarles also notes that users often mention the need for equivalent key-bindings: JupyterLab vision for the next few years jupyterlab/team-compass#80 (comment)

There is more to do on JupyterLab side to get it pairing (assuming this is a goal) with the classic notebook, e.g. Trusting a notebook is not working with JupyterLab https://github.com/jupyterlab/jupyterlab/issues/9765 which sounds to me like a very important feature as related to security/trust.

matthew-brett commented 2 years ago

@agoose77 - it seems to me that this thread does not show consensus for formally shutting down maintenance of the classic notebook interface.

And, yes, that's right - people stick to what they know, unless they have a compelling reason to switch. But the key point here is that, in large part, they do not have a compelling reason to switch. And for the case of those who use custom extensions, they have a good reason not to switch. I think you'd find, in a survey, that people without extensions, just don't care all that much about switching, at least, once they know that RetroLab exists.

So I guess the question is - how much goodwill is the project willing to lose - by trying to force reluctant users to switch? In some cases - for those (like Berkeley) with many extensions, that switch will be very painful, and the loss of goodwill will be significant. Is the gain really worth the cost?

matthew-brett commented 2 years ago

  • The 2020 survey — JupyterLab more common than Notebook (in ranking of top three tools)

Hmm - yes - but these results look very different from the equivalent questions in the Kaggle survey. I wonder whether the Jupyter survey was biased towards those on the Jupyter forums, where you'd expect there to be a strong interest in current code development.

agoose77 commented 2 years ago

@agoose77 - it seems to me that this thread does not show consensus for formally shutting down maintenance of the classic notebook interface.

Again, stop me if we're just crossing wires here, I don't think there is a question of "formally shutting down" anything. I believe that with most Jupyter projects things are "formally shutdown" when no developer touches the project for a while, and it is archived (correct me if I'm wrong here Jupyter team!). Right now, most developers work on Jupyter Lab, and those who are working on Notebook are struggling for the reasons mentioned above. In other words, unless a lot of resources and design work comes along, Notebook is going to keep struggling on, and eventually it's possible that development will cease (once the existing voluntary maintainers lose interest). Archiving at this point is just a reflection of that fact.

And, yes, that's right - people stick to what they know, unless they have a compelling reason to switch. But the key point here is that, in large part, they do not have a compelling reason to switch. [...]

So I guess the question is - how much goodwill is the project willing to lose - by trying to force reluctant users to switch?

These are different statements in my mind:

  1. users not switching because there's no impetus to
  2. users not switching because they prefer Notebook

If a good number of Notebook users are (1), then a deprecation notice will encourage them to move over. We could be more aggressive and have jupyter retro become jupyter notebook, and jupyter notebook become jupyter notebook-legacy, or alternatively, just rename jupyter notebook to jupyter notebook-legacy and have jupyter notebook raise an error. (I'm not advocating that necessarily, it could be problematic). Or as @Carreau suggested elsewhere, perhaps jupyter notebook launches a splash screen. For (2) we need to identify why they prefer Notebook.

Jupyter survey was biased towards those on the Jupyter forums, where you'd expect there to be a strong interest in current code development.

The survey was advertised as follows:

Responses were collected on SurveyMonkey from December 2020 to February 2021. This survey was open to anyone familiar with Project Jupyter with more direct outreach to those involved in the Jupyter ecosystem, current and former. It was advertised on the Jupyter blog, Project Jupyter Google Group, LinkedIn, binder, the Project Jupyter website, Jupyter Discourse, and Project Jupyter Twitter.

So, it should hopefully have reached a good number of "active" Jupyter users. As I mentioned above, I suspect that the "silent" Jupyter users (who just pip install and move on) won't be represented equally, but there are limits to what we can do as an open-source community!

matthew-brett commented 2 years ago

On other results from surveys - (inspired by @blink1073 's comment on the earlier thread).

This is the JetBrains survey - so obviously there is a large bias to PyCharm.

The following are figures for the 'data science' tab of the Editors and IDEs heading.

You'd have to guess, that the users who have not yet switched will be on average more reluctant to switch that the users who have switched. Now imagine putting pressure on them by - for example - archiving this repository and announcing there will be no new releases for the classic notebook. Where are they going to go? JupyterLab, or VSCode?

agoose77 commented 2 years ago

Now imagine putting pressure on them by - for example - archiving this repository and announcing there will be no new releases for the classic notebook. Where are they going to go? JupyterLab, or VSCode?

People use the various notebook frontends for a variety of reasons. I use JupyterLab because I like the notebook extension ecosystem, which currently JetBrains / VSCode don't match (in my opinion). Equally, others might prefer the built-in offerings from these commercial products. At the end of the day, users will use what works best for them (plus/minus some variation from market forces!) — I don't think we can determine what they will do given a deprecation situation without asking them, e.g.

Jupyter Notebook (classic, i.e. not Jupyter Lab) users, what would you need in order to switch to JupyterLab

It's important to note that these Notebook users are already using a deprecated application - the active development community are by-and-large working on JupyterLab, and as such, we've already left them behind in terms of new features (see debugging, etc). Without more developers + funding, I think all that the Jupyter developer community can really do is to make the transition to Lab as welcoming as possible.

Obviously, we're dealing in a lot of hypotheticals here, so bear with me. In this thread, I've seen the following arguments against moving to Jupyter Lab from Notebook

  1. The UX is unfamiliar
  2. It is less simple to develop for
  3. It is more complex to use
  4. It is less performant (?*)

Only one of these is a code-base issue: the actual API for writing extensions is different. The rest of these are UI/UX related.

Extension Writing

@psychemedia wrote

My skill level excludes me from updating my original classic notebook extensions. There is no internal developer support available eg for porting extensions to JupyterLab. I do not have the skill or the time to learn either js frameworks or js dev tools, let alone make sense of JupyterLab API docs and how-to example fragments that are buried in code tests. My instructional material relies on classic Jupyter notebooks with their current extensions.

I was not TS/JS developer (only limited experience prior) when I started writing JupyterLab extensions, and with 3.x I think the landscape has improved: federated extensions have made the development cycle a lot easier. Looking at the documentation, now the examples directory is much more visible. There also seems to be work on a dynamic extension loader to make it easy to prototype extensions inside JupyterLab

We can probably do more work here to make things easier, by re-organising the documentation and making the TS API documentation more discoverable. In particular, I think we would benefit from furthering the work to make the developer documentation friendly for new developers. Sometimes, I wonder whether we overwhelm developers with information in the docs, and it's not always clear to developers where to go when they get stuck. We can likely improve this, e.g. this issue.

AFAICT, the fact that JupyterLab is harder to write for stems from a changing landscape; best practices in web-dev all involve a compiler of one kind or another (see svelte, react, etc). The freedom to patch anything in Jupyter Notebook was good for write-once extensions, but terrible for long-term viability. I think we should try and minimise the inertia for developers getting started with JupyterLab, but I also feel that some of the hurdles are intrinsic to writing modern applications. Perhaps tools like https://github.com/jtpio/ipylab will make it easier to write single-use extensions for users that need them.

UI/UX

This is where asking our users what they want is vital. I strongly believe that the direction taken by RetroLab is a very good start.

matthew-brett commented 2 years ago

Without more developers + funding, I think all that the Jupyter developer community can really do is to make the transition to Lab as welcoming as possible.

I agree that is the implicit assertion - but one I was trying to unpack in my argument, that another path was possible, where the community makes some time and energy to take a long hard look at the project direction.

@Carreau earlier said:

I think that for many [Jupyter developers] you don't have to convince them that it's in danger.

One response, when in danger, is to keep going down the same path, but faster - but it really doesn't look as though that will be the right response, in this case.

eduardobonet commented 2 years ago

Outsider perspective:

I do think maintainer mental health should be the core of the decision making here, and the rest should follow afterwards. This is classical "everyone-wants-to-work-on-the-new-system-but-noone-wants-to-keep-the-legacy-alive" discussion.

If moving to Lab the best alternative for the maintainers, then the question becomes:

The thing is, any big change will cause uproar, ( python2 vs python3 anyone?). From reading this thread, it became clear to me that supporting both versions is taking a heavy toll on the developers mental health, which might push them away of the Jupyter Project entirely.

mwaskom commented 2 years ago

The following are figures for the 'data science' tab of the Editors and IDEs heading.

2019; Classic 13%, Lab 4%, VSCode 23% 2020; Classic 11%, Lab 6%, VSCode 28%

That is a misrepresentation of the survey results: the 13/11% responses are for “Jupyter Notebook” not “Classic.” As others have pointed out, “Jupyter Notebook” means a number of different things, and this survey question as posed is poorly suited to inferring a preference between interfaces. Altering the wording when quoting the results does not seem like an especially good-faith way to make your point, IMO.

matthew-brett commented 2 years ago

From reading this thread, it became clear to me that supporting both versions is taking a heavy toll on the developers mental health, which might push them away of the Jupyter Project entirely.

That's certainly my impression - but the question is - is that because not enough people are using JupyterHub? Or is it because there are some serious structural problems in the project direction, where the slowness of JupyterHub uptake is just one indicator?

It seems to me that is an essential question to answer, because pushing people to JupyterLab may make the problem worse. First, it may result in more, and more frustrated users, increasing support burden, and making support more miserable. And second, by keeping on down the same track, the project may miss the opportunity to stop and address the real problem, if the problem is really not largely confined to the stresses of Jupyter Classic support.

matthew-brett commented 2 years ago

Altering the wording when quoting the results does not seem like an especially good-faith way to make your point, IMO.

Oh man - that simply hadn't occurred to me. I was only trying to use the terms we were using here. And I'm sure that's what the survey people thought they were asking. I don't have the questions to hand though.

jtpio commented 2 years ago

Thanks everyone for sharing thoughts, feedback and concerns :+1:

I wanted to highlight the fact that the future of the Classic Notebook interface has been dear to our hearts as Jupyter and JupyterLab developers for a long time already. It is a topic that JupyterLab developers are actually very concerned about, and has been discussed many times already during the JupyterLab Weekly Developer Meeting and the Notebook Weekly Meetings. See https://github.com/jupyterlab/jupyterlab/issues/8450 and https://github.com/jupyterlab/jupyterlab/issues/9869 for more context.

Most of us devote a lot of time to making things better on a daily basis, with the goal of moving the project forward without leaving anyone behind.

Just wanted to share that, and thank everyone who has put work and efforts into the Jupyter project :heart:

Many here have expressed their concerns about the JupyterLab UI being too complicated. Here is a screenshot comparing the latest version of RetroLab to the Classic Notebook, both running on Binder. With RetroLab, we demonstrate how JupyterLab components can be reused and remixed to produce functional document-oriented (classic-like) interfaces:

image

RetroLab started as a side project and was then self-funded by QuantStack. I'm confident we can make it even better if more people get involved (contributions, feedback, issues). The following issue tries to summarize the current state of the project and lay a path forward for a better integration with JupyterLab: https://github.com/jupyterlab/retrolab/issues/257

As mentioned above, since RetroLab reuses most of the JupyterLab components as is, it directly benefits from all the work put into developing JupyterLab. With RetroLab you get a Classic Notebook experience with:

Again, all the work put into JupyterLab can easily be reused in alternative frontends like RetroLab.

Regarding the "simple things should be doable without strong skills in front-end dev", @jtpio do you think that ipylab would fit for that goal? It looks like it has a great potential

Yes the purpose of ipylab was indeed to ease extension development for JupyterLab and in Python, since Python is more common that JavaScript / TypeScript among the Jupyter users. We might want to continue with that approach, or maybe explore how we could provide Python bindings for developing JupyterLab extensions. Some more thoughts about this in: https://github.com/jupyterlab/jupyterlab/issues/11075

There is also on-going work on the JupyterLab Plugin Playground project, so folks can author extensions from within JupyterLab: https://github.com/jupyterlab/jupyterlab-plugin-playground.

Also, recent developments such as JupyterLite considerably lower the barrier of entry to newcomers who want to learn Jupyter and Python, as everything runs in the browser. This wouldn't have been possible without all the hard work put into JupyterLab.

psychemedia commented 2 years ago

Some observations:

Reviewing all the above, three things jump out at me regarding Classic notebook:

Given significant use in education, I think it would be useful if edu or edu funders could step up as a community to fund developer maintenance and support of classic notebook, rather than end up paying license fees to a Pearson or O'Reilly hosted fork of it. But I don't know how that would work.

Re: JupyterLab:

psychemedia commented 2 years ago

@jtpio Re: RetroLab, as your screenshots show, the RetroLab view can give an experience very much like the Classic UI. For some notebooks. But this is what I see in my notebook UI - coloured cells and collapsible elements (there is a collapsed heading JupyterLab extension, but it uses different metadata to the original extension to identify collapsed headers):

image

I've also got access to bootstrap styling via a simple tags extension:

image

And visual cell execution status indicators (along with some audible heartbeart and cell completion alerts, I think?) (That said, I have a vague memory of seeing something similar for JupyterLab???)

image

I can't "just migrate" to Retrolab...

echarles commented 2 years ago

One response, when in danger, is to keep going down the same path, but faster - but it really doesn't look as though that will be the right response, in this case.

Can not agree more @matthew-brett

On the UI side, a 3rd path I have followed is trying to attract frontend developers to Jupyter with technologies they love like React.js.

Therefor, I have created React.js components you can reuse in https://github.com/datalayer/jupyter-react to assemble and build your own Data Product in a well adopted frontend technology.

Those React.js are backed by JupyterLab where useful and are compatible with Jupyter Server and JupyterHub. However, this does not solve anything related to Classic nor JupyterLab extensions as jupyter-react does not support the existing extension ecosystems, which IMHO is not an issue as each UI widgets toolkit could pick what is indeed in their own existing ecosystem, rather than reimplementing everything from scratch.

Having friendly solutions that React.js, Vue.js, Svelte (just name more) developers is something I am looking for.

In my eyes, I see JupyterLab trying to implement feature-parity with the Jupyter Classic Notebook and Visual Studio Code, which sounds to me like trying to do the splits. I would rather favor various Data Products for various jobs (and not so bad if the beloved keystrokes harcoded in so many fingers are no more available), because the variety of Data Science tasks are diverse enough to ship diverse products.

echarles commented 2 years ago

Jupyter is a set of servers, protocols, a document "standard", and some UIs.

Fully in line with you @psychemedia I have the feeling that this thread, starting from "archive or not the classic notebook repository" is pulling a lot of broader considerations, but in this case, maybe "Everything is in Everything"?

afshin commented 2 years ago

I think there are a couple things we ought to state upfront in order to alleviate concerns about what is to come:

Everything else, on some level, is implementation details. We do not want to disrupt workflows for the millions of users who depend on the tools we build.

If you missed it above, please reconsider Jeremy's screenshot. It is clarifying in a way text is not:

I'll reiterate: One of these is built with libraries that come from JupyterLab. The other is the classic Jupyter Notebook.

If you have to peer very closely to tell the difference, that is the point. This is a tractable problem: give users the simple user interface they want built with a modern, composable components that we can maintain and keep secure while supporting the latest and greatest. Here is a thread where we are discussing how to make this UI part of the JupyterLab core offering (and potentially even to make it the default UI a new user encounters when opening JupyterLab).

I know our users and the community at large depend on and have affection for the Jupyter Notebook. We do, too. That's why we've been doing our best to maintain this repository, keep it secure, and slowly move it forward toward a happy path where the Jupyter UIs can be coherent and easy to reason about.

astrojuanlu commented 2 years ago

To summarize some of the points raised here, some users want to keep

https://github.com/jupyterlab/retrolab/issues/257 will surely help users discover RetroLab, especially if it becomes the default (+1!)

The other two points are related. Perhaps there could be a list of popular extensions somewhere that need help with migration, or new alternatives to old extensions. And clarity on whether ipylab is in a state that can be used by extension writers.

To add another data point, I know perfectly how to spit custom HTML to a classic notebook, but I don't have the slightest idea of how to do that with JupyterLab. Some friends tried for many years and gave up, because they're Python folks that don't understand frontend development, don't care about learning "the proper way", and got burned out by the changes on the underlying mechanisms.

As a user, I was thrilled to see prebuit extensions land at last so I don't have to know a single bit of Node.js to use Jupyter (in fact I was waiting for it to start taking JLab seriously) and I guess this convenience hasn't translated yet to extension writers.

Carreau commented 2 years ago

sorry to put you on the spot here, and please feel free to ignore me - but if someone waved a magic wand, and gave you the power to decide where the Jupyter project would go next, what do you think you would do?

and

  • I remember very early discussions around the UI, where a simple config file had been promised so eg you could trivially define a "just a file dir" and "just the notebook" views.

To respond to both of those, I don't think I have a specific end state, but that there should be a better transition. As pointed by other folks the initial plan for what became JLab, was to get a identical experience to classic notebook. Whether classic notebook UI/keyboard are good or bad, I still think it is important for adoption because many human don't like changes. We've seen it for Python2 ->3, and matplotlib theme change, except matplotlib was handled better as you still ad the ability to switch to identical classic theme. As a physicist I would really have loved a quasi static transition.

The threshold where people realized that things have changed depend on who you talk to. I can't type on a mac with touch bar cause I miss escape and know the logic behind tab completion in classic so well that any change between it and lab disturbs me (lab completion is objectively better, but still contribute to my difficulty migrating because I'm too used to old behavior). On the opposite end my grandmother is supper happy her "windows 10" is faster since she got a mac...

For me the things that still bother is that single document mode is still not really single document. As other pointed out, I would prefer to have multiple documents and tabs handled by my browser and not lab. That include separate files navigation, and links open in new tabs. I haven't use retrolab enough to judge – and my muscle memory is still typing "jupyter notebook" which i'm sure is part of the problem.

I also don't believe that the "you can switch to single document mode and hide sidebars" is the right answer, it's like people telling you "you can install vim and configure it to behave like a normal text editor".

The default matters a lot, and you really need to subtle changes if you don't want to break anybody's workflow (xkcd).

There is some discussion in https://github.com/jupyterlab/retrolab/issues/257, and maybe retrolab is the right intermediate step.

That piqued my interest - I don't follow .Net - what happened there?

There are actually two things:

matthew-brett commented 2 years ago

@Carreau - I am sorry to ping you again, but I was very interested in this:

I have to touch the source rarely but everytime do it takes me hours to get it to properly work when build from source. I had to patch an extension for a client so I had to track time, and it took me 3:30 to get it to build properly.

I have also heard before, from experienced developers, that writing JupyterLab extensions is very hard, and that seems to be a common message from this thread.

Could you say something about why this is - and what you think could be done about it? Can it be easily addressed with the current architecture, or will it require more major changes?

benbovy commented 2 years ago

I have also heard before, from experienced developers, that writing JupyterLab extensions is very hard, and that seems to be a common message from this thread.

I can share my experience as someone who recently worked on a couple of ipywidgets extensions (not jupyterlab extensions, but I assume both have a rather similar development process), with no prior experience in developing front-end libraries / applications apart from hacking with some poor javascript.

I might be wrong, but I don't think that writing jupyterlab extensions is harder than writing any other "production" lib or app using a modern stack of front-end libraries, frameworks and tools. The current front-end ecosystem may look scary for newcomers (that was my experience) and the learning curve is quite steep. This may be sometimes frustrating, but in the end this is also very rewarding to acquire some new, broader skills (of course if we have time and interest to invest in that).

Extensions cookiecutters (link) greatly helped me to get started and to avoid spending too much time on figuring out how to write all the boilerplate for a new extension. It actually helped me more than the documentation.

Upgrading the extensions to new versions of JupyterLab (e.g., 2 to 3) was not super smooth but it didn't cause me major trouble either. I guess that this will stabilize more, especially since JLab 3 and the federated extensions mechanism. Extension cookiecutters seem to be updated regularly so that may help too.

The good thing with extensions built this way is that if I want to distribute them to the broader Jupyter community, I'll be more confident about their robustness and compatibility with all user environments.

So in summary, I'd answer that, yes, it is hard to write "in-production" Jupyterlab extensions just as it is hard to write any other software meant to be used in production, but there's already some tooling available to make the process easier.

I think it is also important to distinguish between "in-production" (large) extensions and (small) extensions meant for personal use (or shared with the community with a "use-at-your-own-risk" notice). Premium support should exist for both kinds of extensions IMO. It looks like there's not much infrastructure available yet for developing the latter kind of extension, but ipylab and other things mentioned in @jtpio's https://github.com/jupyter/notebook/issues/6210#issuecomment-951890332 seem promising.

lbignone commented 2 years ago

As an end user what is stopping me to switching completely to JupyterLab is that the notebook experience is not polished enough.

For example. The default behavior with a single notebook open is to have the text fill the full screen width. That make it difficult to read on a normal monitor. The classic notebook has by default a more reasonable width. To me that contributes a lot to its look and fell.

To activate word wrapping on Lab you have to go to the advance settings, which are confusing for most users, and even then to my knowledge you can't wrap rendered markdown text .

It is not that I want to have exactly the same user interface as the classic notebook, I welcome changes that bring new features and usability.

For example. Creating new views from outputs is great and really showcases the capabilities of the multi-panel interface. Except that you can't rename output view tabs, which really limits the usability of the feature. Both RetroLab and simple view currently remove the feature completely.

In my opinion, If the normal JupyterLab interface paid more attention to usability refinements there would be less of a need to recreate the classic interface and current classic users will have less problems switching.

rabernat commented 2 years ago

Just chiming in very briefly from the perspective of the Pangeo community, a large open science group in the geosciences: our community loves JupyterLab and uses it heavily! Extensions like the JupyterLab Dask Extension enable very rich user experiences that would be impossible in classic notebook. I recognize the good points raised by the classic notebook advocates and see the need for a stable, realistic sustainability and / or migration plan. But I push back strongly against the suggestions that the Jupyter community has somehow failed by putting effort behind JupyterLab. From my perspective, JupyterLab is a huge step forward from classic notebook.

matthew-brett commented 2 years ago

But I push back strongly against the suggestions that the Jupyter community has somehow failed by putting effort behind JupyterLab. From my perspective, JupyterLab is a huge step forward from classic notebook.

I feel the pushback! But, in case it wasn't clear, I am certainly not saying that JupyterLab has failed, and I don't think anyone else has either. And of course, 'failed' or 'succeeded' are, generally, very murky questions. The question is not "is JupyterLab good?", or even "is JupyterLab better than the classic notebook?". For me, at least, the two questions are:

The problem is, that the issues get very cloudy if the argument starts getting tangled up in praise and blame.

Carreau commented 2 years ago

Could you say something about why this is - and what you think could be done about it? Can it be easily addressed with the current architecture, or will it require more major changes?

I think it is inherent complexity due to the typescript/javascript that need to work with Python, and the wrappers around this. It might actually be easier to create an extension from scratch with a stable version of JLab than to be on dev versions with editable install. But honestly I don't believe I'm the right person to ask as I don't know enough about what's happening under the hood.

I think that may also be indirectly due which type of machines you are using, I got a new one yesterday, and during lab startup I now see the jupyter logo spin not even one full revolution instead of three, and will make quite the difference when I have to iterate on an extension.

For me, at least, the two questions are:

  • Is it possible that there are two sets of users here - one of which is well-served with the simplicity of the classic notebook...

I'm fairly convinced that is true. And I'm also quite convinced that this may even depends for a single person on which project they are working on.

...and less well served by the additional complexity of JupyterLab?

I guess here by complexity you mean number of element in the UI, and not complexity of how to write extensions ?

And if so, does this subset of users need to find some way of keeping the classic notebook going, or planning for a replacement that is a better match for what they need?

As a codebase, I don't think so. I think that classic notebook codebase really need to die. As a UI, I think it is still important.

  • Is it possible that some of the complexity in JupyterLab results from architecture or design or user interface decisions that should be revisited?

Architecture and design personally I don't think so, this is a technically sane and maintainable codebase. UI I would put it another way. I think that going from current retro-lab to jupyter-lab layout and vice-versa should be achievable almost only via configuration options, and that jupyter should have sort of "distributions" (ok, really bad names), that are opinionated starting points for users. Basic, Full, ML, image-processing, education... I think that is achievable. Which is sort of already the case as JupyterLab is a set of plugins, we need to find sufficient manpower to get it sufficiently easy for end users to get the one they need, and instructions & branding clear enough to distinguish both.

matthew-brett commented 2 years ago

@Carreau - thanks - that's very helpful, and encouraging.

I think that going from current retro-lab to jupyter-lab layout and vice-versa should be achievable almost only via configuration options, and that jupyter should have sort of "distributions" (ok, really bad names), that are opinionated starting points for users. Basic, Full, ML, image-processing, education... I think that is achievable.

What do you think needs to happen, in order to make this possible?

And - what about the difficulty in writing simple extensions, such as those being used by @psychemedia ? Are we in sight of a practical solution to that problem? And if not, how do you think we should get there?

rgbkrk commented 2 years ago

I think that classic notebook codebase really need to die. As a UI, I think it is still important.

This is exactly my sentiment as well.

labarba commented 2 years ago

Hi everyone 👋

I arrived here from the link on the Jupyter Community Forum email digest, which had a post by Zach with a link to this thread. I started reading and my hart sank, with deep concern—I got about 3/4 of the way down, reading and worrying. I am an engineering prof and I use Jupyter (classic) every day. I've taught and continue to teach dozens and dozens of engineering students that coding is for them, and I have written a large amount of (open) educational materials using Jupyter notebooks. Over many years, I have developed a workflow that is dependent on nbgrader, including raising funds to (pay others to) develop integrations for an open learning management system (Open edX). My way of teaching would be nuked out of this earth without classic notebooks, and I would have to rethink everything. Years of investment: gone. Back to square one. It's terrifying to think of the amount of time it might take for me to rebuild a workflow with auto-grading assignments and all that. I'm aware that other tools exist for this, but I'm unfamiliar with them, so it's still back-to-square-one.

I am not a developer, but over the years I have volunteered hundreds and hundreds of hours to the community, organizing, leading through the NumFOCUS board of directors for seven years, leading JupyterCon 2020 in extraordinary times, and making all of my educational content always open and reusable by others. As an early adopter, vocal promoter, and prolific writer of Jupyter-based courses, I know for a fact that my work has had a positive impact in the Jupyter community. Please don't throw my work out the window without giving it a second thought.

Carreau commented 2 years ago

@matthew-brett

I'd like to reframe the concerns of people that are afraid of loosing the classic notebook in increasing difficulty to achieve:

  1. They don't want to loose single page notebook UI.
  2. They don't want to loose existing extensions they are using for single page notebook UI.
  3. They don't want to rewrite existing extensions they already wrote for single page notebook UI.
  4. They don't want to learn a different API to write new extensions.
  5. They don't want the new API to be more complex.

I believe each item above is subset of the precedent, and that they should be tackled in this order.

And - what about the difficulty in writing simple extensions, such as those being used by @psychemedia ? Are we in sight of a practical solution to that problem? And if not, how do you think we should get there?

It sort of depends what you mean by basic/simple. Do extensions not do much? Or are the extension composed of relatively few lines ? I believe that there is always the possibility to have some categories of extensions than can be make easy to write, but as always it's a tradeoff.

Jupyter notebook classic has the tradeoff that everything and anything can be done by extension, so the codebase is stuck. Advantage for classic is that extensions don't need lots of boiler plate, though you get the N^2 problem, that any extension can conflict with any other. Generic JupyterLab extensions then have a larger threshold if you want to do everything and anything. Though if your extension want to relinquish some capabilities, then it might be easier to setup. A good example is mime-renderer. Mime renderer extensions are dead-simple, but can't do anything else than rendering.

So for "simple" extensions the questions is what are the categories of simples extensions, and for each category how can we make that simple to write ? That will likely mean more complexity in the core codebase though, but you need those categories.

In general I'm also fairly certain that we would have higher quality and more extensions if the ecosystem was less spread around, as you would have more consistent docs, and more users. But I don't think tackling how to write simple extensions should be the first item.

My guess is that large majority of people who say 5. , really don't want 3. And that once you are in a ecosystem where many extension exists, you just fork one that exist and modify to suits your needs, so never really need to learn the full API.

Though I believe that for most existing extensions 3. it might be way less work than we think to port them as the behavior is already well defined and scoped. If you stick to doing what an already existing extension does, and not try to make it better (like you know any of us do when we rewrite code), it is completely doable, and might be even more approachable to new contributors than inventing a new extensions. The original authors are just the worst kind of person to ask to rewrite. It's also much easier to outsource that kind of work.

That's where I believe a strong leadership/community manager could have much impact. It's a really hard position as you have to say no a lot and convince people to work on things they don't like to reach another milestone they are interested in. And while it's hard to get funding for such a person, it's not impossible (as shown by the recent CZI grants).

cezarymierzejek commented 2 years ago

I saw this thread shared on twitter and wanted to share some longish thoughts as a huge jupyter fan and lurker based on my experiences as a data science lead in a large tech enterprise.

re: adoption of notebook vs lab

based on other comments here, I can definitly corroborate that many folks confuse jupyter notebook and jupyter lab in surveys. having done some detailed surveys of these in the past, something like 50-75% of the folks would select jupyter notebook even though they were using lab because they interpreted the question differently. to put this in exact terms, if the survey came out 50 for notebook and 50 for lab the true result was 20ish for notebook and 80ish for lab (based on direct 1-1 follow up).

this is also reflected in the way that major firms in the enterprise data space are selling jupyter flavored services as well. AWS Sagemaker, Google Cloud, IBM (where I am), and Cloudera (among others) all default to JupyterLab for their jupyter-as-a-service products or plan to do so. my experience here is that enterprise IT is kind of like the underwater iceburg portion of jupyer users. as more companies are moving their data science workloads to the cloud they often default to providing lab since it comes with everything a data scientist/analyst needs to be productive (notebooks, text editor, terminal, debugger, git support etc).

generally the biggest holdouts for classic notebook in my experience are folks in the education space (educators, professors, etc) due to completely reasonable concerns about needing to go back and update lessons plans, videos, and educational materials due to the interface changes and double checking that students can replicate the outputs of the code consistently.

the one area that I think could be improved to help adoption is marketing jupyterlab in a much more prominent way within the classic interface especially with the use-cases it supports. I find that if folks were not introduced to jupyter lab directly then they had no idea that it existed and simply stayed with classic and when they do adopt lab, are not always aware of what is possible (e.g., using shift enter on a text file will send code to the console is something many people discover by accident).

I believe this is more of a behavioral issue because people are used to being prompted to update their software these days vs proactively seeking out updates (vscode and its extension updates or pycharm updating itself are good examples). So something that prompts users within classic to check out jupyter lab (maybe a non-intrusive banner?) and perhaps provide a link to take feedback would be a good place to start.

re: productivity and complexity

personally, I have never encountered anyone that switched back to classic once they had a chance to try jupyterlab in the past ~1.5 to 2 years or so. Things like being able to link notebooks and consoles to the same kernel, viewing the same file in multiple ways, git integration, debugging capabilities, ability to view multiple files (and file types) at the same time provide enormous leaps in productivity. This isn’t scientific but switching a cohort of classic notebook users to lab led to a 30% decline in failed notebook jobs on a past team I worked on. The reason was that when they were editing and debugging large, long running, notebooks they would attach a console to the same kernel when trying to prototype new data transformations / debug issues instead of editing the notebook itself where they would invariably leave ad hoc code in the notebook by accident.

I think the complexity point that folks are raising is a fair one but I also feel like it reflects the changes in the state/maturity of the field of data science and data analytics. Back in 2014-2015 running a notebook with scikit learn as a nightly job was a game changer and nowadays you get taught that in ML 101. The bag of tools I see folks use day to day usually include VS Code, PyCharm, RStudio, database IDEs (e.g., Dbeaver) and I do not see Jupyterlab as being inherently more complex than any of these nor have I seen folks struggle with Jupyterlab in particular.

I do believe JupyterLab’s capabilities and workflows are somewhat hidden to new users making it difficult to tap into all of the super powers available. my recommendation here would be to aggressively lean into the command palette (e.g., a sample command that launches a notebook and a console attached to the same kernel). The command palette paradigm is becoming more and more common across tools that data people use anyway (VS Code and PyCharm have it, RStudio recently added it as well) so it requires little additional learning.

If jupyter sticked with just the notebook interface then I think it would eventually suffer the same fate as sublime text at some point (still a great and fast editor but by sticking to being an editor it invariably got overtaken by VS code).

I hope some of these comments are helpful to the discussion. As with any perpsective, mine is coming from a very specific background so does not apply to folks working in very different environments.

matthew-brett commented 2 years ago

Things like being able to link notebooks and consoles to the same kernel, viewing the same file in multiple ways, git integration, debugging capabilities, ability to view multiple files (and file types) at the same time provide enormous leaps in productivity.

I think that does relate to the different use-cases that you've indentified. I don't know, but you may have identified the use-case JupyterLab was particularly desiged for - a data scientist who spends does a lot of their work in notebooks

And I think you're right - that's not what I do as a teacher - or as an experienced Python coder. From talking to other people - including Jupyter developers - I think I'm reasonably typical - in that I use notebooks as sketches, and as tutorials and demos. I often buff-up a notebook sketch by converting to RMarkdown, going into my text editor and spending a lot of time editing there. I very rarely debug in the notebook, because if the code is at all substantial, I switch to my text editor again, and put the code into a .py files where I can edit them more easily, write tests, use standard version control. I find the notebook very uncomfortable for working with any significant amount of code, and I have trained myself to switch out of the notebook quickly when I realize that is happening.

For those cases - the notebook we need is something much lighter. It's not meant to be a platform for your coding life, but a gateway to other things. But a gateway that is of enormous use and benefit. I think you're right to draw that distinction, and you can see it everywhere in this thread, but particularly from comments by @psychemedia and @Carreau.

Carreau commented 2 years ago

Today's governance call was used to discuss many of the above topics. It just finished a few minutes ago, and we're going to posts the notes we took during this call after a bit of necessary cleanup.

cezarymierzejek commented 2 years ago

Things like being able to link notebooks and consoles to the same kernel, viewing the same file in multiple ways, git integration, debugging capabilities, ability to view multiple files (and file types) at the same time provide enormous leaps in productivity.

I think that does relate to the different use-cases that you've indentified. I don't know, but you may have identified the use-case JupyterLab was particularly desiged for - a data scientist who spends does a lot of their work in notebooks

And I think you're right - that's not what I do as a teacher - or as an experienced Python coder. From talking to other people - including Jupyter developers - I think I'm reasonably typical - in that I use notebooks as sketches, and as tutorials and demos. I often buff-up a notebook sketch by converting to RMarkdown, going into my text editor and spending a lot of time editing there. I very rarely debug in the notebook, because if the code is at all substantial, I switch to my text editor again, and put the code into a .py files where I can edit them more easily, write tests, use standard version control. I find the notebook very uncomfortable for working with any significant amount of code, and I have trained myself to switch out of the notebook quickly when I realize that is happening.

For those cases - the notebook we need is something much lighter. It's not meant to be a platform for your coding life, but a gateway to other things. But a gateway that is of enormous use and benefit. I think you're right to draw that distinction, and you can see it everywhere in this thread, but particularly from comments by @psychemedia and @Carreau.

Completely agree with your comments re: the different use cases, users, and what they are looking for.

I think part of what is fueling the debate on what direction JupyterLab (or Jupyter in general) needs to take is that it typically operates in close proximity with other tools in many data science/data analytics workflows. This is in contrast to what you see in similar communities (e.g., RStudio, Stata, Matlab) where typically there is 1 tool/tool chain to rule them all.

Most people I see pair JupyterLab with either VS Code or PyCharm for the exact reasons that you described - as certain pieces of code or logic get matured in a notebook it eventually makes more sense to transition and package them into .py files.

What I think has changed is that these adjacent tools, which largely ignored notebooks in the past and were primarily used in this transition and packaging step, are now trying to take a bigger bite of the notebook workflow. VS Code and PyCharm being the most prominent examples.

I personally favor the “platform for your coding life” that you mentioned but can understand that is a very unwelcome direction for those who are looking for a more lightweight tool. Optimistically, I think projects like retrolab look like they can get there and hopefully a solution can be found for those critical extensions only available for classic notebooks.

matthew-brett commented 2 years ago

I personally favor the “platform for your coding life” that you mentioned but can understand that is a very unwelcome direction for those who are looking for a more lightweight tool.

Well - also - I really wonder whether it is realistic for JupyterLab to compete in that space, when VSCode - for example - is already so prominent there, and has shown so much interest in developing tools for working with notebooks.

cezarymierzejek commented 2 years ago

I personally favor the “platform for your coding life” that you mentioned but can understand that is a very unwelcome direction for those who are looking for a more lightweight tool.

Well - also - I really wonder whether it is realistic for JupyterLab to compete in that space, when VSCode - for example - is already so prominent there, and has shown so much interest in developing tools for working with notebooks.

It’s a fair question.

I feel more optimistic that it can because in my experience VS Code’s implementation of notebook capabilities (at least the last time I tried it a couple of months ago) are quite buggy. 


On paper, VS Code appears to have replicated many features of classic notebook/JupyterLab or have been faster to implement a subset of long standing asks (e.g., variable viewer like RStudio). In practice, they break / crash quite often when working with datasets of any decent size. e.g., 1 GB or bigger (which I consider quite small).

It’s possible that VS Code will fix many of these issues eventually but I think that is a big if given that its primary audience is developers working with text files. I wonder how much fixing these issues requires core changes in VS Code and potentially degrading their experience vs what is possible to do as an extension (e.g., popular features like shift tab completion being open issues for 2+ years makes me believe it is harder to copy what Jupyter can do than it looks)

My guess is that these limitations are part of the reason why they are focused on the volume of features e.g., try to match 100% of what classic/lab can do even if they are only 50% as good in practice. The benefit of this approach is that they can scoop up the jupyter users who are running very simple notebook workflows, but also represent a big population.

Overall, I think that there is sufficient difference in use-cases between doing data/analytical work with pure development work that I think JupyterLab can compete. And at bigger data scales these differences are much more substantial.