[ENH] - Extension to present and pull a set of examples/gallery

[krassowski]

Feature description

Nebari provides jupyterlab.initial_repositories option which allows to pre-populate user space with content needed for them to start work, or to explore pre-defined examples:

https://github.com/nebari-dev/nebari/blob/7d0c28ef2815943f528f54541fd55e8db0c62fc9/src/_nebari/stages/kubernetes_services/__init__.py#L238

However, this requires cloning the entire set of repositories each time, is problematic to synchronise the repositories when they change, and does not provide the user with an interactive way to explore the repositories ahead of time.

We could create a JupyterLab extension that would present the examples, either in the sidebar or in the main area, together with rich metadata (icon/screenshot, title, description) and allow user to setup the example in the background.

Value and/or benefit

• we would avoid space disk inflation due to cloning all the example repositories into the folder of each user
• users would be able to explore the available examples in a gallery
• the gallery could be governed by a separate yaml/json file that could be updated independently of the deployment cycle
• in the future we could give users control over whether they want to sync the example repository with the latest upstream version (for example with git stash/git pull/git stash pop sequence)
• this extension could be co-maintained with broader Jupyter community as it would be also useful for the JupyterLab extensions examples
• extending the example setup process beyond the simple git pull would allow to also setup, or activate any required conda environments if associated with the example repository

Anything else?

There are existing JupyterLab extensions which use git for cloning or fetching repository content:

• jupyterlab-git
• jupyterlab-github
• jupyterlab-gitlab

While there is an overlap with these extensions, the proposed gallery has specific presentation and user interface requirements which set it apart from the existing extensions; that said, we could leverage these extensions to simplify the backend implementation.

The extension could support PAT or other authentication mechanisms; the entire sequence of pulling (and optional auth) would happen in a jupyter-server extension, so the secretes would never be exposed to the end user.

[viniciusdc]

I think that's an excellent idea, and while the current superlab.initial_repositories overlap with the extension in some senses, I believe they aim to solve two different scenarios (as you described above). And we could leverage/document the current workflow to the users:

• The current available initial_repositories would be used by sys admins to pre-populate user-required instructions/data;
• The new extension would be used by users in their ecosystems/workflows whenever needed;

[dharhas]

Also similar - https://github.com/jupyterhub/nbgitpuller

[krassowski]

Two questions to think about:

• UI: we can put the gallery in the sidebar, or in a main area widget (like Launcher/notebooks); (we can have it configurable - there is not much overhead, but there is some); the tradeoffs are:
  • sidebar:
    • nebari already has a number of extensions in the sidebars
    • the presentation of examples (icon size, description length) will be limited by the sidebar size
    • we will need to style the gallery responsively for minimum width of sidebar the user can set
  • main area widget
    • it can either open automatically (will annoy users who do not want to use it at the moment), or it would need to be open manually either from a menu or from a button added somewhere in the UI (e.g. top bar next to menu, filebrowser sidebar, status bar)
    • it can present much longer descriptions and more metadata about the repositories
  • the configuration file with links to repositories, descriptions, etc can be either a custom toml/json/yaml file or a traitlet (py/json file); the question here boils down to preferences on file format, and:
  • if we use a traitlet we can re-use the jupyter-server config files (py/json); the python files could read in the PAT token from environment variables and embed it in the repository URL used for cloning
  • if we use a custom file (toml/yaml/json), we will need to implement variable substitution

[kcpevey]

sidebar vs main area widget - I think its important that its obvious to users that the examples exist. One concern I have is that if its just an icon on the left (e.g. the dask icon for the dask extension), it will mean nothing to users and they will never discover the examples. I can see how a main area widget opening automatically would annoy people (me).

What are the other options for the sidebar? We could have a reserved area like the "favorites" section but that would be very limiting.

What about adding a top menu item called "Examples" or "Example Gallery"? That would be pretty visible and then if clicked, it would open in the main area and we would have lots of space?

[krassowski]

What about adding a top menu item called "Examples" or "Example Gallery"?

On smaller screens the menu gets cut off already, so not sure here.

Some other ideas:

• we could add a tile in the Launcher
• we could add a button in the file browser
• we could add a menu entry in "Help"

I think these are not mutually exclusive (nor is the sidebar idea).

[kcpevey]

I like the tile in the launcher idea! Would that be compatible with the new launcher?

[krassowski]

Would that be compatible with the new launcher?

Technically - sure. Conceptually? Not sure. Here is what I mean:

• if it goes where "Notebook"/"Console"/"Python File" etc tiles are, then this section no longer aligns with "Create Empty" (we already have this problem with VS Code tile though, see https://github.com/nebari-dev/jupyterlab-new-launcher/issues/35)
• if it goes to the top-right corner where we put "contextual help" and "scheduler" tiles, then it would be not very noticeable

I think there is a lot of value in having "Create Empty" as it is, even moving VS Code out of there to make it conceptually simple. Still, it would make a lot of sense to have a place to expose "VS Code" and "Gallery" tiles on the launcher. Not sure how to make it work in the layout yet.

[krassowski]

Strictly, I would think gallery tile should go to the top-right corner. Maybe the solution is having multiple ways to access it.

[krassowski]

Also similar - https://github.com/jupyterhub/nbgitpuller

It looks like we can reuse the GitPuller class for the server extension easily. Do we want to have the merging mechanism as implemented by nbgitpuller? For reference see https://nbgitpuller.readthedocs.io/en/latest/topic/automatic-merging.html

The alternative is using GitPython which can give a bit nicer progress but the logic for merging would need to be re-implemented. I do not think that using plain git CLI is wise except for very simple things.

[dharhas]

I see two options related to putting it in the "new launcher"

1. A single icon saying "Download Examples" that opens the example browser in a main windows
2. A new expandable section/row called "Gallery/Examples" in the launcher that shows all the available examples.

[dharhas]

Do we want to have the merging mechanism as implemented by nbgitpuller?

Their strategy seems reasonable. But it would be nice to have a progress bar.

[dharhas]

Strictly, I would think gallery tile should go to the top-right corner

Maybe part of the solution is to make those icons in top right corner larger and more obviously discoverable also including the text (gallery, context help etc)

[krassowski]

A new expandable section/row called "Gallery/Examples" in the launcher that shows all the available examples.

This sounds great for when the new-launcher extension is available; if it is not installed we can just add a tile which would open a main area widget in the default launcher.

But it would be nice to have a progress bar.

To clarify nbgitpuller does support some form of progress reporting, but as far as I understand it does not give us a smooth progress bar when cloning large repositories. Though if we end up needing that smooth progress bar, then we can override the public initialize_repo() in a subclass and still defer to nbgitpuller for merging changes afterwards.

[krassowski]

I got a backend re-using nbgitpuller working in https://github.com/nebari-dev/jupyterlab-gallery. There is more work needed on reading and processing the configuration file and then on the UI.

The way nbgitpuller fakes progress is actually somewhat funny.

[kcpevey]

I have a slight concern about using nbgitpuller for private repos. It looks like their recommended way of creating the links is to put it their web UI (https://nbgitpuller.readthedocs.io/en/latest/link.html) which would not be secure. This is a longstanding issue in the issue tracker.

[krassowski]

We are and will not be using any of nbgitpuller web UI. Further, we can override methods in our subclass of GitPuller if we need so.

[krassowski]

Some thoughts after completing an initial implementation that puts the gallery in the new launcher:

• we may want to have a very subtle progress indicator, and possibly move the detailed reporting into a notification in the corner; ideally a progress bar but spinner would be ok for now (https://github.com/nebari-dev/jupyterlab-gallery/issues/6)
• I envision placing the setup/open/update button as an overlay over the icon (they would be icon buttons + text on hover)
• we may want to allow changing the title "Gallery" to "Examples" in settings
• the MDI gallery icon is ok for now but not very appealing, it would be great to have something better (https://github.com/nebari-dev/jupyterlab-gallery/issues/7)
• if we place the example in the launcher, having a longer description may look out of place; maybe it could show up on hover or be hidden behind an information icon ⓘ
• it would be nice if tiles with examples to matche the size of tiles in "Launch New", or were exactly twice as big

Switching gears to work on backend, while waiting for any potential discussion/ideas on UI/UX

[kcpevey]

Some questions on usability:

• Can users specify where to clone the repo? Or do ALL repos clone into the same base folder? How can we raise that information to the users.
• If I've already cloned the repo and I want to pull and update, how would I do that? Maybe a second button next to "open" for "update".

progress indicator

responded in the issue

MDI gallery icon

responded in the issue

"Gallery" vs "Examples" customization

I agree, for my usecase I'd like "Examples" but I can see many other usecases where "gallery" would be more appropriate. It would be nice to set this as the admin level.

it would be nice if tiles with examples to match the size of tiles in "Launch New", or were exactly twice as big

Right now it looks like they are roughly the size of 2x2 of the other tiles which lines up nicely. I'm happy with that and I think it gives enough extra space for more information without being too large.

if we place the example in the launcher, having a longer description may look out of place; maybe it could show up on hover or be hidden behind an information icon

I think for this we just need to decide on how to handle overflow, right? So maybe it can show as many characters as will fit in the 2x2 square above and then have ... and the info bubble to click on for more? (just brainstorming and I dont know the LOE for those ideas)

[krassowski]

Can users specify where to clone the repo? Or do ALL repos clone into the same base folder? How can we raise that information to the users.

In my current thinking the destination is owned by the server, and can be configured by admin only. I think we can make this user customizable if this is desirable; it is a little bit more complicated as would now need to track where the repositories got cloned, but something that can be done in half a day.

[kcpevey]

In my current thinking the destination is owned by the server, and can be configured by admin only. I think we can make this user customizable if this is desirable; it is a little bit more complicated as would now need to track where the repositories got cloned, but something that can be done in half a day.

I'm ok with that for the POC given the LOE and its a "nice to have" anyway. We can try this out as-is and add it as a new feature in the future if there is a significant interest.