New project UX is too slow (unusable) when fetching list of repositories.

[gtsiolis]

Problem to solve

Currently, during the onboarding flow, fetching all repositories can take a while which can be misleading for some users and affect user experience.

Proposal

We could introduce an initial loading state when the request takes longer than X amount of time, paginate the list of repositories, and load more list entries with infinite scrolling.

Cc @AlexTugarev

TBD (To be discussed)

[jankeromnes]

(Side note: I don't think pagination will provide enough of a speedup. When we request uncached data from GitHub or GitLab, it seems to be always slow, regardless of the size of the list you're requesting -- it's slow even if it's just one item.

I believe the more effective performance fix should be to cache the results, so that we load all data from our DB (fast) and not from a third party API like GitHub or GitLab (slow).)

[jldec]

Adding this to groundwork inbox for engineering investigation to better understand scope, and determine whether this could work without new caching infrastructure or whether it necessarily depends on something like Redis.

[jldec]

Similar to new workspace modal

Could we show a subset of repos in the list (with minimal delay), and then also support repo-URL paste from clipboard. i.o.w. treat the repo list as an auto-suggest rather than requiring all new project repos to appear in the list first.

[jldec]

It appears that when there are enough repos, the UI breaks completely. Adjusting priority and rewording description.

See support thread (internal)

[jldec]

We should at least offer a workaround to create a project by entering a repo URL instead of choosing from the list.

[gtsiolis]

It appears that when there are enough repos, the UI breaks completely.

Do we have a case where the UI breaks? The repositories list eventually always loads, correct?

Also, could it help to first improve the interaction, accessibility, and perceived loading time of the loading phase as a minimum viable change, by reusing the loading element from the branches, etc?

BEFORE	AFTER

[jankeromnes]

Do we have a case where the UI breaks? The repositories list eventually always loads, correct?

FYI, I remember creating an org with 3500 repositories in it (under our GitHub Enterprise instance/user) because a customer was complaining that with several thousand repositories, the UI would take too long to load.

But actually, trying this now, the UI did take a little bit to load but wasn't too bad, and I do get the full list of repositories (all 3500+).

We should at least offer a workaround to create a project by entering a repo URL instead of choosing from the list.

I absolutely agree that we need this workaround somewhere. It doesn't need to be too prominent, but we've seen that in various edge cases, users cannot get past the "New Project" screen:

• When there are too many repositories (I guess more than 3500?)
• When there are many repositories with the exact same name (e.g. you have 200 orgs that all have a repo called "config" 😬 this actually happened)

So, allowing to input a repository URL sounds like a simple way to cover these edge cases where the UI fails.

Also, could it help to first improve the interaction, accessibility, and perceived loading time of the loading phase as a minimum viable change, by reusing the loading element from the branches, etc?

Hey, this looks nice ✨ and perfectly conveys what is actually happening (i.e. it's not Gitpod being slow 😅). I love it ⬆️

[jldec]

Scheduling this because it's one of the most urgent dashboard UX issues and can prevent project onboarding.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[github-activity]

test

[gtsiolis]

Coming back to this as the proposal described in https://github.com/gitpod-io/gitpod/issues/5075#issuecomment-1134456530 has been added in https://github.com/gitpod-io/gitpod/pull/12833.

See also relevant issues with the new proejct flow.

• https://github.com/gitpod-io/gitpod/issues/9112
• https://github.com/gitpod-io/gitpod/issues/8524
• https://github.com/gitpod-io/gitpod/issues/8597

Removing this from the product design board for now. 🥯

In the meantime, cross-posting a relevant discussion (Cc @AlexTugarev @jankeromnes) which contains some potential next iterative steps:

... maybe we could go with a boring solution to :a: default to your personal account first and :b: load in the background the rest of your groups or orgs. See design drafts below.

Cc @geropl @jldec

Dropdown (Loading Pending)	Dropdown (Loading Complete)

[atduarte]

👋 Coming back here as I learned from a customer that this was blocking the creation of projects and therefore the use of prebuilds and workspace classes for some.

We should at least offer a workaround to create a project by entering a repo URL instead of choosing from the list.

I also believe this is the right 🛹 . @gtsiolis what do you think? How would this look like?

[mbrevoort]

Can we start caching the list of projects instead of having to fetch them from the SCM in realtime?

The easiest would be to cache them client side, load view from cache, fetch update async and update client when they arrive. This doesn't solve the FTUX but would solve most other use cases.

Alternatively, cache server side (e.g. in Redis). When request is made, if in cache return it and async fetch update of repo list to update Redis. On subsequent request, return values in Redis. This would allow us to lazy load as you type (fast from Redis) and not have to load 1000's of repos while balancing the edge case where you just created a repo and want to find it. We would have to cache per-user.

The more complex alternative would be to keep a live sync in cache, in the database and subscribe to updates in Redis. This would require us to either keep a separate collection per-user or keep a copy of ACLs.

[gtsiolis]

@mbrevoort Re: Caching would be nice and has been brought up a couple of times in the past. Not sure what it would take to introduce some kind of caching, but I saw that @easyCZ recently added Redis support in https://github.com/gitpod-io/gitpod/pull/16327—maybe this help push us towards a viable solution. 🤷

We discussed with @atduarte about this problem earlier this week (see relevant discussion (internal)) and worked on how this could look like if we did cache all the available projects.

Inspired after the work https://github.com/gitpod-io/gitpod/issues/15979 (Cc @svenefftinge), this could use a plain search input where users could search for all available repositories as well as paste the full repository URL, similar to the new workspace modal, while having a different JTBD (Job-to-be-done).	New Workspace	New Project

Other ideas discussed as potential MVCs, following up from https://github.com/gitpod-io/gitpod/issues/5075#issuecomment-1337430977:

1. Default to your personal account as the default selected option
2. Load the rest of the Orgs (GitHub) or Groups (GitLab) in the background.
3. Moving the fetching action after selecting a specific Org or Group.

[atduarte]

👌 @gtsiolis. One thing I like about this new method is that even if the repositories are loading, the user can still input the full repository link and go ahead with the project creation.