Epic: JetBrains Prebuilds for Maven

[loujaybee]

Is there any kind of guidance how handle importing+indexing in the prebuilds? Our project is very large and if opened from scratch would likely take IntelliJ 10+ minutes to download and index all dependencies (source)

We haven't yet investigated this, this ticket is a placeholder for any changes required on our side to make indexing easier for IntelliJ and/or document any processes that improve the performance.

Remaining

• [x] Move warmup login in JetBrains Launcher
  • [x] Update JetBrains Launcher to support warmup command [1]
  • [x] Move warmup logic from ide-service to JetBrains Launcher [2]
  • [x] deploy and rebuild all stable and latest images (re-run nightly jobs)
• [x] switch 100% of traffic to new ide-service resolver workspace config endpoint
• [x] decouple JetBrains Launcher from JB IDE images
• [ ] Remove warmup logic from ide-service.ts in server

Completed

• [x] Allow layering multiple JetBrains products in one image

Value & measuring the impact

• [ ] Sync with Dev Rel on some possible announcements and content or blog posts (e.g. blog post showing working with a single project like petclinic for Maven. Show some performance before/after benchmarks and talk about the relation and relevance to CDEs, Gitpod Prebuilds, etc)

Considerations

• Performance and cost implications - Ideally we need to find a way for a project to opt-in to indexing, without burdening the user with too much (or any configuration) if we can.
• Backends included in prebuilds - Currently backends are loaded on top of an existing workspace, not during pre-build, so for the indexing to be done before the workspace starts we need to have the backend executed within a prebuild.

Related issues:

• https://github.com/gitpod-io/gitpod/issues/9189
• See internal link from discussion with Jetbrains
• Also see internal thread from Gitpodders discussing implementation
• https://github.com/gitpod-io/gitpod/issues/15387

[U1F984]

The remote-dev-server.sh seems to support a warm-up command (similar to installPlugins) which could be used.

[akosyakov]

@U1F984 yes, but there are several issues:

• desktop IDEs right now are not running in prebuilds
• index is not stored in /workspace folder, so it is not cached and does not survive restart

[U1F984]

If it is unfeasible to store the system directory in /workspace maybe Shared project indexes could be used. However this is an ultimate edition feature.

[loujaybee]

@akosyakov can you link to full workspace back-up issues, and incremental prebuild work which might be blocking this issue?

[akosyakov]

System config and cache directories were moved under /workspace folder as a part of https://github.com/gitpod-io/gitpod/pull/8158.

We need to add ability to include JB backend into prebuilds. There following things to consider:

• indexes created by one backend is mostly not reusable by created by another backend, so there should be a way to configure for which backend indexes should be created per repo/project
• there are different places where we could configure it, like in .gitpod.yml or in settings of project.

As a prototype, for the start we could support only IntelliJ indexing and it in .gitpod.yml as an experimental feature.

[svenefftinge]

It should be in .gitpod.yml if we create a dependency from the init task. Would it as an intermediate solution be ok to simply download the binaries (in docker or init task) instead of configuring the IDE on project level?

[kiview]

We are currently trying out Gitpod as maintainers of https://github.com/testcontainers/testcontainers-java/ and the IntelliJ integration is a big improvement for us a Java developer.

However, one of my main use cases is to review and try out PRs and because of the nature of our repository, IntelliJ will take 10 minutes or more to index the project. This makes using workspaces as throwaway instances for trying out things very hard for me.

So I just tried to explain our use case and highlight how such a feature would considerably unblock our Gitpod adoption 🙂

[akosyakov]

We don't support it natively yet, but I got it working in this branch: https://github.com/akosyakov/spring-petclinic

Important bits:

• make sure that your gradle/maven projects have configured java target compatible with Java in your image (11 in default image) otherwise JDK indexing is stuck: https://github.com/akosyakov/spring-petclinic/blob/d02d76efe688f5d339c69f075d73a03239841dcb/pom.xml#L32-L33
• add prebuild task: https://github.com/akosyakov/spring-petclinic/blob/836130206689e091a763311ca3070ad204e1a34c/.gitpod.yml#L3-L17

This task downloads IntelilJ backend which MUST be aligned with https://github.com/gitpod-io/gitpod/blob/63d893c6c23594c1c273f0aec13e256bae71606c/WORKSPACE.yaml#L11 Then it configures to move indexes to the same directory which will be used by real backend in regular workspace and runs warm up.

If something does not work (hangs) you can run the same task in regular workspace to trace with jstack. Important thought you should use another folder, since only one backend can lock the project folder at the same time. Please capture then warmup task logs and jstack output and share here.

[akosyakov]

We shipped an experimental support of indexing for IntelliJ IDEA using Prebuilds.

[loujaybee]

Adding note that we have:

• https://github.com/gitpod-io/gitpod/issues/7426

To move prebuild configurations to the project page. It might therefore make sense to move the JetBrains prebuilds configurations also to that page in the UI rather than in the configuration.

[loujaybee]

Renamed the title of this issue, it was created a long time ago when our understanding of JetBrains Gateway was just forming. We can see now the opportunity to focus on Maven for the time being to get the Prebuilds and opening IDE experience working really well. I'll scope this issue down now to just Maven. However, we'll come back and open a companion issue for Gradle also in time. Learnings from integrating Maven fully with Gitpod will then also carry over.

[loujaybee]

@andreafalzetti has been doing some analysis to compare:

• starting without prebuild to starting with prebuild
• restarting a stopped container

Will share on the issue. JetBrains mentioned improving the warmup features in February, we'll need to make sure our changes and the JetBrains changes are compatible.

[loujaybee]

Work currently paused, waiting on some input from JetBrains.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.