Support custom IDEs provided through docker images

meysholdt commented 3 years ago

Since we now have the foundation in place to install a custom IDE in Gitpod using a Docker Image, It would be great to have a an example and docs that explain the minimal requirements Gitpod imposes on such an image.

csweichel commented 3 years ago

A rough, and still changing list of requirements we have towards IDE images:

all static assets that we need to serve before the workspace pod is up and running MUST be in the very first layer
there must be an executable file in /ide/startup.sh that's executed to start the IDE itself. This can be changed in the IDE supervisor config (see below).
no system-folder must be overwritten, e.g. WORKDIR /usr/bin possibly creates an empty /usr/bin directory which would shadow the actual /usr/bin in the workspace.

IDE supervisor config

IDEs can provide some configuration to customise supervisor's behaviour. This file is expected in /ide/supervisor-ide-config.json and may look like this:

{
  "entrypoint": "/ide/startup.sh",
  "readinessProbe": {
    "type": "http",
    "http": {
      "path": "static/manifest.json"
    }
  }
}

The entrypoint configures the executable that get's run when starting the IDE. The readinessProbe configures a probe that determines when the IDE is actually running. Right now we only support http probes which attempt a GET http://localhost:<ide-port>/<path> every five seconds. If that request returns 200 OK we consider the IDE running. If there's no probe configured, we consider the IDE running right after it got started (i.e. the entrypoint was executed).

Environment Variables coming from Gitpod

IDEs are expected to run on a particular port, specifically the one indicated on the GITPOD_THEIA_PORT environment variable. Expect this name to change in the near future.

Environment Variables coming from the IDE image

IDE images can contribute environment variables to a workspace. The details are work in progress.

csweichel commented 3 years ago

We might want to introduce a linter for images that can verify those requirements. In its first iteration that linter would just be a development tool, but could conceivably be integrated into the system as a whole.

Also, we should look at ways for finding good defaults to relax those constraints over time.

akosyakov commented 3 years ago

IDE should have index.html and integrate supervisor as described in https://github.com/gitpod-io/gitpod/blob/c749c277e4f0ab308e143502c470b7d0690c7820/components/supervisor/frontend/src/index.ts#L7-L11 is another requirement.

csweichel commented 3 years ago

First step would to be produce a working image for another IDE/editor, e.g. vim or https://github.com/JetBrains/projector-docker

svenefftinge commented 3 years ago

Let's do Theia as the first step.

svenefftinge commented 3 years ago

To create a great user experience we will need to provide a rich integration with Gitpod for accessing services such as ports, env vars, tokens etcs. We should investigate if we really want to go down the path of switching the primary IDE or rather let users start an additional IDE from within their workspaces. That way they could still use VS Code to interact with Gitpod and use A projector IDE or Jupyter Notebook for the coding task.

yyoncho commented 3 years ago

Is there a minimal example of how to create custom IDE image? I tried to create such using python -m http.server at https://github.com/yyoncho/lsp-gitpod but still, VScode pops up.

svenefftinge commented 3 years ago

We have removed this item from the roadmap for now, as it would create too much development and maintenance resources for too little value. The prioritization was based on the assumption that Jetbrains Projektor is a good alternative for Jetbrains users, but it turned out to leave too much to desire. We are doubling down on an integration with locally running IDEs instead.

yyoncho commented 3 years ago

@svenefftinge

I am looking for help with this in order to create integration between gitpod and Emacs (which can be maintained by emacs community here or in a separate repo if possible).

We are doubling down on an integration with locally running IDEs instead.

Can you elaborate?

ericdallo commented 3 years ago

I am looking for help with this in order to create integration between gitpod and Emacs (which can be maintained by emacs community here or in a separate repo if possible).

Yeah, it'd be great if that could be possible @svenefftinge, having Emacs on Gitpod would be really awesome to the community I think

rmchale commented 3 years ago

This goes against what what gitpod is to me which is a full-web IDE. I want to develop from my iPad.

We have removed this item from the roadmap for now, as it would create too much development and maintenance resources for too little value. The prioritization was based on the assumption that Jetbrains Projektor is a good alternative for Jetbrains users, but it turned out to leave too much to desire. We are doubling down on an integration with locally running IDEs instead.

behrica commented 3 years ago

I am looking for help with this in order to create integration between gitpod and Emacs (which can be maintained by emacs community here or in a separate repo if possible).

Yeah, it'd be great if that could be possible @svenefftinge, having Emacs on Gitpod would be really awesome to the community I think

I am just doubting that an "emacs base image" allone would be enough, because:

There are a lot of different flavors of emacs
The .emacs.d directory of every emacs needed to be copied in some form into the image.

One very different idea to allow a "custom IDE" could be: https://github.com/gitpod-io/gitpod/issues/6668

behrica commented 3 years ago

I am looking for help with this in order to create integration between gitpod and Emacs (which can be maintained by emacs community here or in a separate repo if possible).

Yeah, it'd be great if that could be possible @svenefftinge, having Emacs on Gitpod would be really awesome to the community I think

See here for a working example: https://github.com/behrica/my-gitpod-template

This installs and starts Doom Emacs in novnc in browser including "my personal" doom emacs config.

buggtb commented 3 years ago

I like the idea of providing the ability to integrate IDE's but in reality Idea and Projector seem like enough for me. I can potentially develop locally at home or via web browser when remote using an interface I am familiar with.

The bigger issue I have faced with leveraging Docker images(specifically the projector template) is that in that example the .jetbrains folder is mounted to provide the ability to persist stuff, but its also persisted to git, and the stuff written in there does version, but isn't the most suited for version control. I've used Git LFS for now because its large and binary.

But that involved a few tweaks and in reality it would be better to allow the docker containers to mount some form of "disk", NFS or whatever to allow this stuff to be stored on. I'm not sure how you all do it with VSCode and user preferences, but the custom docker images require something similar.

behrica commented 3 years ago

I am looking for help with this in order to create integration between gitpod and Emacs (which can be maintained by emacs community here or in a separate repo if possible).

Yeah, it'd be great if that could be possible @svenefftinge, having Emacs on Gitpod would be really awesome to the community I think

I made an Gitpod configuraton for Emacs in novnc here: https://github.com/behrica/my-gitpod-template

behrica commented 3 years ago

I am perfectly fine, with the fact that in my configuration VsCode is started "as well". So I don't see it super important to "replace VsCode".

I am wondering if Gitpod should gain some form of customisation options, so that some "part" of the final Dockerfile (see line 21 onwards) which "I want" (and not the project) can be customise by me. So that the project Dockerfile (which would be part of teh project) might contain only:

# PROJECT specific configuration
FROM gitpod/workspace-full-vnc:commit-24b60e9fc9a28e7967a129861736578d66101661

# install Clojure
RUN sudo curl -O https://download.clojure.org/install/linux-install-1.10.3.1020.sh && sudo chmod +x linux-install-1.10.3.1020.sh && sudo ./linux-install-1.10.3.1020.sh

# install emacs27
RUN sudo add-apt-repository -y ppa:kelleyk/emacs
RUN sudo apt-get update
RUN sudo apt-get remove -y emacs26-common emacs26 emacs-common emacs apel flim w3m-el emacs-el emacs-bin-common 
RUN sudo DEBIAN_FRONTEND=noninteractive apt-get install -y keyboard-configuration
RUN sudo DEBIAN_FRONTEND=noninteractive apt-get install -y emacs27 fonts-hack rlwrap i3-wm

and the rest of the configuration is "somewhere else". This needed to be a "per user / per workspace" setting

# use my doom config
RUN rm -rf ~/.emacs.d/ && git clone --depth 1 https://github.com/hlissner/doom-emacs ~/.emacs.d
RUN ~/.emacs.d/bin/doom env 
RUN yes | ~/.emacs.d/bin/doom -y install
RUN rm -rf /home/gitpod/.doom.d/ && git clone https://github.com/behrica/doom.d.git /home/gitpod/.doom.d/
RUN GIT_SSL_NO_VERIFY=true ~/.emacs.d/bin/doom -y sync

# autostart emacs
ENV WINDOW_MANAGER openbox-session
RUN  mkdir -p  /home/gitpod/.config/openbox
RUN echo "emacs" >  /home/gitpod/.config/openbox/autostart

My example asks somehow for 2 Dockerfiles, a project specific and a user specific, which the Gitpod workspace start combines in some form.

behrica commented 3 years ago

Maybe this could be done rather simple:

Before building an image, Gitpod is looking in the repo for a user specific Dockerfile. This could be a file following a convention , linked to my gitpod user name / email: .gitpod_Carsten Behring_.dockerfile
This user specific Dockerfile get appended as is to the dockerfile specified in: gitpod.yml/image/dockerfile
Image gets build from combined Dockerfile, as usual

An other variant could be to allow user specific lines in .gitpod.yaml, which allow as well to express the location of a user specific Dockerfile (maybe even as public URL)

Or maybe account variables could be used in some form for this, pointing to a user + project specific second Dockerfile. I made an feature request for this as well: https://github.com/gitpod-io/gitpod/issues/6668

behrica commented 3 years ago

I thought further about it, and it is true that Gitpod supports what I want already today via "tasks".

A start task (which can be any script) could somehow install all needed, and allow emacs start on "single command". The "personalisation" of that could maybe be done, once dotfile support is there.

Nevertheless I am not 100 % happy about this.

The reason is that the choice of IDE is very personal and should be fully customizable by a user. even though a lot of project will support "officially" only one IDE / editor (which will be baked into the central dev-image config)

The approach of "start tasks" is "per project", not "per user".

But we have as well a conflicting need from the user. I don't want to see a "copy" of my Emacs config in every project I work on... I want to be able to "reuse" "my configuration", while still able to "re-use and change".

So ideally I am free to "customize" the IDE for every project, but the customization should not require me to "copy" neither.

yyoncho commented 3 years ago

@behrica FYI there is https://github.com/emacs-lsp/lsp-gitpod

behrica commented 3 years ago

@yyoncho Very interesting. In my approach above everything hardcoded, but I load "my" doom config from github.

I still see the issue, that a "project setup" vs "IDE setup" would somehow require 2 dockerfiles.

And one of this is "fixed and project specific"

while the other is "user specific, and more then one needed per user, eventually slightly different per project".

Maybe the user specific part can be addressed by "scripts and env variables", so not needing change to Gitpod. But I am not sure about this.

behrica commented 3 years ago

If a user X works on 3 project

Project A
Project B Project C

he should in my view be able to says any of:

I want to use the "same" Emacs setup in A,B,C. (same means pointing to one place wit a common setup) -> changing it + rebuild images applies the changes in A+B+C
I want to use "three different" Emacs setup for A,B,C -> so independent Emacs setup for A,B,C
Anything between. So 2 projects share one setup, and a third is independent

How to realize a "setup" is an orthogonal question. (Dockerfile snippet, vs. included scripts, vs. tasks)

yyoncho commented 3 years ago

@behrica this is what lsp-gitpod does - it has the config in lsp-gitpod and then you can open whatever project you want using that config. Similarly, you can fork lsp-gitpod project, change the docker image and use that repo as a root. Of course, it will be better and more convenient if you can configure the docker image to run via env/url variable.

behrica commented 3 years ago

Yes, but the "project maintainer" needed to worry about this different Emacs setups.

In my view, a project contributor should be able to "swap IDE", independent from tee gitpod.* files of the project. So he should be able to "add" to it in some form.

As a project maintainer I want to support a single IDE only, maybe for convenience always the default VSCode only.

yyoncho commented 3 years ago

In my view, a project contributor should be able to "swap IDE", independent from tee gitpod.* files of the project. So he should be able to "add" to it in some form

Again, this is what lsp-gitpod does. The project's .gitpod files are irrelevant - it is using lsp-gitpod's .gitpod files.

behrica commented 3 years ago

I am the maintainer of https://github.com/scicloj/sklearn-clj as example. This needs Clojure + Python. As a maintainer I want to worry about ONE setup which has both working. So for "easiness" I would decide to just support Gitpod, with a very simple Dockerfile + calva extension.

If now one of my contributors want to use Emacs, it's his problem. But it would be "nice, if that user could re-use my Dockerfile or something else I configured in "some form". He can of course just build the Dockerfile, but could not run "startup tasks" in gitpod.yaml very easely, for example.

He could of course "read teh gitpod.yml" and execute by hand the tasks. So he woukld "simulate" the startup of gitpod. Would be better, if this could be automated.

behrica commented 3 years ago

Maybe your example "https://github.com/emacs-lsp/lsp-gitpod" is not the best for our discussion, as it is about an Emacs plugin.

I talk about projects, which can be developed either using Emacs or VsCode, for example Clojure projects. or python projects.

I (as maintainer) would like to :

specify one IDE setup in the code repository
leave contributors the freedom to swap the IDE while still supporting them with a partial setup (the non-IDE part)

In my example repo(https://github.com/scicloj/sklearn-clj) above there are three setups, conceptually:

python
java + clojure libs
IDE

All contributors SHOULD use the same python + java/clojure versions. So I would like to "freeze" this configuration in a Dockerfile. The IDE I don't care (as maintainer). Emacs, VI, vscode, whatever. (leave it to Contributors to decide) Nevertheless I would like to configure a default iDE, so new-comers (which maybe don't care on the IDE) get a quickstart

behrica commented 3 years ago

Current Gitpod does not allow do do this, im my view. I am not sure what exactly is missing, but it is about lack of "user/project specific customisation options".

Or maybe options to "modularize" the Dockerfile, into a projects specific and a user specific part.

behrica commented 3 years ago

VsCode does not allow this neither, so there I cannot look neither for the feature I would like to see.

behrica commented 3 years ago

An other "interesting use case" would be to be able to "reuse" the .gitpod and .gitpod.Dockerfile without using GitPod at all. (This should not be discussed here, as we are in "discussions of GitPod features).

Maybe some contributors of my project above, don't want to use Cloud Editors at all. It would be nice, if they could use the gitpod files to setup something local (build the image, runs starts tasks full automatically, etc.)

VsCode has a "devcontainer cli" tool, which does a part of this for devcontainer.json files. It builds the image as VSCode would build it, but outside VsCode, on command line.

dunnry commented 2 years ago

Just commenting that we would be very interested in supporting custom IDEs in Gitpod.

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

loujaybee commented 2 years ago

Whilst it seems very unlikely this issue will be picked up in the short-term, I'm keen to keep the issue open, to hear of other use-cases from other communities who might want to leverage this type of functionality.

axonasif commented 2 years ago

It might make more sense for SSH users.

inlann commented 2 years ago

I think there will be more and more specialized IDEs for different industries in the future, and cloud-based versions will be the way of the future. If Gitpod can turn the IDE access into an API, I believe more IDE companies will do business with Gitpod in the future 😄

loujaybee commented 1 year ago

Re-opening as we think about the longer term possibilities of general integrations around IDEs. No guarentee of significant movement on this issue in the short term, but would like to bring again more visibility to this issue.

gengjiawen commented 1 year ago

Is it an options to config IDE in .gitpod.yml ?

Ref: https://github.com/gitpod-io/gitpod/issues/3274#issue-816554458

erikmd commented 1 year ago

Dear Gitpod maintainers,

I'm very interested to ask: what is the status of the support for custom IDEs and/or for Emacs in particular?

Just to complement my question:

I tested https://github.com/emacs-lsp/lsp-gitpod successfully: it is really nice! — albeit the current requirement that "a VScode tab has to be open first and kept open" is a major limitation…
if ever you'd remark that Emacs is not as popular as VScode or JetBrains IDEs, it still happens it is the most used IDE in the OCaml community for instance (cf. e.g. the OCaml User Survey 2022)

Thanks for your time!

axonasif commented 1 year ago

Hi @erikmd,

I think emacs is very usable with Gitpod at the current state. You could configure your Emacs setup through dotfiles installation script and then use SSH or xterm.js based browser terminal (private beta).

I'm personally not very familiar with Emacs but I think I've a decent neovim setup through my dotfiles installer, it also has an option for auto installing emacs but probably incomplete.

erikmd commented 1 year ago

Hi @axonasif, thanks a lot for your quick reply!

I think emacs is very usable with Gitpod at the current state.

AFAICT, yes it works, but there is one major usability issue, as documented here:

https://github.com/emacs-lsp/lsp-gitpod#limitations

VScode tab has to be kept open and you have to click every 30 minute on it to keep the session open. This issue will be sorted out with gitpod team.

E.g. if one browses

https://gitpod.io/#ORG=emacs-lsp,PROJECT=demo-projects,PROFILE=spacemacs/https://github.com/emacs-lsp/lsp-gitpod

we get a VScode window (while we'd really like to just get an Emacs tab, out-of-the-box).

Though, your reply makes me doubt… Do you think the limitation above mentioned in the emacs-lsp project has already been addressed? so this would just be a documentation issue 🙂

Thanks!

axonasif commented 1 year ago

@erikmd I wasn't talking about https://github.com/emacs-lsp/lsp-gitpod but the currently available Gitpod features that you could utilize to run emacs from a terminal emulator over SSH or this.

erikmd commented 1 year ago

@axonasif OK I see! thanks for this information 😊

Indeed, vanilla GNU Emacs and spacemacs provide both GUI and TTY front-ends, but a TTY spacemacs that works out of the box in GitPod is already a very good news!

loujaybee commented 1 year ago

what is the status of the support for custom IDEs

@erikmd if you sign up at gitpod.io/labs we can sign you up for the private beta for the terminal based editor in the browser that @axonasif mentioned, it would be great to get your feedback on how/if that works for you. Regarding the emacs lsp project, we have also been experimenting with custom IDE support, which could potentially be used in this context. We've been looking for more concrete examples that might use this type of feature, so that project would also be a good one for us to experiment with.

erikmd commented 1 year ago

@loujaybee OK, thanks for your advice and feedback!

@erikmd you sign up at gitpod.io/labs

Done! Looking forward to being able to test all this soon 🙂

erikmd commented 1 year ago

Hi @loujaybee ! after I signed up at the URL you mentioned, is there any further step to do so that I'll be able to select a browser based terminal setting in https://gitpod.io/new ? Thanks for your time!

(FYI, my email set in https://gitpod.io/user/account is erikmd@mailfence.com)

loujaybee commented 1 year ago

Hey @erikmd, no further steps, you should have access now, can you retry?

erikmd commented 1 year ago

Indeed, @loujaybee, thanks !

BTW I posted some feedback on https://github.com/gitpod-io/gitpod/issues/17724#issuecomment-1597862258

erikmd commented 1 year ago

@erikmd I wasn't talking about https://github.com/emacs-lsp/lsp-gitpod but the currently available Gitpod features that you could utilize to run emacs from a terminal emulator over SSH or this.

@axonasif I was able to test emacs in a Browser Terminal workspace, but out of curiosity, is the spacemacs-based .gitpod.yml you used in the above screenshot available online?

axonasif commented 1 year ago

is the spacemacs-based .gitpod.yml you used in the above screenshot available online?

No, I'm using a dotfiles installation script to install it on all workspaces automatically (also mentioned in my initial message 😉 )

erikmd commented 1 year ago

Thanks! 👍 for the record, I needed to:

Browse https://gitpod.io/user/preferences
- Scroll to Dotfiles
- Put https://github.com/axonasif/dotsh
- Click on Save
Browse https://gitpod.io/user/variables
- Add DOTFILES_EDITOR ; emacs ; */*
- Add DOTFILES_EDITOR_PRESET ; spacemacs (or doomemacs) ; */*
Open https://gitpod.io/new

gitpod-io / gitpod